In this scenario, the NFS Server unexpectedly went offline. The command would hang when we attempt to run any commands on the NFS mount point. Since our quorum disk is on this NFS Server, we will temporarily move the the voting disk from CRS to CRSTEMP to avoid disruption to the database services.
Note:
Outputs/usernames/node names in the examples are from my system. Make sure you use the values from your system.
$GRID_HOME is the software location of the Grid Infrastructure (GI).
grid user is the GI software owner.
rac1/rac2 are the node names of Oracle RAC.
nfssrv1 is the NFS server where the quorum disk is located.
The current OCR and voting disk are in the CRS diskgroup. CRSTEMP diskgroup was created from 3 3GB SAN volumes that were presented to both RAC nodes.
Step 1 – Check the current OCR and voting disk configurations.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1656
Available space (kbytes) : 407912
ID : 116950245
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
# cat /etc/oracle/ocr.loc
ocrconfig_loc=+CRS/rac-cluster/OCRFILE/registry.255.1005327841
local_only=false
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a3b727e82c154f1ebfa440d68bc9816b (/voting_disk/vote_stnsp0506) [CRS]
2. ONLINE a6f0520bcfd34fbcbf82d27272c3fe86 (ORCL:CRS1) [CRS]
3. ONLINE 8498d867e1ab4f61bf1d5863e4215c5d (ORCL:CRS2) [CRS]
Located 3 voting disk(s).
Step 2 – Backup the OCR before we make any changes.
Note: Oracle does not provide a way to directly backup the voting disk but it is not needed. In 12c, Oracle automatically backs up the voting file data in OCR whenever any configuration changes are made. Unless otherwise specified, run the commands in this step as root user.
# $GRID_HOME/bin/ocrconfig -manualbackup
rac1 2019/04/25 16:10:51 /oracle/oraBase/12.1.0.2/grid/cdata/rac-cluster/backup_20190425_161051.ocr 683622755
Step 3 – Moving the OCR from CRS to CRSTEMP.
Add OCR to CRSTEMP.
# $GRID_HOME/bin/ocrconfig -add +CRSTEMP
Verify OCR is on both CRS and CRSTEMP
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1728
Available space (kbytes) : 407840
ID : 116950245
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File Name : +CRSTEMP
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Remove it from +CRS.
# $GRID_HOME/bin/ocrconfig -delete +CRS
Verify that OCR is only located in CRSTEMP.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1704
Available space (kbytes) : 407864
ID : 116950245
Device/File Name : +CRSTEMP
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Step 4 – Move the voting disks to CRSTEMP.
# $GRID_HOME/bin/crsctl replace votedisk +CRSTEMP
Successful addition of voting disk f1e6ebddd7a24f8abf5ef8972f4131cc.
Successful addition of voting disk 7ceb309034034f2abf3bcf6f1e88673f.
Successful addition of voting disk 87aa827160124f13bf92d1d3c03b7dc0.
Successful deletion of voting disk a3b727e82c154f1ebfa440d68bc9816b.
Successful deletion of voting disk a6f0520bcfd34fbcbf82d27272c3fe86.
Successful deletion of voting disk 8498d867e1ab4f61bf1d5863e4215c5d.
Successfully replaced voting disk group with +CRSTEMP.
CRS-4266: Voting file(s) successfully replaced
Verify voting disks are now located in CRSTEMP.
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE f1e6ebddd7a24f8abf5ef8972f4131cc (ORCL:CRS3) [CRSTEMP]
2. ONLINE 7ceb309034034f2abf3bcf6f1e88673f (ORCL:CRS4) [CRSTEMP]
3. ONLINE 87aa827160124f13bf92d1d3c03b7dc0 (ORCL:CRS5) [CRSTEMP]
Located 3 voting disk(s).
Note:
The remaining steps will relocate the voting disks from CRSTEMP back to CRS once the NFS Server comes back online.
Step 5 – Reboot the RAC nodes one at a time to clear the NFS hang issues.
Note: You can skip step 5 if you don’t have any NFS hang issues on rac1/rac2 after the NFS Server came back online.
On both RAC nodes, comment out the NFS mount from /etc/fstab so the system won’t hang during reboot due to unreachable NFS mount.
#nfssrv1:/votedisk /voting_disk nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 0 0
Next, reboot rac1.
# reboot
Verify all rac1 database resources are up. For now, you can ignore ora.CRS.dg (CRS diskgroup) being down since /voting_disk is not mounted.
$ $GRID_HOME/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE OFFLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.CRSTEMP.dg
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.DATA.dg
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.FRA.dg
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.asm
ONLINE ONLINE rac1 Started,STABLE
ONLINE ONLINE rac2 Started,STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac1 169.254.96.182 10.10
.10.11,STABLE
ora.cvu
1 ONLINE ONLINE rac2 STABLE
ora.oc4j
1 ONLINE ONLINE rac2 STABLE
ora.prod.ads.svc
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.prod.cview.svc
1 ONLINE ONLINE rac2 STABLE
ora.prod.db
1 ONLINE ONLINE rac1 Open,STABLE
2 ONLINE ONLINE rac2 Open,STABLE
ora.prod.wfm.svc
1 ONLINE ONLINE rac2 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
Warning: Wait until all the rac1 database services are completely up before rebooting rac2 to avoid downtime.
Next, reboot rac2 and wait for the cluster resources to come up before going to step 6.
# reboot
Step 6 – Moving the OCR from CRSTEMP back to CRS.
Uncomment /voting_disk in /etc/fstab for both RAC nodes.
Mount the /voting_disk on both RAC nodes.
# mount /voting_disk
# df -h /voting_disk
Filesystem Size Used Avail Use% Mounted on
nfssrv1:/votedisk 50G 12G 39G 24% /voting_disk
As user grid, mount CRS diskgroup on both RAC nodes.
SQL> alter diskgroup CRS mount;
Diskgroup altered.
Add OCR to CRS.$ asmcmd lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 4096 1048576 13310 13067 5119 2455 0 N CRS/
MOUNTED NORMAL N 512 4096 4194304 15348 14512 5116 4698 0 Y CRSTEMP/
MOUNTED NORMAL N 512 4096 4194304 102384 36456 25596 5430 0 N DATA/
MOUNTED NORMAL N 512 4096 4194304 40944 34488 10236 12126 0 N FRA/
# $GRID_HOME/bin/ocrconfig -add +CRS
Remove it from +CRSTEMP.
# $GRID_HOME/bin/ocrconfig -delete +CRSTEMP
Verify that OCR is now located in CRS.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1704
Available space (kbytes) : 407864
ID : 116950245
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Step 7 – Move voting disks to CRS.
# $GRID_HOME/bin/crsctl replace votedisk +CRS
Successful addition of voting disk c25c668c48c04ff5bf6205586bbd8cf7.
Successful addition of voting disk b6e73f8207c24f60bf59ebbcf0c00145.
Successful addition of voting disk 6517509aff3d4fe9bf813125d2484ae9.
Successful deletion of voting disk f1e6ebddd7a24f8abf5ef8972f4131cc.
Successful deletion of voting disk 7ceb309034034f2abf3bcf6f1e88673f.
Successful deletion of voting disk 87aa827160124f13bf92d1d3c03b7dc0.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced
Verify voting disks are now located in CRS.
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a3b727e82c154f1ebfa440d68bc9816b (/voting_disk/vote_stnsp0506) [CRS]
2. ONLINE a6f0520bcfd34fbcbf82d27272c3fe86 (ORCL:CRS1) [CRS]
3. ONLINE 8498d867e1ab4f61bf1d5863e4215c5d (ORCL:CRS2) [CRS]
Located 3 voting disk(s).
Step 8 – Backup the OCR to capture the new configuration settings.
# $GRID_HOME/bin/ocrconfig -manualbackup