In this scenario, the NFS Server that holds the quorum disk unexpectedly went offline. The command would hang when there are any attempts to run them on the NFS mount point. An attempt to drop/re-add the voting disk has put CRS in a failure state on rac1. CRS will not start on rac1 after reboot. The NFS mount is still hung on rac2. A reboot of rac2, followed by a reboot of rac1 brought both RAC nodes back online. But now the voting disk is missing and trying to add it back causes “ORA-15272: failgroup CRS_0001 is not a quorum failgroup”.
Note:
Outputs/usernames/node names in the examples are from my system. Make sure you use the values from your system.
$GRID_HOME is the software location of the Grid Infrastructure (GI).
grid user is the GI software owner.
rac1/rac2 are the node names of Oracle RAC.
nfssrv1 is the NFS server where the quorum disk is located.
The current OCR and voting files are in the CRS diskgroup. CRSTEMP diskgroup was created from 3 3GB SAN volumes that were presented to both RAC nodes. Go here for details on how to add CRSTEMP: Moving An NFS Quorum Disk to another NFS Server On Oracle 12c Extended RAC Cluster
As seen below, the voting disk (ie. /voting_disk/vote_20230831) is missing from the CRS diskgroup.
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 6445db120e694f53bf1c08014fc8b288 (/dev/oracleasm/disks/ASMCRSB001) [CRS]
2. ONLINE 049cf5f59d214f02bf699033f68f40f9 (/dev/oracleasm/disks/ASMCRSC001) [CRS]
Located 2 voting disk(s).
Trying to add the voting disk back into the CRS diskgroup failed with:
SQL> alter diskgroup CRS add quorum disk '/voting_disk/vote_20230831' size 3072M;
alter diskgroup CRS add quorum disk '/voting_disk/vote_20230831' size 3072M
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15272: failgroup CRS_0001 is not a quorum failgroup
Solution: To get around this problem, we will move the voting files and the OCR into CRSTEMP. Then we will drop and recreate the CRS diskgroup and move the voting files and the OCR back into the CRS diskgroup.
Step 1 – Check the current OCR and voting disk configurations.
# /oracle/oraBase/12.1.0.2/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1796
Available space (kbytes) : 407772
ID : 1939117622
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 6445db120e694f53bf1c08014fc8b288 (/dev/oracleasm/disks/ASMCRSB001) [CRS]
2. ONLINE 049cf5f59d214f02bf699033f68f40f9 (/dev/oracleasm/disks/ASMCRSC001) [CRS]
Located 2 voting disk(s).
Step 2 – Backup the OCR before we make any changes.
Note: Oracle does not provide a way to directly backup the voting disk but it is not needed. In 12c+, Oracle automatically backs up the voting file data in OCR whenever any configuration changes are made. Unless otherwise specified, run the commands in this step as root user from rac1.
# $GRID_HOME/bin/ocrconfig -manualbackup
Step 3 – Moving the OCR from CRS to CRSTEMP.
Add OCR to CRSTEMP.
# $GRID_HOME/bin/ocrconfig -add +CRSTEMP
Remove it from +CRS.
# $GRID_HOME/bin/ocrconfig -delete +CRS
Verify that OCR is now located in CRSTEMP.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1820
Available space (kbytes) : 407748
ID : 1939117622
Device/File Name : +CRSTEMP
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Move voting disks to CRSTEMP.
# $GRID_HOME/bin/crsctl replace votedisk +CRSTEMP
Successful addition of voting disk 70f1dcf8ffc64f6dbf50ec891fba534e.
Successful addition of voting disk 69ff03149a274f35bf0764f358eff97d.
Successful addition of voting disk 6d8132f125714fe6bf8173b727d33d37.
Successful deletion of voting disk 6445db120e694f53bf1c08014fc8b288.
Successful deletion of voting disk 049cf5f59d214f02bf699033f68f40f9.
Successfully replaced voting disk group with +CRSTEMP.
CRS-4266: Voting file(s) successfully replaced
Verify voting disks are now located in CRSTEMP.
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 70f1dcf8ffc64f6dbf50ec891fba534e (/dev/oracleasm/disks/ASMCRSB002) [CRSTEMP]
2. ONLINE 69ff03149a274f35bf0764f358eff97d (/dev/oracleasm/disks/ASMCRSB003) [CRSTEMP]
3. ONLINE 6d8132f125714fe6bf8173b727d33d37 (/dev/oracleasm/disks/ASMCRSC003) [CRSTEMP]
Located 3 voting disk(s).
Step 4 – Drop and recreate the CRS diskgroup.
On rac2, dismount the CRS:
SQL> alter diskgroup crs dismount;
On rac1, drop and recreate the CRS diskgroup.
SQL> drop diskgroup crs including contents;
Diskgroup dropped.
SQL> CREATE DISKGROUP CRS NORMAL REDUNDANCY
DISK '/dev/oracleasm/disks/ASMCRSC001',
'/dev/oracleasm/disks/ASMCRSB001'
QUORUM DISK '/voting_disk/vote_20230831' SIZE 3072M
ATTRIBUTE 'compatible.asm'='12.1.0.0.0','compatible.rdbms' = '12.1.0.0.0','au_size'='4M';
Diskgroup created.
On rac2, mount the CRS diskgroup.
SQL> alter diskgroup crs mount;
Diskgroup altered.
Step 5 – Move the OCR from CRSTEMP back to CRS.
On rac1, add OCR in the CRS diskgroup and remove it from CRSTEMP.
$GRID_HOME//bin/ocrconfig -add +CRS
$GRID_HOME/bin/ocrconfig -delete +CRSTEMP
Verify OCR and voting disks are in CRS.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1816
Available space (kbytes) : 407752
ID : 1939117622
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Move voting disks to CRS.
$GRID_HOME/bin/crsctl replace votedisk +CRS
Successful addition of voting disk 1eda34786b704f16bf5f5f9934895704.
Successful addition of voting disk 6cfdf40d4ecd4f3fbf40cecf38d07044.
Successful addition of voting disk 3ef55bf38ed94fe4bfb9781dd646b8d9.
Successful deletion of voting disk 70f1dcf8ffc64f6dbf50ec891fba534e.
Successful deletion of voting disk 69ff03149a274f35bf0764f358eff97d.
Successful deletion of voting disk 6d8132f125714fe6bf8173b727d33d37.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced
Verify voting disks are now located in CRS and the quorum disk is on /voting_disk NFS mount.
$GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1eda34786b704f16bf5f5f9934895704 (/voting_disk/vote_20230831) [CRS]
2. ONLINE 6cfdf40d4ecd4f3fbf40cecf38d07044 (/dev/oracleasm/disks/ASMCRSC001) [CRS]
3. ONLINE 3ef55bf38ed94fe4bfb9781dd646b8d9 (/dev/oracleasm/disks/ASMCRSB001) [CRS]
Step 6 – Backup the OCR to capture the new configuration settings.
# $GRID_HOME/bin/ocrconfig -manualbackup