In this guide we will be moving an NFS quorum disk in ASM diskgroup CRS from NFS server nfssrv1 to nfssrv2.
The current OCR and voting disk are in ASM disk group CRS. The NFS quorum disk is on nfssrv1. By the end of this guide, we will have the NFS quorum disk on nfssrv2.Requirements:
I. Add 3 new 3GB disks
II. NFS Server on nsfsrv2 is set up according to one of these instructions:
> For NFS on Linux 6, see Setting Up NFS on Oracle/Red Hat Linux 6
> For NFS on Linux 7, see Setting Up NFS on Oracle/Red Hat Linux 7
Note: Unless indicated otherwise, run the commands as root user on rac1. Replace $GRID_HOME with the software location of your Grid Infrastructure.
Step 1 – Create 3 ASM disks for a new diskgroup to temporarily hold the OCR and voting disks. For normal redundancy in ASM, the voting disk requires a minimum of 3 ASM disks in the disk group. Replace the /dev/sd* with your new disks. If your system is using multipath, make sure you use the correct multipath device names.
Create a partition on each disk.
fdisk /dev/sdf
fdisk /dev/sdg
fdisk /dev/sdh
Create ASM disks on the partitions you just created.
On rac2, scan for the new ASM disk:# /usr/sbin/oracleasm createdisk CRS3 /dev/sdf1
# /usr/sbin/oracleasm createdisk CRS4 /dev/sdg1
# /usr/sbin/oracleasm createdisk CRS5 /dev/sdh1
# /usr/sbin/oracleasm scandisks
Step 2 – Create a temporary diskgroup call CRSTEMP.
# su - grid
$ sqlplus / as sysasm
SQL> CREATE DISKGROUP CRSTEMP NORMAL REDUNDANCY
DISK 'ORCL:CRS3',
'ORCL:CRS4',
'ORCL:CRS5'
ATTRIBUTE 'compatible.asm'='12.1.0.0.0','compatible.rdbms' = '12.1.0.0.0','au_size'='4M';
Diskgroup created.
On rac2, mount the new diskgroup.
SQL> ALTER DISKGROUP CRSTEMP MOUNT;
Diskgroup altered.
Hint: If oracle can’t find the ASM disks, check your ASM_DISKSTRING parameter.
Step 3 – Check the current OCR and voting disk configurations.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1656
Available space (kbytes) : 407912
ID : 116950245
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
# cat /etc/oracle/ocr.loc
ocrconfig_loc=+CRS/rac-cluster/OCRFILE/registry.255.1005327841
local_only=false
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a3b727e82c154f1ebfa440d68bc9816b (/voting_disk/vote_stnsp0506) [CRS]
2. ONLINE a6f0520bcfd34fbcbf82d27272c3fe86 (ORCL:CRS1) [CRS]
3. ONLINE 8498d867e1ab4f61bf1d5863e4215c5d (ORCL:CRS2) [CRS]
Located 3 voting disk(s).
Step 4 – Backup the OCR before we make any changes.
Note: Oracle does not provide a way to directly backup the voting disk but it is not needed. In 12c, Oracle automatically backs up the voting file data in OCR whenever any configuration changes are made.
# $GRID_HOME/bin/ocrconfig -manualbackup
rac1 2019/04/20 18:32:30 /oracle/oraBase/12.1.0.2/grid/cdata/rac-cluster/backup_20190420_183230.ocr 683622755
Step 5 – Move the OCR from CRS to CRSTEMP.
Add OCR to CRSTEMP.
# $GRID_HOME/bin/ocrconfig -add +CRSTEMP
Remove it from +CRS.
# $GRID_HOME/bin/ocrconfig -delete +CRS
Verify that OCR is now located in CRSTEMP.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1704
Available space (kbytes) : 407864
ID : 116950245
Device/File Name : +CRSTEMP
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Step 6 – Move voting disks to CRSTEMP.
# $GRID_HOME/bin/crsctl replace votedisk +CRSTEMP
Successful addition of voting disk f1e6ebddd7a24f8abf5ef8972f4131cc.
Successful addition of voting disk 7ceb309034034f2abf3bcf6f1e88673f.
Successful addition of voting disk 87aa827160124f13bf92d1d3c03b7dc0.
Successful deletion of voting disk a3b727e82c154f1ebfa440d68bc9816b.
Successful deletion of voting disk a6f0520bcfd34fbcbf82d27272c3fe86.
Successful deletion of voting disk 8498d867e1ab4f61bf1d5863e4215c5d.
Successfully replaced voting disk group with +CRSTEMP.
CRS-4266: Voting file(s) successfully replaced
Verify voting disks are now located in CRSTEMP.
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE f1e6ebddd7a24f8abf5ef8972f4131cc (ORCL:CRS3) [CRSTEMP]
2. ONLINE 7ceb309034034f2abf3bcf6f1e88673f (ORCL:CRS4) [CRSTEMP]
3. ONLINE 87aa827160124f13bf92d1d3c03b7dc0 (ORCL:CRS5) [CRSTEMP]
Located 3 voting disk(s).
Step 7 – Create a 3GB file on the new NFS Server. It will be our new quorum disk.
In this guide, /voting_disk is mounted to the current NFS Server – nsrsrv1. And /voting_disk2 is mounted to the new NFS Server – nsrsrv2.On rac1 and rac2, we should have these 2 mount points.
# df -h /voting*
Filesystem Size Used Avail Use% Mounted on
nfssrv1:/votedisk 50G 11G 40G 22% /voting_disk
nfssrv2:/votedisk2 50G 11G 40G 22% /voting_disk2
On nfssrv2, run these commands to create the 3GB file.
# su - grid
$ dd if=/dev/zero of=/votedisk2/vote_20190425 bs=1M count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 3.4506 s, 934 MB/s
$ ls -lh /votedisk2/
total 3.0G
-rw-r--r--. 1 grid oinstall 3.0G Apr 25 13:26 vote_20190425
Step 8 – Find the name of the NFS quorum disk that needs to be removed from the CRS diskgroup.
# su - grid
$ sqlplus / as sysasm
SQL> set line 200
SQL> col path for a30
SQL> col name for a15
SQL> select name,path from v$asm_disk where path like '%voting%';
NAME PATH
--------------- ------------------------------
CRS_0002 /voting_disk/vote_stnsp0506
Step 9 – Drop the quorum disk.
Warning: Make sure you enter the correct disk name to drop.
SQL> alter diskgroup CRS drop quorum disk CRS_0002 force;
Diskgroup altered.
Step 10 – Remove the old voting disk from the OS so it doesn’t show up in ASM.
$ rm /voting_disk/vote_stnsp0506
Step 11 – Add /voting_disk2 to the asm_diskstring parameter so ASM knows where to find the new disk.
Note: We are also removing the old /voting_disk from asm_diskstring in the same command.
Check the current settings.
SQL> show parameter asm_diskstring
NAME TYPE VALUE
------------------------------------ --------------------------------- ------------------------------
asm_diskstring string /dev/oracleasm/disks/*, /votin
g_disk/*
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*','/voting_disk2/*' sid='*' scope=both;
System altered.
SQL> show parameter asm_diskstring
NAME TYPE VALUE
------------------------------------ --------------------------------- ------------------------------
asm_diskstring string /dev/oracleasm/disks/*, /votin
g_disk2/*
Step 12 – Add the new quorum disk on nfssrv2 to CRS diskgroup.
SQL> alter diskgroup CRS add quorum disk '/voting_disk2/vote_20190425' size 3072M;
Diskgroup altered.
Step 13 – Move the OCR from CRSTEMP back to CRS.
Run these commands as root on rac1.
Add OCR to CRS.
# $GRID_HOME/bin/ocrconfig -add +CRS
Remove it from +CRSTEMP.
# $GRID_HOME/bin/ocrconfig -delete +CRSTEMP
Verify that OCR is now located in CRS.
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1704
Available space (kbytes) : 407864
ID : 116950245
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Step 14 – Move voting disks to CRS.
# $GRID_HOME/bin/crsctl replace votedisk +CRS
Successful addition of voting disk c25c668c48c04ff5bf6205586bbd8cf7.
Successful addition of voting disk b6e73f8207c24f60bf59ebbcf0c00145.
Successful addition of voting disk 6517509aff3d4fe9bf813125d2484ae9.
Successful deletion of voting disk f1e6ebddd7a24f8abf5ef8972f4131cc.
Successful deletion of voting disk 7ceb309034034f2abf3bcf6f1e88673f.
Successful deletion of voting disk 87aa827160124f13bf92d1d3c03b7dc0.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced
Verify voting disks are now located in CRS and the quorum disk is on /voting_disk2.
# $GRID_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE c25c668c48c04ff5bf6205586bbd8cf7 (ORCL:CRS1) [CRS]
2. ONLINE b6e73f8207c24f60bf59ebbcf0c00145 (ORCL:CRS2) [CRS]
3. ONLINE 6517509aff3d4fe9bf813125d2484ae9 (/voting_disk2/vote_20190425) [CRS]
Step 15 – Backup the OCR to capture the new configuration settings.
# $GRID_HOME/bin/ocrconfig -manualbackup