Sunday, November 17, 2013

Moving the 11R2 RAC database servers across datacenter with changed IP address

1) Shutdown the CRS and DISABLE CRS auto start
2) Move the servers to new data center
3) Start the CRS – only on first node(./crsctl start crs)
4) Modify the IP details on OCR using oifcfg
5) Update the GPGNP profile in 11gR2
6) Now bounce CRS on rac1 and start the CRS on rac2
.........................................
note :
PUB.LI.CI.PP -- public ip address (eth1)
PRI.VA.TE.IP -- private ip address ( eth2)
 
4) Modify the IP details on OCR using oifcfg
root@rac1 bin]# ./oifcfg delif -global eth1
root@rac1 bin]# ./oifcfg setif -global eth1/PUB.LI.CI.PP:public -- New ip address netmask
root@rac1 bin]# ./oifcfg delif -global eth2
root@rac1 bin]# ./oifcfg setif -global eth2/PRI.VA.TE.IP:cluster_interconnect --- New ip address netmask
root@rac1 bin]# ./oifcfg getif
eth1 PUB.LI.CI.PP global public
eth2 PRI.VA.TE.IP global cluster_interconnect
root@rac1 bin]# ./srvctl config nodeapps

----
VIP exists: /rac1-vip/<newip>/<old netmask>/255.255.255.128/bond0, hosting node rac1

VIP exists: /rac2-vip/<newip>/<old netmask>/255.255.255.128/bond0, hosting node rac2
----

[root@rac1 bin]# ./srvctl modify nodeapps -n rac1 -A rac1-vip.us.oracle.com/255.255.255.0/eth1
[root@rac1 bin]# ./srvctl modify nodeapps -n rac2 -A rac2-vip.us.oracle.com/255.255.255.0/eth1
[root@rac1 bin]# ./srvctl config nodeapps

----
VIP exists: /rac1-vip/<New IP address>/<PUB.LI.CI.PP>/255.255.255.0/eth1, hosting node rac1

VIP exists: /rac2-vip/<New IP address>/<PUB.LI.CI.PP>/255.255.255.0/eth1, hosting node rac2
----
when try to bounce CRS on rac1 and start the CRS on rac2
Now looking at 11gR2 with same scenario of moving RAC to new datacenter.
When we Start the CRS – we got below errors when we tried starting the CRS with server running on different subnet with new IP address.
” [crsd(21366)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-44: Error in network address and interface operations Network address and interface operations error [7]]. Details at (:CRSD00111:) in /u01/app/grid/11.2.0/log/rac1/crsd/crsd.log.”

What is the difference between 10g/11gR1 and 11gR2??
As of 11.2 Grid Infrastructure, the CRS daemon (crsd.bin) now has a dependency on the private network configuration stored in the gpnp profile and OCR. If the private network is not available or its definition is incorrect, the CRSD process will not start and any subsequent changes to the OCR will be impossible NOTE:1094024.1

5) Update the GPGNP profile in 11gR2

STEPS UPDATE TO 11GR2 GRID GPGNP PROFILE 
(run all the steps on rac1 as ROOT unless specified dont run on other nodes)

STEP1: Start CRS on all the nodes (though CRSD won’t come up we need gpnp deamon running on all the cluster nodes)
[root@rac1 bin]# cd $GRID_HOME/bin/
[root@rac1 bin]#./crsctl start crs
[root@rac1 bin]# ps -ef | grep gpnp

STEP 2: On rac1: As the from 11gR2 – CRS startup is based on GPNP profile we need to update profile to reflect new IP address (to view current profile infomation
[root@rac1 bin]# cd $GRID_HOME/bin/
[root@rac1 bin]# ./gpnptool get >> we can notice the old ip and interface name

STEP 3: on rac1: Take backup of existing GPNP profile before editing (we are going have temp profile XML file and do our modification there in that file)
[root@rac1 bin]#mkdir /home/oracle/gpnp
[root@rac1 bin]#export GPNPDIR=/home/oracle/gpnp
[root@rac1 bin]# ./gpnptool get -o=$GPNPDIR/profile.original
[root@rac1 bin]# cp $GPNPDIR/profile.original $GPNPDIR/p.xml

STEP 4: on rac1: Get current profile sequence number – we need to increment it while editing profile
i.e current prf_sq +1
[root@rac1 bin]# ./gpnptool getpval -p=$GPNPDIR/p.xml -prf_sq -o- ===> this output +1 will be used for prf_sq during edit command
STEP 5: on rac1: Available network interfaces
[root@rac1 bin]#./oifcfg iflist
STEP 6: on rac1: Get the current NETWORK ID from profile and crosscheck the xml file
[root@rac1 bin]# ./gpnptool getpval -p=$GPNPDIR/p.xml -net -o-

exampleOUTPUT : 

[root@rac1 bin]#./gpnptool getpval -p=$GPNPDIR/p.xml -net -o-
net1 net2

[root@rac1 bin]#more $GPNPDIR/p.xml (compare the above net1/net2 with xml file) and append below command based on " (d), (e) and (f)"

[root@rac1 bin]#./gpnptool edit -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -prf_sq=2 -net1:net_use=public -net1:net_ip=PUB.LI.CI.PP -net1:net_ada=eth3

[root@rac1 bin]#./gpnptool edit -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -prf_sq=2 -net2:net_use=cluster_interconnect -net2:net_ip=PRI.VA.TE.IP -net2:net_ada=eth4

[root@rac1 bin]#more $GPNPDIR/p.xml >> we should be seeing new ip and interface names here

STEP 7: on rac1: Sign the XML file with GPNP tool and upload it to update the current profile
[root@rac1 bin]#./gpnptool sign -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -w=cw-fs:peer

[root@rac1 bin]#./gpnptool put -p=$GPNPDIR/p.xml >> At this step if GPNP deamon not running then profile won't get 
updated on that node so we need to make sure it running on all the cluster nodes

STEP 8: on All the nodes : Start the CRS and check Status
[root@rac1 bin]#./gpnptool get >> we will be seeing new IP and interface name
[root@rac1 bin]#./crsctl start res ora.crsd –init
[root@rac1 bin]#./crsctl status res -t
At this point all the CRS deamons will be up (make sure listener/DB not up)

STEP 9: Now we run oifcfg getif — interesting piece of warning/error which ignorable as we edited GPNP profile and its not matching with OCR
[root@rac1 bin]# ./oifcfg getif
eth3 PUB.LI.CI.PP global public
eth4 PRI.VA.TE.IP global cluster_interconnect
Only in OCR: bond0 SCA.NV.IP.IP global public
Only in OCR: bond2 YYY.YY.yy.yy global cluster_interconnect
PRIF-30: Network information in OCR and GPnP profile differs

STEP 10: update OCR configuration using oifcfg
[root@rac1 bin]# ./oifcfg delif -global bond0
[root@rac1 bin]# ./oifcfg delif -global bond2
[root@rac1 bin]# ./oifcfg setif -global eth3/PUB.LI.CI.PP:public
[root@rac1 bin]#./oifcfg setif -global eth4/PRI.VA.TE.IP:cluster_interconnect

STEP 11: Modifing SCAN IP is very simple compare to VIP/Interconnect changes.
[root@rac1 bin]# ./srvctl config scan -- Verifiying existing configuration
SCAN name: node-scan, Network: 1/PUB.LI.CI.PP/255.255.255.0/eth3
SCAN VIP name: scan1, IP: /node-scan/SCA.NV.IP.IP
SCAN VIP name: scan2, IP: /node-scan/SCA.NV.IP.IP
SCAN VIP name: scan3, IP: /node-scan/SCA.NV.IP.IP

[root@rac1 bin]# ./srvctl config scan_listener -- Verifiying existing configuration
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

[root@rac1 bin]# ./srvctl status scan_listener -- Making sure scan listener is down
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is not running
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is not running

[root@rac1 bin]# ./srvctl modify scan -n node-scan -- Modifying SCAN IP's on OCR, this command will pickup the new ip addres from DNS and updates OCR

[root@rac1 bin]# ./srvctl config scan -- Verifying new ip's registered on OCR
SCAN name: node-scan, Network: 1/PUB.LI.CI.PP/255.255.255.0/eth3
SCAN VIP name: scan1, IP: /node-scan/PUB.LI.CI.PP
SCAN VIP name: scan2, IP: /node-scan/PUB.LI.CI.PP
SCAN VIP name: scan3, IP: /node-scan/PUB.LI.CI.PP
Do clean bounce of CRS on all the nodes using
  • crsctl stop crs 
  • crsctl start crs 
  • crsctl enable crs
This concludes 11gR2 clusterware IP address modified

No comments:

Post a Comment