最近新裝一套19c兩節點RAC，客戶要求改一下私網IP，修改過程相比11g還是有很大的不一樣，記錄一下。

背景：

（1）私網IP由192.168.1.71/72改為192.168.1.2/3
（2）涉及subnet變化
192.168.1.71/72的subnet是192.168.1.64
192.168.1.2/3的subnet是192.168.1.0
（3）網卡名無需修改
參考文檔：How to Modify Private Network Information in Oracle Clusterware (Doc ID 283684.1)

1. 備份GPNP profile

[grid@xyjzxdb01 ~]$ cd $ORACLE_HOME/gpnp/profiles/peer
[grid@xyjzxdb01 peer]$ cp profile.xml profile.xml.bak

2. 確保所有節點crs為啟動狀態

[grid@xyjzxdb01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

3. 獲取當前網絡接口信息

[grid@xyjzxdb01 ~]$ oifcfg getif
team0  172.19.245.64  global  public
team1  192.168.1.64  global  cluster_interconnect,asm

4. 修改網絡接口信息，在一個節點執行

[grid@xyjzxdb01 ~]$ oifcfg setif -global team1/192.168.1.0:cluster_interconnect,asm
[grid@xyjzxdb01 ~]$ oifcfg getif
team0  172.19.245.64  global  public
team1  192.168.1.64  global  cluster_interconnect,asm
team1  192.168.1.0  global  cluster_interconnect,asm

NOTE：

• 即使接口不可用，也可以使用-global選項，但是如果接口不可用，則不能使用-node選項，否則會導致節點退出。
• oifcfg iflist --列出網絡接口和子網地址，即使Oracle Clusterware沒有運行，也可以運行該命令

5. 停所有節點crs

[root@xyjzxdb01 ~]# crsctl stop crs

6. 修改OS網絡配置，確保修改后的接口在所有節點上都可用

[root@xyjzxdb01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.19.245.71  xyjzxdb01
172.19.245.72  xyjzxdb02

172.19.245.73  xyjzxdb01-vip
172.19.245.74  xyjzxdb02-vip

172.19.245.75  xyjzxdb-scan

192.168.1.2  xyjzxdb01-priv
192.168.1.3  xyjzxdb02-priv

[root@xyjzxdb01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-team1
TEAM_CONFIG='{"runner":{"name":"activebackup"},"link_watch":{"name":"ethtool"}}'
NAME=team1
DEVICE=team1
ONBOOT=yes
DEVICETYPE=Team
IPADDR=192.168.1.2
PREFIX=26
MTU=1500

[root@xyjzxdb02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-team1
TEAM_CONFIG='{"runner":{"name":"activebackup"},"link_watch":{"name":"ethtool"}}'
NAME=team1
DEVICE=team1
ONBOOT=yes
DEVICETYPE=Team
IPADDR=192.168.1.3
PREFIX=26
MTU=1500

重啟網卡

[root@xyjzxdb01 ~]# systemctl restart network

確認所有節點修改成功

[root@xyjzxdb01 ~]# ifconfig -a
......
team1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.2  netmask 255.255.255.192  broadcast 192.168.1.63
        inet6 fe80::250:56ff:fe9a:a3a0  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:9a:a3:a0  txqueuelen 1000  (Ethernet)
        RX packets 7  bytes 374 (374.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11  bytes 782 (782.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@xyjzxdb02 ~]# ifconfig -a
......
team1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.3  netmask 255.255.255.192  broadcast 192.168.1.63
        inet6 fe80::250:56ff:fe9a:f1d8  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:9a:f1:d8  txqueuelen 1000  (Ethernet)
        RX packets 2  bytes 122 (122.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11  bytes 830 (830.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

確認相互能ping通

[root@xyjzxdb01 ~]# ping 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=0.334 ms
64 bytes from 192.168.1.3: icmp_seq=2 ttl=64 time=0.307 ms
64 bytes from 192.168.1.3: icmp_seq=3 ttl=64 time=0.557 ms
[root@xyjzxdb02 ~]# ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.293 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.427 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=0.420 ms

查看修改后的網絡接口和子網地址

[root@xyjzxdb01 ~]# oifcfg iflist
team1  192.168.1.0
team0  172.19.245.64
[root@xyjzxdb02 ~]# oifcfg iflist
team1  192.168.1.0
team0  172.19.245.64

7. 啟動所有節點crs

[root@xyjzxdb02 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.crf
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.crsd
      1        ONLINE  OFFLINE       xyjzxdb02                STABLE
ora.cssd
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.ctssd
      1        ONLINE  ONLINE       xyjzxdb02                ACTIVE:0,STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.evmd
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.gipcd
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.gpnpd
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.mdnsd
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.storage
      1        ONLINE  ONLINE       xyjzxdb02                STARTING   ---一直STARTING，無法ONLINE
--------------------------------------------------------------------------------
--只能第一個節點啟動，第二個節點始終起不來，ora.storage一直是STARTING狀態，ASM實例也沒有啟動

8. 刪除原來的網絡接口后再嘗試啟動第二個節點

[root@xyjzxdb01 ~]# oifcfg delif -global team1/192.168.1.64
[root@xyjzxdb01 ~]# oifcfg getif
team0  172.19.245.64  global  public
team1  192.168.1.0  global  cluster_interconnect,asm

第二個節點仍然無法啟動

9. 發現正常的1節點，ora.ASMNET1LSNR_ASM.lsnr和ora.asmnet1.asmnetwork狀態都不正常，懷疑跟它有關

[root@xyjzxdb01 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       xyjzxdb01                STABLE
ora.chad
               ONLINE  ONLINE       xyjzxdb01                STABLE
ora.net1.network
               ONLINE  ONLINE       xyjzxdb01                STABLE
ora.ons
               ONLINE  ONLINE       xyjzxdb01                STABLE
ora.proxy_advm
               OFFLINE OFFLINE      xyjzxdb01                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  OFFLINE                               STABLE   ----正常的1節點也是OFFLINE狀態
      2        ONLINE  OFFLINE                               STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.FRA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.OCRDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                Started,STABLE
      2        ONLINE  OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  OFFLINE                               STABLE   ----正常的1節點也是OFFLINE狀態
      2        ONLINE  OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.qosmserver
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.xyjzxdb.db
      1        ONLINE  ONLINE       xyjzxdb01                Open,HOME=/oracle/ap
                                                             p/oracle/product/19c
                                                             /dbhome_1,STABLE
      2        ONLINE  OFFLINE                               STABLE
ora.xyjzxdb01.vip
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.xyjzxdb02.vip
      1        ONLINE  INTERMEDIATE xyjzxdb01                FAILED OVER,STABLE
--------------------------------------------------------------------------------

10. 檢查各種配置

[root@xyjzxdb01 ~]# srvctl config asmnetwork
ASM network 1 exists
**Subnet IPv4: 192.168.1.64//**
Subnet IPv6:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:

[root@xyjzxdb01 ~]# srvctl config listener -asmlistener
Name: ASMNET1LSNR_ASM
Type: ASM Listener
Owner: grid
**Subnet: 192.168.1.64**
Home: <CRS home>
End points: TCP:1525
Listener is enabled.
Listener is individually enabled on nodes: 
Listener is individually disabled on nodes: 

asm listener里的Subnet配置仍然顯示為192.168.1.64，確認是它的問題

11. 添加新的asmlistener，并刪除原來的

[grid@xyjzxdb01 ~]$ srvctl add listener -asmlistener -l ASMNEWLSNR_ASM -subnet 192.168.1.0
[grid@xyjzxdb01 ~]$ srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force

[grid@xyjzxdb01 ~]$ srvctl config listener -asmlistener
Name: ASMNEWLSNR_ASM
Type: ASM Listener
Owner: grid
Subnet: 192.168.1.0     --已修改成功，原來為192.168.1.64
Home: <CRS home>
End points: TCP:1526
Listener is enabled.
Listener is individually enabled on nodes: 
Listener is individually disabled on nodes: 
[grid@xyjzxdb01 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +OCRDG/orapwASM
Backup of Password file: +OCRDG/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNEWLSNR_ASM   --已修改成功，原來為ASMNET1LSNR_ASM

確認已經修改成功

12. 在19c中，有一個名為ora.asmnet1的新資源，也需要更新

[root@xyjzxdb01 ~]# srvctl remove asmnetwork -netnum 1
[root@xyjzxdb01 ~]# srvctl add asmnetwork -netnum 1 -subnet 192.168.1.0

13. 完成后，啟動正常

[root@xyjzxdb01 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       xyjzxdb01                STABLE
               ONLINE  ONLINE       xyjzxdb02                STABLE
ora.chad
               ONLINE  ONLINE       xyjzxdb01                STABLE
               ONLINE  ONLINE       xyjzxdb02                STABLE
ora.net1.network
               ONLINE  ONLINE       xyjzxdb01                STABLE
               ONLINE  ONLINE       xyjzxdb02                STABLE
ora.ons
               ONLINE  ONLINE       xyjzxdb01                STABLE
               ONLINE  ONLINE       xyjzxdb02                STABLE
ora.proxy_advm
               OFFLINE OFFLINE      xyjzxdb01                STABLE
               OFFLINE OFFLINE      xyjzxdb02                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNEWLSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
      2        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
      2        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.FRA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
      2        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.OCRDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
      2        ONLINE  ONLINE       xyjzxdb02                STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       xyjzxdb01                Started,STABLE
      2        ONLINE  ONLINE       xyjzxdb02                Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        OFFLINE OFFLINE                               STABLE     --兩個節點都變成了OFFLINE，但是對數據庫沒有影響
      2        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.qosmserver
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.xyjzxdb.db
      1        ONLINE  ONLINE       xyjzxdb01                Open,HOME=/oracle/ap  --2個節點db都ONLINE
                                                             p/oracle/product/19c
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       xyjzxdb02                Open,HOME=/oracle/ap
                                                             p/oracle/product/19c
                                                             /dbhome_1,STABLE
ora.xyjzxdb01.vip
      1        ONLINE  ONLINE       xyjzxdb01                STABLE
ora.xyjzxdb02.vip
      1        ONLINE  ONLINE       xyjzxdb02                STABLE
--------------------------------------------------------------------------------

• 兩個節點的ora.asmnet1.asmnetwork都變成了OFFLINE，但是db都已經ONLINE，對數據庫應該沒有影響
• 本來打算查下啟動的命令，但既然沒有用到，索性不啟動了（啟動命令：srvctl start asmnetwork -netnum 1）