啟動集群報錯

2022-05-17 21:27:19.909 [CSSDMONITOR(31896)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 31896
2022-05-17 21:27:19.961 [CSSDAGENT(31902)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 31902
2022-05-17 21:27:21.376 [CSSDAGENT(31995)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 31995
2022-05-17 21:27:26.349 [OCSSD(32014)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 32014
2022-05-17 21:27:27.450 [OCSSD(32014)]CRS-1713: CSSD daemon is started in hub mode
2022-05-17 21:27:27.634 [OCSSD(32014)]CRS-1637: Unable to locate configured voting file with ID 6f82aa99-0aad4f63-bf4afada-327bc34e; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc 
2022-05-17 21:27:27.635 [OCSSD(32014)]CRS-1637: Unable to locate configured voting file with ID 13fa2481-284d4f7b-bfaf8e77-2dd203ea; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-17 21:27:27.635 [OCSSD(32014)]CRS-1637: Unable to locate configured voting file with ID 1381ba73-db3d4fb7-bf76c4cf-93143522; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-17 21:27:27.636 [OCSSD(32014)]CRS-1637: Unable to locate configured voting file with ID 71c65157-fa374fcf-bfe8a8d4-3819f33c; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-17 21:27:27.636 [OCSSD(32014)]CRS-1705: Found 1 configured voting files but 3 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-17 21:27:28.638 [OCSSD(32014)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-17 21:27:28.677 [OCSSD(32014)]CRS-1652: Starting clean up of CRSD resources.
2022-05-17 21:27:28.684 [OCSSD(32014)]CRS-1653: The clean up of the CRSD resources failed.
2022-05-17T21:27:30.709540+08:00
Errors in file /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc  (incident=345):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/19c/grid/diag/crs/nxdbcs01/crs/incident/incdir_345/ocssd_i345.trc

表決盤掛載不了

[root@nxdbcs01 ~]# crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@nxdbcs01 ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
         Device/File Name         :       +OCR

2022-05-07 00:26:16.264 :   SKGFD:2104772352: Handle 0x7fc2240f8370 from lib :UFS:: for disk :/dev/asm_data01:

2022-05-07 00:26:16.266 :    CLSF:2104772352:  Warning: Voting disk: /dev/asm_data01 is Hard mounted

2022-05-07 00:26:16.266 :   SKGFD:2104772352: Lib :UFS:: closing handle 0x7fc2240f8370 for disk :/dev/asm_data01:

2022-05-07 00:26:16.266 :    CSSD:2104772352: [     INFO] clssnmvDiskVerify: discovered a potential voting file
2022-05-07 00:26:16.267 :   SKGFD:2104772352: Handle 0x7fc2240f7ed0 from lib :UFS:: for disk :/dev/asm_data01:

2022-05-07 00:26:16.267 :    CLSF:2104772352: Opened hdl:0x7fc2240f7e80 for dev:/dev/asm_data01:
2022-05-07 00:26:16.277 :    CSSD:2104772352: [     INFO] clssnmvDiskCreate: name /dev/asm_data01 blocksz 512
2022-05-07 00:26:16.279 :    CSSD:2104772352: [     INFO] clssnmvDiskCreate: clguidoff_volinfo =44, siteoff_volinfo = 77
2022-05-07 00:26:16.280 :    CSSD:2104772352: [     INFO] clssnmvDiskCreate: site guid during discovery = 00112233445566778899aabbccddeeff
2022-05-07 00:26:16.280 :    CSSD:2104772352: [     INFO] clssnmvDiskVerify: Successful discovery for disk /dev/asm_data01, UID 6ec58cde-24bb4fd8-bfa91e25-5e44661a, SID 00112233-44556677-8899aabb-ccddeeff, Pending CIN 0:1648651938:0, Committed CIN 0:1648651938:0
2022-05-07 00:26:16.280 :    CLSF:2104772352: Closing handle:0x7fc2240f7e80
2022-05-07 00:26:16.281 :   SKGFD:2104772352: Lib :UFS:: closing handle 0x7fc2240f7ed0 for disk :/dev/asm_data01:

2022-05-07 00:26:16.281 :    CSSD:2104772352: [     INFO] clssnmvDiskVerify: Successful discovery of 1 disks
2022-05-07 00:26:16.281 :    CSSD:2104772352: [     INFO] clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvCopyCFG: cinhdrsize = 60 , nmcfg size = 60,
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmCompleteVFDiscovery: Completing voting file discovery
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvFindConfiguredVFs: Changing state for vdisk 0x7fc2240fa3b0, /dev/asm_data01to pending config
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvDiskStateChange: state from discovered to pending disk /dev/asm_data01
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvConfigureVFs: Changing state for vdisk 0x7fc2240fa3b0, /dev/asm_data01to configured
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvDiskStateChange: state from pending to configured disk /dev/asm_data01
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmVFMajority: In site 00112233-44556677-8899aabb-ccddeeff : Insufficient voting files found, found 1 of 5 configured, needed 3 vf's
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvVerifyCommittedConfigVFs : No Majority for site 00112233-44556677-8899aabb-ccddeeff, configured VFs 5  Acessible VFs 1, Min needed 3 
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvVerifyCommittedConfigVFs: configured Sites = 1, Incative sites = 1, Mininum Sites required = 1
2022-05-07 00:26:16.282 :    CSSD:2104772352: [     INFO] clssnmvVerifyCommittedConfigVFs: Insufficient voting files found, found 1 of 5 configured, needed 3 voting files
2022-05-07 00:26:16.288 :    CSSD:2104772352: [     INFO] (:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 1, id 6f82aa99-0aad4f63-bf4afada-327bc34e not found
2022-05-07 00:26:16.297 :    CSSD:2104772352: [     INFO] (:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 2, id 13fa2481-284d4f7b-bfaf8e77-2dd203ea not found
2022-05-07 00:26:16.303 :    CSSD:2104772352: [     INFO] (:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 3, id 1381ba73-db3d4fb7-bf76c4cf-93143522 not found
2022-05-07 00:26:16.309 :    CSSD:2104772352: [     INFO] (:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 4, id 71c65157-fa374fcf-bfe8a8d4-3819f33c not found
2022-05-07 00:26:16.315 :    CSSD:2104772352: ASSERT clssnm1.c 5095
2022-05-07 00:26:16.315 :    CSSD:2104772352: (:CSSNM00021:)clssnmCheckQuorum: Found 1 voting files, but 3 are required.  Terminating due to insufficient configured voting files

2022-05-16 19:55:18.702 [OCSSD(3797)]CRS-1637: Unable to locate configured voting file with ID 6f82aa99-0aad4f63-bf4afada-327bc34e; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-16 19:55:18.702 [OCSSD(3797)]CRS-1637: Unable to locate configured voting file with ID 13fa2481-284d4f7b-bfaf8e77-2dd203ea; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-16 19:55:18.703 [OCSSD(3797)]CRS-1637: Unable to locate configured voting file with ID 1381ba73-db3d4fb7-bf76c4cf-93143522; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-16 19:55:18.703 [OCSSD(3797)]CRS-1637: Unable to locate configured voting file with ID 71c65157-fa374fcf-bfe8a8d4-3819f33c; details at (:CSSNM00020:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-16 19:55:18.704 [OCSSD(3797)]CRS-1705: Found 1 configured voting files but 3 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-16 19:55:19.705 [OCSSD(3797)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc
2022-05-16 19:55:19.744 [OCSSD(3797)]CRS-1652: Starting clean up of CRSD resources.
2022-05-16 19:55:19.750 [OCSSD(3797)]CRS-1653: The clean up of the CRSD resources failed.
2022-05-16 19:55:21.767 [OCSSD(3797)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 3797 experienced fatal signal or exception code 6.
2022-05-16T19:55:21.799860+08:00
Errors in file /u01/app/19c/grid/diag/crs/nxdbcs01/crs/trace/ocssd.trc  (incident=25):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []

集群日志顯示服務加載ocr所在磁盤組

檢查udev.rules,發現主機重啟后盤符發生了變化，因為是單快盤分區做的，uuid相同，沒法使用uuid，那么只能臨時修改rules盤符。
再次啟動
此時asm可以啟動，但是顯示沒有磁盤組，檢查asmdisk_string參數發現沒有問題。
show parameter spfile可以看到spfile位置，create pfile報錯
檢查asm日志

OTE: Diskgroup used for OCR backup is:OCR
NOTE: Diskgroups listed in ASM_DISKGROUP are
         DATA01

2022-05-18T07:59:57.606087+08:00
NOTE: ASM instance +ASM1 is discoverable by local clients on node nxdbcs01
2022-05-18T07:59:57.623534+08:00
NOTE: cache registered group DATA01 1/0x00646E62
NOTE: cache began mount (first) of group DATA01 1/0x00646E62
NOTE: cache registered group OCR 2/0x00746E63
NOTE: cache began mount (first) of group OCR 2/0x00746E63
WARNING: detected duplicate paths to the same disk:
  '/dev/sdd1' and
  '/dev/asm_data01'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdd1' and
  '/dev/asm_data01'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc7' and
  '/dev/asm_ocr05'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc7' and
  '/dev/asm_ocr05'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc6' and
  '/dev/asm_ocr04'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc6' and
  '/dev/asm_ocr04'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc5' and
  '/dev/asm_ocr03'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc5' and
  '/dev/asm_ocr03'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc2' and
  '/dev/asm_ocr02'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc2' and
  '/dev/asm_ocr02'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc1' and
  '/dev/asm_ocr01'
More trace information dumped to '/u01/app/19c/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26839.trc'
WARNING: detected duplicate paths to the same disk:
  '/dev/sdc1' and
  '/dev/asm_ocr01'

重復盤符問題，很常見，什么原因導致的呢，

臨時解決辦法，先去掉一個盤符唄
刪除/etc/udev/rules/oracle.rules

/sbin/udevadm control --reload-rules
/sbin/udevadm trigger --type=devices --action=change

alter diskgroup all mount;

成功

臨時解決辦法可行，具體原因以后再說，睡覺