关于Oracle 11G RAC双节点之间存在防火墙导致只能一个节点运行

################关于Oracle 11G RAC双节点之间存在防火墙导致只能一个节点运行 #问题背景： 深信服云平台内部分布式防火墙开启后，所有的主机之间都会存在防火墙。开始不知道分布式防火墙的启动。 #问题现象： 故障节点启动集群时会出现异常，导致集群无法完成启动。alert日志: 2021-10-28 07:30:56.944:  [ohasd(2313)]CRS-2112:The OLR service started on node cmsorcl2. 2021-10-28 07:30:56.955:  [ohasd(2313)]CRS-1301:Oracle High Availability Service started on node cmsorcl2. 2021-10-28 07:30:56.958:  [ohasd(2313)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred 2021-10-28 07:30:57.283:  [/u01/app/11.2.0/grid/bin/oraagent.bin(7198)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/cmsorcl2/agent/ohasd/oraagent_grid/oraagent_grid.log" 2021-10-28 07:31:00.326:  [/u01/app/11.2.0/grid/bin/orarootagent.bin(7202)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).  2021-10-28 07:31:02.503:  [ohasd(2313)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).  2021-10-28 07:31:02.504:  [gpnpd(8934)]CRS-2328:GPNPD started on node cmsorcl2.  2021-10-28 07:31:04.840:  [cssd(10891)]CRS-1713:CSSD daemon is started in clustered mode 2021-10-28 07:31:06.672:  [ohasd(2313)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2021-10-28 07:31:06.673:  [ohasd(2313)]CRS-2769:Unable to failover resource 'ora.diskmon'. 2021-10-28 07:31:30.624:  [cssd(10891)]CRS-1707:Lease acquisition for node cmsorcl2 number 2 completed 2021-10-28 07:31:32.008:  [cssd(10891)]CRS-1605:CSSD voting file is online: /dev/asm-diskc; details in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log. 2021-10-28 07:31:32.048:  [cssd(10891)]CRS-1605:CSSD voting file is online: /dev/asm-diskb; details in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log. 2021-10-28 07:31:32.089:  [cssd(10891)]CRS-1605:CSSD voting file is online: /dev/asm-diska; details in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log. 2021-10-28 15:38:34.926:  [/u01/app/11.2.0/grid/bin/cssdagent(10858)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/cmsorcl2/agent/ohasd/oracssdagent_root/oracssdagent_root.log. 2021-10-28 15:38:34.926:  [cssd(10891)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log 2021-10-28 15:38:34.926:  [cssd(10891)]CRS-1603:CSSD on node cmsorcl2 shutdown by user. 2021-10-28 15:38:40.254:  [ohasd(2313)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'cmsorcl2'. 2021-10-28 15:38:41.668:  [cssd(6922)]CRS-1713:CSSD daemon is started in clustered mode 2021-10-28 15:38:43.435:  [ohasd(2313)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2021-10-28 15:38:43.435:  [ohasd(2313)]CRS-2769:Unable to failover resource 'ora.diskmon'. 2021-10-28 15:38:57.400:  [cssd(6922)]CRS-1707:Lease acquisition for node cmsorcl2 number 2 completed #解决方案： css异常，一般都是节点间的访问异常，在ocssd.log中可查看具体信息，可看到一些私网端口访问异常的信息。 在防火墙中把数据库pub,priv,vip,scan IP之间的限制全部取消。 如果数据库不启用HAIP的话，这时数据库应该就恢复正常了；如果启用了HAIP（默认启用，169段ip），需要把HAIP之间的限制取消掉。如果不取消掉，asm实例无法启动。现象如下： Wed Nov 17 22:05:45 2021 MMON started with pid=20, OS id=6979  Wed Nov 17 22:05:45 2021 MMNL started with pid=21, OS id=6981  lmon registered with NM - instance number 2 (internal mem no 1) Wed Nov 17 22:07:45 2021 PMON (ospid: 6911): terminating the instance due to error 481              <-----问题现象，参考http://blog.itpub.net/29615408/viewspace-1384760/ Wed Nov 17 22:07:45 2021 ORA-1092 : opitsk aborting process Wed Nov 17 22:07:45 2021 System state dump requested by (instance=2, osid=6911 (PMON)), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_diag_6949_20211117220745.trc Dumping diagnostic data in directory=[cdmp_20211117220745], requested by (instance=2, osid=6911 (PMON)), summary=[abnormal instance termination]. Instance terminated by PMON, pid = 6911 Wed Nov 17 22:08:06 2021 NOTE: No asm libraries found in the system 双节点之间HAIP有防火墙的现象提示不是很明显，asm的alert日志如上，手动启asm到nomount时报ORA-03113: end-of-file on communication channel ，集群告警日志和trc文件中的报错都没有明确指出HAIP的问题。PMON (ospid: 6911): terminating the instance due to error 481 的出现原因如下： Case1: link local IP (169.254.x.x) is being used by other adapter/network   Case2: firewall exists between nodes on private network (iptables etc)   Case3: HAIP is up on some nodes but not on all   Case4: HAIP is up on all nodes but some do not have route info 最终关闭了HAIP之间的防火墙策略后数据库恢复！