RAC节点hang住, oracle bug导致了cpu过高，无法启动集群隔离

问题描述 1) RAC1节点hang住, oracle bug导致了cpu高，然后集群启动隔离，但是cpu太高，隔离不了 问题原因 1) Bug 21286665 - "Streams AQ: enqueue blocked on low memory" waits with fix 18828868 - superseded (文档 ID 21286665.8) 日志如图：     解决方案 1) 主库shutdown abort，业务调整到从库。正常运行 2) 排查数据库日志，寻找问题 3) 发现问题所在，找到BUG 1) Bug 21286665 - "Streams AQ: enqueue blocked on low memory" waits with fix 18828868   4) 下载补丁修复BUG p22502456_112040_Linux-x86-64.zip 升级opatch 下载optach的最新版本 p6880880_112000_Linux-x86-64.zip、、 补丁升级过程： 升级补丁需要关闭数据库 1。下载补丁上传至$ORACLE_HOME/Opatch目录下，解压备用 2. [oracle@xxx OPatch]$ pwd /home/app/oracle/product/11.2.0/OPatch [oracle@xxx OPatch]$ cd 22502456/ [oracle@xxx 22502456]$ ../OPatch/opatch apply 3. 验证 ../OPatch/opatch lsinventory 4. 或者进入系统应用一下这些补丁，然后查询验证： @?/rdbms/admin/catbundle.sql psu apply select * from dba_registry_history; 2主库升级完补丁以后，要开启数据库startup时报错 ORA-15025: could not open disk "/dev/asm_ssd3" ORA-27041: unable to open file Linux-x86_64 Error: 13: Permission denied Additional information: 9 ORA-15025: could not open disk "/dev/asm_ssd4" ORA-27041: unable to open file Linux-x86_64 Error: 13: Permission denied Additional information: 9 ORA-15025: could not open disk "/dev/asm_ssd5" ORA-27041: unable to open file Linux-x86_64 Error: 13: Permission denied Additional information: 9 SUCCESS: diskgroup SSDDATA was dismounted ERROR: diskgroup SSDDATA was not mounted ORA-15025: could not open disk "/dev/asm_ssd2" ORA-27041: unable to open file Linux-x86_64 Error: 13: Permission denied Additional information: 9 ORA-15025: could not open disk "/dev/asm_ssd3" ORA-27041: unable to open file Linux-x86_64 Error: 13: Permission denied Additional information: 9 ORA-15025: could not open disk "/dev/asm_ssd4" ORA-27041: unable to open file Linux-x86_64 Error: 13: Permission denied Additional information: 9 ORA-15025: could not open disk "/dev/asm_ssd5" ORA-27041: unable to open file 需要使用grid用户登录后用集群命令开启数据库 [grid@orcl1 ~]$ srvctl status database -d orcl Instance orcl1 is running on node orcl1 Instance orcl2 is running on node orcl2 srvctl status instance  -d orcl -i orcl1 srvctl stop/start instance  -d orcl -i orcl1 两个RAC节点都升级完以后，发现DG归档日志报错， SUCCESS: diskgroup SSDDATA was dismounted ERROR: diskgroup SSDDATA was not mounted ORA-00210: cannot open the specified control file ORA-00202: control file: '+SSDDATA/shdbrac/controlfile/current.257.946912997' ORA-17503: ksfdopn:2 Failed to open file +SSDDATA/orcl/controlfile/current.257.946912997 ORA-15001: diskgroup "SSDDATA" does not exist or is not mounted ...skipping...       returning error ORA-16191 ------------------------------------------------------------ PING[ARC2]: Heartbeat failed to connect to standby 'stbdb'. Error is 16191. Error 1017 received logging on to the standby ------------------------------------------------------------ Check that the primary and standby are using a password file and remote_login_passwordfile is set to SHARED or EXCLUSIVE,  and that the SYS password is same in the password files.       returning error ORA-16191 DG关闭数据库， 重启后发生以下报错： ORA-16136 signalled during: alter database recover managed standby database cancel... alter database open read only AUDIT_TRAIL initialization parameter is changed to OS, as DB is NOT compatible for database opened with read-only access Fri Apr 03 21:22:56 2020 Beginning Standby Crash Recovery. Serial Media Recovery started Managed Standby Recovery starting Real Time Apply Media Recovery Log /oradata/stbdb/archivelog/1_63869_946912997.dbf Media Recovery Waiting for thread 2 sequence 29011 Fri Apr 03 21:23:53 2020 Standby Crash Recovery aborted due to error 1013. Errors in file /home/app/oracle/diag/rdbms/stbdb/stbdb/trace/orcl_ora_11251.trc: ORA-01013: user requested cancel of current operation Recovery interrupted! Some recovered datafiles maybe left media fuzzy Media recovery may continue but open resetlogs may fail Completed Standby Crash Recovery. Errors in file /home/app/oracle/diag/rdbms/stbdb/stbdb/trace/stbdb_ora_11251.trc: ORA-10458: standby database requires recovery ORA-01196: file 1 is inconsistent due to a failed media recovery session ORA-01110: data file 1: '/oradata/stbdb/datafile/system.266.946913027' ORA-10458 signalled during: alter database open read only... alter database open Beginning Standby Crash Recovery. Serial Media Recovery started Managed Standby Recovery starting Real Time Apply Media Recovery Log /oradata/stbdb/archivelog/1_63869_946912997.dbf Media Recovery Waiting for thread 2 sequence 29011 Fri Apr 03 21:24:12 2020 Standby Crash Recovery aborted due to error 1013. Errors in file /home/app/oracle/diag/rdbms/stbdb/stbdb/trace/stbdb_ora_11251.trc: ORA-01013: user requested cancel of current operation Recovery interrupted! Some recovered datafiles maybe left media fuzzy Media recovery may continue but open resetlogs may fail Completed Standby Crash Recovery. Errors in file /home/app/oracle/diag/rdbms/stbdb/stbdb/trace/stbdb_ora_11251.trc: ORA-10458: standby database requires recovery ORA-01196: file 1 is inconsistent due to a failed media recovery session ORA-01110: data file 1: '/oradata/stbdb/datafile/system.266.946913027' ORA-10458 signalled during: alter database open... Shutting down instance (abort) License high water mark = 7 USER (ospid: 11251): terminating the instance Fri Apr 03 21:24:16 2020 opiodr aborting process unknown ospid (11280) as a result of ORA-1092 Fri Apr 03 21:24:16 2020 ORA-1092 : opitsk aborting process Instance terminated by USER, pid = 11251 Fri Apr 03 21:24:19 2020 Instance shutdown complete Fri Apr 03 21:25:38 2020 Starting ORACLE instance (normal) 经检查，问题原因在于口令文件有误 将节点1的$ORACLE_HOME/dbs下面的口令文件，copy到节点二和DG后，问题解决。