ORACLE RAC 11.2.0.4 ASM加盘导致集群重启之ASM sga设置过小

    最近，一同事为一2节点的ORACLE RAC 11.2.0.4集群ASM加盘，没有注意到ASM的sga设置过小，加盘reblance时导致集群重启。详细描述如下：     1、问题描述     ORACLE RAC 11.2.0.4集群ASM加盘，没有注意到ASM的sga设置过小，加盘reblance时导致集群重启。     2、问题分析     2.1 首先，接到运维监控告警，某某平台发生严重告警     2.2 登陆相关平台查看数据库和集群ASM日志，发现节点1集群发生重启，相关报错日志如下： Wed Dec 26 17:20:29 2018 ARB0 started with pid=43, OS id=22557  NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O NOTE: Attempting voting file refresh on diskgroup ORADATA Wed Dec 26 17:22:37 2018 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_22557.trc  (incident=208353): ORA-04031: unable to allocate 7072 bytes of shared memory ("shared pool","unknown object","sga heap(1,1)","ges resource ") Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208353/+ASM1_arb0_22557_i208353.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Dec 26 17:22:39 2018 Dumping diagnostic data in directory=[cdmp_20181226172239], requested by (instance=1, osid=22557 (ARB0)), summary=[incident=208353]. Wed Dec 26 17:22:46 2018 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc  (incident=208345): ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable") Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208345/+ASM1_m000_22771_i208345.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc: ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable")     2.3 数据库运维人员注意到，此前的主要操作主要就是给磁盘组加盘，观察ASM告警日志，相关操作记录如下： Wed Dec 26 17:20:16 2018 SQL> ALTER DISKGROUP ORADATA ADD  DISK '/dev/rdisk/VD-DATA21' SIZE 182272M , '/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */  NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (2,20) to disk (/dev/rdisk/VD-DATA21) NOTE: Assigning number (2,21) to disk (/dev/rdisk/VD-DATA22) NOTE: requesting all-instance membership refresh for group=2 NOTE: initializing header on grp 2 disk ORADATA_0020 NOTE: initializing header on grp 2 disk ORADATA_0021 NOTE: requesting all-instance disk validation for group=2 Wed Dec 26 17:20:19 2018 NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance. NOTE: requesting all-instance disk validation for group=2 NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance. NOTE: initiating PST update: grp = 2 Wed Dec 26 17:20:25 2018 GMON updating group 2 at 7 for pid 42, osid 22238 NOTE: PST update grp = 2 completed successfully  NOTE: membership refresh pending for group 2/0x6b4139d2 (ORADATA) GMON querying group 2 at 8 for pid 18, osid 5128 NOTE: cache opening disk 20 of grp 2: ORADATA_0020 path:/dev/rdisk/VD-DATA21 NOTE: cache opening disk 21 of grp 2: ORADATA_0021 path:/dev/rdisk/VD-DATA22 Wed Dec 26 17:20:29 2018 NOTE: Attempting voting file refresh on diskgroup ORADATA GMON querying group 2 at 9 for pid 18, osid 5128 SUCCESS: refreshed membership for 2/0x6b4139d2 (ORADATA) Wed Dec 26 17:20:29 2018 SUCCESS: ALTER DISKGROUP ORADATA ADD  DISK '/dev/rdisk/VD-DATA21' SIZE 182272M , '/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */ NOTE: starting rebalance of group 2/0x6b4139d2 (ORADATA) at power 1 Starting background process ARB0 Wed Dec 26 17:20:29 2018 ARB0 started with pid=43, OS id=22557  NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O NOTE: Attempting voting file refresh on diskgroup ORADATA     2.4 到此，基本上可以确定是ASM加盘，ASM在进行reblance的时候，由于ASM的SGA设置过小导致的集群重启     3、问题处理     调整相关集群的ASM的SGA大小到2G，重新启动集群，ASM开始正常的reblance: Wed Dec 26 17:56:32 2018 NOTE: client SYXPTDB1:SYXPTDB registered, osid 1074, mbr 0x1 Wed Dec 26 18:35:33 2018 ALTER SYSTEM SET memory_max_target='2G' SCOPE=SPFILE SID='*'; ALTER SYSTEM SET memory_target='2G' SCOPE=SPFILE SID='*'; ALTER SYSTEM SET sga_max_size='2G' SCOPE=SPFILE SID='*'; Wed Dec 26 20:50:11 2018 NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=2 Wed Dec 26 20:50:14 2018 NOTE: membership refresh pending for group 2/0x1edaa5d7 (ORADATA) Wed Dec 26 20:50:17 2018 GMON querying group 2 at 7 for pid 18, osid 26306 SUCCESS: refreshed membership for 2/0x1edaa5d7 (ORADATA) NOTE: Attempting voting file refresh on diskgroup ORADATA     到此问题处理完成！