本文与我之前写过的 同步工具导致Oracle ASM实例审计日志暴增场景类似。 适用范围 Oracle Database 11g+
问题概述 Oracle RAC 节点2数据库软件基础目录/u01使用率超过99%,并且在持续增加,应用连接数据库异常。
问题原因 节点2 数据库审计日志目录审计日志产生大量审计日志。根本原因是配置同步工具DRS后调整了数据库归档备份和删除策略。 【知识点】什么是DRS?
数据复制服务(Data Replication Service,简称DRS)是一种易用、稳定、高效、用于数据库实时迁移、数据库实时同步的云服务。 实时同步是指在不同的系统之间,将数据通过同步技术从一个数据源拷贝到其他数据库,并保持一致,实现关键业务的数据实时流动。 实时同步不同于迁移,迁移是以整体数据库搬迁为目的,而实时同步是维持不同业务之间的数据持续性流动。
解决方案 1、应急处理方式。备份并清理数据库审计日志。 2、/u01基础软件目录扩容。有100g扩容至300G。 3、调整数据库归档备份和删除策略。 分析过程: 1、检查目录使用率。 这套环境数据库服务器是AIX 7.1
# df -g Filesystem GB blocks Free %Used Iused %Iused Mounted on /dev/hdoracle 100.00 0.54 99% 10749 8% /u01 /dev/hd2 6.00 3.55 41% 51985 6% /usr /dev/hd9var 4.00 0.85 79% 6616 4% /var /dev/hd3 10.00 9.91 1% 886 1% /tmp /dev/hd1 2.00 1.98 1% 142 1% /home /dev/hdarch 200 80 60% 9025 2% /arch /dev/lidump 2.00 2.00 1% 4 1% #
/u01目录是Oracle数据库软件基础目录,数据库软件和审计日志存储在该目录,该目录使用率已超过99%,并且持续增长。/arch是共享目录,用来存储数据库归档日志当前使用时60%。 2、检查审计日志
more /u01/app/oracle/admin/xfdb/adump/xfdb2_ora_17105260_20250704112601877496143795.aud Audit file /u01/app/oracle/admin/xfdb/adump/xfdb2_ora_17105260_20250704112601877496143795.aud Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1System name: AIX Node name: host02Release: 1Version: 7Machine: host02Instance name: xfdb2Redo thread mounted by this instance: 2Oracle process number: 450Unix process pid: 105260, image: oracle@host02 (TNS V1-V3) Fri Jul 4 11:26:01 2025 +08:00LENGTH : '155'ACTION :[7] 'CONNECT'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '318'ACTION :[168] 'select decode(status, 'OPEN', 1, 0), decode(archiver, 'FAILED', 1, 0), decode(database_status, 'SUSPENDED', 1, 0) into :status, :archstuck, :dbsuspended from v$instance'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '226'ACTION :[77] 'select decode(open_mode, 'READ WRITE', 1, 0) into :read_write from v$database'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '221'ACTION :[72] 'select value into :vcomp_txt from v$parameter where name = 'compatible''DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '615'ACTION :[465] 'declare dot1st number; dot2nd number; dot3rd number; comptxt varchar2(255) := :vcomp_txt; begin comptxt := comptxt || '.0.0'; dot1st := instr(comptxt, '.', 1, 1); dot2nd := instr(comptxt, '.', 1, 2); dot3rd := instr(comptxt, '.', 1, 3); comptxt := lpad(substr(comptxt, 1, dot1st - 1), 2, '0') || lpad(substr(comptxt, dot1st + 1, dot2nd - dot1st - 1), 2, '0') || lpad(substr(comptxt, dot2nd + 1, dot3rd - dot2nd - 1), 2, '0');:vcomp_ub4 := to_number(comptxt); end;'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '693'ACTION :[543] 'begin :vsn_min := dbms_backup_restore.protocol_version_number_min; :rel_min := dbms_backup_restore.protocol_release_number_min; :upd_min := dbms_backup_restore.protocol_update_number_min; :cmp_min := 0; :vsn_max := dbms_backup_restore.protocol_version_number_max; :rel_max := dbms_backup_restore.protocol_release_number_max; :upd_max := dbms_backup_restore.protocol_update_number_max; :cmp_max := 0; :pkg_vsn := to_char(:vsn_max, 'fm00')||'.'|| to_char(:rel_max, 'fm00')||'.'|| to_char(:upd_max, 'fm00')||'.'|| to_char(:cmp_max, 'fm00'); end; 'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '425'ACTION :[275] 'begin :cmp_min := dbms_backup_restore.protocol_component_number_min; :cmp_max := dbms_backup_restore.protocol_component_number_max; :pkg_vsn := to_char(:vsn_max, 'fm00')||'.'|| to_char(:rel_max, 'fm00')||'.'|| to_char(:upd_max, 'fm00')||'.'|| to_char(:cmp_max, 'fm00'); end; 'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '386'ACTION :[236] 'declare vsn varchar2(20); begin vsn := dbms_rcvman.getPackageVersion; :pkg_vsn := vsn; if vsn is not null then :pkg_vsnub4 := to_number(substr(vsn,1,2) || substr(vsn,4,2) || substr(vsn,7,2) || nvl(substr(vsn,10,2),'00')); end if; end; 'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '386'ACTION :[236] 'declare vsn varchar2(20); begin vsn := dbms_rcvman.getPackageVersion; :pkg_vsn := vsn; if vsn is not null then :pkg_vsnub4 := to_number(substr(vsn,1,2) || substr(vsn,4,2) || substr(vsn,7,2) || nvl(substr(vsn,10,2),'00')); end if; end; 'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '386'ACTION :[236] 'declare vsn varchar2(20); begin vsn := dbms_rcvman.getPackageVersion; :pkg_vsn := vsn; if vsn is not null then :pkg_vsnub4 := to_number(substr(vsn,1,2) || substr(vsn,4,2) || substr(vsn,7,2) || nvl(substr(vsn,10,2),'00')); end if; end; 'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '386'ACTION :[236] 'declare vsn varchar2(20); begin vsn := dbms_rcvman.getPackageVersion; :pkg_vsn := vsn; if vsn is not null then :pkg_vsnub4 := to_number(substr(vsn,1,2) || substr(vsn,4,2) || substr(vsn,7,2) || nvl(substr(vsn,10,2),'00')); end if; end; 'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'Fri Jul 4 11:26:02 2025 +08:00LENGTH : '386'ACTION :[236] 'declare vsn varchar2(20); begin vsn := dbms_rcvman.getPackageVersion; :pkg_vsn := vsn; if vsn is not null then :pkg_vsnub4 := to_number(substr(vsn,1,2) || substr(vsn,4,2) || substr(vsn,7,2) || nvl(substr(vsn,10,2),'00')); end if; end; 'DATABASE USER:[1] '/'PRIVILEGE :[6] 'SYSDBA'CLIENT USER:[6] 'oracle'CLIENT TERMINAL:[0] ''STATUS:[1] '0'DBID:[10] '468337914'...
从审计日志看,是Oracle用户以sysdba角色基于操作系统认证执行了查询 v$instance,v$database,v$ v$parameter等动态视图,同时调用了dbms_rcvman,dbms_backup_restore等rman备份相关的dbms包。通过查看审计日志产生的时间,是每5分钟产生4个(有规律)。 【说明】这是Oracle数据库正常的审计行为。 2、检查定时任务
# crontab -l */5 * * * * /home/oracle/scripts/delarch.sh
定时任务中每5分钟执行一次清理归档。这个与审计日志产生的频率一致。 3、检查清理归档脚本
$RMAN target / nocatalog log $RMAN_LOG_FILE append << EOF
crosscheck archivelog all;
RUN{
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' rate 100M;
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE' rate 100M;
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE' rate 100M;
ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE' rate 1200M;
SEND 'NB_ORA_POLICY=$ARCH_POLICY,NB_ORA_CLIENT=host02';BACKUP
FILESPERSET 5
FORMAT 'al_%s_%p_%t'
ARCHIVELOG ALL NOT BACKED UP; DELETE NOPROMPT ARCHIVELOG ALL COMPLETED BEFORE 'SYSDATE -2';RELEASE CHANNEL ch00;RELEASE CHANNEL ch01;RELEASE CHANNEL ch02;RELEASE CHANNEL ch03;
}
EOF
...
RMAN_PROCESS=`ps -ef |grep rman | grep -v grep`
if [ "${RMAN_PROCESS}" == "" ]; then
if [ ${NUM} -gt 60 ]; then
echo $DATE": Clear all primary archive log use rman with ORACLE_SID ${1}"
CLEAR_PRIMARY_ARCH_USE_RMAN ${1} ALL
else
IS_NEED_RMAN=`find ${2} -maxdepth 1 -name '*.dbf' -type f -ctime +${KEEP_DAYS} | wc -l`
if [ ${IS_NEED_RMAN} -gt 0 ]; then
CLEAR_PRIMARY_ARCH_USE_RMAN ${1}
else
echo $DATE": No need to clean archive log use rman with ORACLE_SID ${1}"
fi
fi
else
echo $DATE"
fi
...
通过脚本逻辑可以看到,清理归档日志前,先通过NBU对归档日志进行备份,/arch超过60%触发清理归档日志任务。 业务高峰时产生的归档日志会超过归档目录/arch使用率的60%,定时任务每5分钟执行一下,会触发清理归档和NBU备份归档的任务,审计日志记录的时NBU备份归档日志的动作。 -thd end-
------------------------------------------------------------------------------------------------------------------------------------
**MyBologs:**
https://www.myhfxf.com
https://www.xiaofeihuangfu.com
CSDN: https://blog.csdn.net/xfhuangfu
ITPUB: https://blog.itpub.net/28373936/
微信公众号:xfhuangfu
------------------------------------------------------------------------------------------------------------------------------------
