RC3: Archive log rejected (thread 1 sequence 30452) at host 'testadg' ORA-16401

1 数据库报错，30496归档到ADG报错 Wed Jun 02 15:16:18 2021 ARC3: Archive log rejected (thread 1 sequence 30452) at host 'testadg' FAL[server, ARC3]: FAL archive failed, see trace file. ARCH: FAL archive failed. Archiver continuing ORACLE Instance primary1 - Archival Error. Archiver continuing. 2  在15:18时查看ADG 应用日志情况，发现实例1目前已经应用到30459， 即证明30452redo日志已经应用完成。 从此处可以认为此错误可以忽略。 select process,CLIENT_PROCESS,thread#,sequence#,status,DELAY_MINS from v$managed_standby;SQL>  3 查看ADG历史应用日志情况，发现在报错时间点，有5分钟的延迟 SQL>   SELECT * FROM V$STANDBY_EVENT_HISTOGRAM order by 5 ; NAME          TIME UNIT                COUNT LAST_TIME_UPDATED ------------------ -------------- ---------- ----------------------------- apply lag        0 seconds          19713768 06/02/2021 14:38:58 apply lag        1 seconds           1743900 06/02/2021 14:39:00 apply lag        2 seconds             25809 06/02/2021 14:39:02 apply lag        3 seconds             19377 06/02/2021 14:39:04 apply lag        4 seconds             18169 06/02/2021 14:39:06 apply lag        5 seconds             15032 06/02/2021 14:39:07 apply lag        6 seconds             10071 06/02/2021 14:39:08 apply lag        7 seconds              6186 06/02/2021 14:39:11 apply lag        8 seconds              3760 06/02/2021 14:39:12 apply lag        9 seconds              2437 06/02/2021 14:39:13 apply lag       10 seconds              1706 06/02/2021 14:39:15 apply lag       11 seconds              1233 06/02/2021 14:39:16 apply lag       12 seconds               957 06/02/2021 14:39:18 apply lag       13 seconds               821 06/02/2021 14:39:19 apply lag       14 seconds               688 06/02/2021 14:39:20 apply lag       15 seconds               643 06/02/2021 14:39:21 apply lag       16 seconds               550 06/02/2021 14:39:24 apply lag       17 seconds               518 06/02/2021 14:39:25 apply lag       18 seconds               520 06/02/2021 14:39:26 apply lag       19 seconds               524 06/02/2021 14:39:29 apply lag       20 seconds               551 06/02/2021 14:39:30 apply lag       21 seconds               600 06/02/2021 14:39:31 apply lag       22 seconds               541 06/02/2021 14:39:33 apply lag       23 seconds               548 06/02/2021 14:39:34 apply lag       24 seconds               553 06/02/2021 14:39:35 apply lag       25 seconds               553 06/02/2021 14:39:36 apply lag       26 seconds               568 06/02/2021 14:39:37 apply lag       27 seconds               577 06/02/2021 14:39:38 apply lag       28 seconds               543 06/02/2021 14:39:39 apply lag       29 seconds               583 06/02/2021 14:39:43 apply lag       30 seconds               559 06/02/2021 14:39:44 apply lag       31 seconds               606 06/02/2021 14:39:46 apply lag       32 seconds               620 06/02/2021 14:39:47 apply lag       33 seconds               596 06/02/2021 14:39:48 apply lag       34 seconds               613 06/02/2021 14:39:50 apply lag       35 seconds               612 06/02/2021 14:39:51 apply lag       36 seconds               563 06/02/2021 14:39:52 apply lag       37 seconds               551 06/02/2021 14:39:53 apply lag       38 seconds               570 06/02/2021 14:39:54 apply lag       39 seconds               525 06/02/2021 14:39:56 apply lag       40 seconds               536 06/02/2021 14:39:57 apply lag       41 seconds               499 06/02/2021 14:39:59 apply lag       42 seconds               583 06/02/2021 14:40:00 apply lag       43 seconds               568 06/02/2021 14:40:02 apply lag       44 seconds               560 06/02/2021 14:40:03 apply lag       45 seconds               584 06/02/2021 14:40:04 apply lag       46 seconds               590 06/02/2021 14:40:06 apply lag       47 seconds               580 06/02/2021 14:40:07 apply lag       48 seconds               607 06/02/2021 14:40:08 apply lag       49 seconds               647 06/02/2021 14:40:09 apply lag       50 seconds               716 06/02/2021 14:40:11 apply lag       51 seconds               698 06/02/2021 14:40:12 apply lag       52 seconds               678 06/02/2021 14:40:13 apply lag       53 seconds               690 06/02/2021 14:40:15 apply lag       54 seconds               678 06/02/2021 14:40:16 apply lag       55 seconds               751 06/02/2021 14:40:17 apply lag       56 seconds               729 06/02/2021 14:40:18 apply lag       57 seconds               731 06/02/2021 14:40:20 apply lag       58 seconds               762 06/02/2021 14:40:21 apply lag       59 seconds               683 06/02/2021 14:40:23 apply lag        5 minutes              1125 06/02/2021 14:59:23 apply lag        4 minutes              5964 06/02/2021 15:46:23 apply lag        2 minutes             44187 06/02/2021 15:53:35 apply lag        3 minutes             26792 06/02/2021 15:55:26 123 rows selected. 4 查看数据库生成的trace文件 -rw-r----- 1 oracle asmadmin     42035 Jun  2 16:05 primary1_lgwr_48970.trm -rw-r----- 1 oracle asmadmin    152518 Jun  2 16:05 primary1_lgwr_48970.trc -rw-r----- 1 oracle asmadmin      2032 Jun  2 16:06 primary1_arc0_49532.trm -rw-r----- 1 oracle asmadmin     13728 Jun  2 16:06 primary1_arc0_49532.trc -rw-r----- 1 oracle asmadmin      1538 Jun  2 16:06 primary1_arc1_49534.trm -rw-r----- 1 oracle asmadmin      9047 Jun  2 16:06 primary1_arc1_49534.trc -rw-r----- 1 oracle asmadmin  48311632 Jun  2 16:06 alert_primary1.log 5 查看trace文件的内容，发现报ORA-16401错误 [oracle@primarydb1 trace]$ more primary1_arc1_49534.trc Trace file /u01/app/oracle/diag/rdbms/primary/primary1/trace/primary1_arc1_49534.trc Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1 System name:    Linux Node name:      primarydb1 Release:        3.10.0-862.el7.x86_64 Version:        #1 SMP Wed Mar 21 18:14:51 EDT 2018 Machine:        x86_64 Instance name: primary1 Redo thread mounted by this instance: 1 Oracle process number: 51 Unix process pid: 49534, image: oracle@primarydb1 (ARC1) *** 2021-05-02 00:05:00.093 *** SESSION ID:(1633.21) 2021-05-02 00:05:00.093 *** CLIENT ID:() 2021-05-02 00:05:00.093 *** SERVICE NAME:(SYS$BACKGROUND) 2021-05-02 00:05:00.093 *** MODULE NAME:() 2021-05-02 00:05:00.093 *** ACTION NAME:() 2021-05-02 00:05:00.093   *** TRACE FILE RECREATED AFTER BEING REMOVED *** Error 16401 creating standby archive log file at host 'testadg' kcrrwkx: unknown error:16401 Error 16401 creating standby archive log file at host 'testadg' kcrrwkx: unknown error:16401 Error 16401 creating standby archive log file at host 'testadg' *** 2021-05-02 00:18:37.836 kcrrwkx: unknown error:16401 Error 16401 creating standby archive log file at host 'testadg' kcrrwkx: unknown error:16401 *** 2021-05-02 00:22:16.469 Error 16401 creating standby archive log file at host 'testadg' kcrrwkx: unknown error:16401 6 查看ORA-16401错误，Oracle认为此错误可以忽略 [oracle@primarydb1 trace]$ oerr ora 16401 16401, 00000, "archive log rejected by Remote File Server (RFS)" // *Cause:  An attempt was made to re-archive an existing archive log. // *Action: See alert log and trace file for more details. //          No action is necessary. This is an informational statement provided //          to record the event for diagnostic purposes. 7 查看日志切换，发现日志切换太频繁，20秒一个，每个日志组大小为2G 8 根据Oracle官网的文档 Doc ID 1243177.1 认为此报错可以忽略,或增加redo log日志组的大小ORA-16401 and ORA-16055 reported in primary alert.log when redolog switch is over frequently (Doc ID 1243177.1)CAUSE  --原因The Problem here is that the Primary Database is switching Logs too frequently.  --日志切换太频繁Using ARCH to send the archives, every time there's a log switch the Primary has to send the Archivelog to the Standby,  --主库切换日志并发送redo日志到备库meanwhile another Log Switch occurred on the Primary which causes also another Archivelog to be sent to the Standby,     --同时意味着主库切换到下一个日志组，这会导致另一个日志组再次发送到备库。but the first one has not finished yet, a GAP is formed and detected by the Standby.    --这时由于前一个redo还没有应用完成，有间隙被诊断出来At this Time the first Archivelog is also sent as FAL Request,  --前一个redo正在发送请求but this one will fail because the first one is still being archiving, locked, so the second one fails.  --故导致第二个请求被拒绝，应用失败，导致告警Also check for the case explained in below note,Ora-16401: Archivelog Rejected By Rfs (Doc ID 1183143.1)SOLUTION  --解决方法1 Ignore these Messages as long as the Standby Database keeps synchronized with the Primary  --只要数据进行同步，此告警可以忽略2 Database Increase the Size of the Online Redologs to reduce Redolog Switch Frequency       --增加redo日志组的大小，减少切换频率3 Increase Network Bandwith between the Primary and Standby Database                         --增加主备库之间的带宽