故障描述
某次深夜，用户反馈某核心业务系统故障，提示无法连接数据库。之后，用户同步尝试在堡垒机上，使用客户端工具连接数据库，同样命中报错，输出TNS-12514无法连接数据库。相信但凡有经验的DBA，遇到此类报错，可能会想“这不是很常见的连接问题嘛”，但事实真的有这么简单么？
根因分析
首先，用户就检查了数据库alert日志，其中也未发现ORA-报错信息，在和用户的交流当中，其实也大概排除了网络问题以及可能的配置变更，如下图所示：
其次查看集群状态crsctl stat res -t显示正常，CRS日志该时间点未发现错误信息，系统日志/var/log/messages该时间点未发现错误信息，监听状态正常但监听日志里报错无法连接，如下显示：
Sat Jan 28 00:08:04 2023 28-JAN-2023 00:08:04 * service_update * ecrac1 * 0 28-JAN-2023 00:08:07 * service_update * ecrac1 * 0 Sat Jan 28 00:08:16 2023 28-JAN-2023 00:08:16 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ecrac1)(CID=(PROGRAM=sqlplus)(HOST=infolexdr)(USER=mon))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.128.243.41)(PORT=19157)) * establish * ecrac1 * 12514 TNS-12514: TNS:listener does not currently know of service requested in connect descriptor 28-JAN-2023 00:08:16 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ecrac1)(CID=(PROGRAM=sqlplus)(HOST=infolexdr)(USER=mon))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.128.243.41)(PORT=19158)) * establish * ecrac1 * 12514 TNS-12514: TNS:listener does not currently know of service requested in connect descriptor 28-JAN-2023 00:08:16 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ecrac1)(CID=(PROGRAM=sqlplus)(HOST=infolexdr)(USER=mon))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.128.243.41)(PORT=19159)) * establish * ecrac1 * 12514 TNS-12514: TNS:listener does not currently know of service requested in connect descriptor 28-JAN-2023 00:08:16 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ecrac1)(CID=(PROGRAM=sqlplus)(HOST=infolexdr)(USER=mon))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.128.243.41)(PORT=19160)) * establish * ecrac1 * 12514 TNS-12514: TNS:listener does not currently know of service requested in connect descriptor 28-JAN-2023 00:08:16 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ecrac1)(CID=(PROGRAM=sqlplus)(HOST=infolexdr)(USER=mon))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.128.243.41)(PORT=19161)) * establish * ecrac1 * 12514 TNS-12514: TNS:listener does not currently know of service requested in connect
通过检查TNS-12514错误输出，大致判断跟SERVICE_NAME相关，返回查看alert日志里有修改service_name的操作，引起关注。与客户核实确认问题时间点是在执行expdp逻辑备份数据库操作，近期无数据库参数修改等操作，且第一次出现该故障。重点分析alert日志，发现逻辑备份前系统执行alter system set 语句如下：
Sat Jan 28 00:05:02 2023 ALTER SYSTEM SET service_names='ecrac1','SYS$SYS.KUPC$C_1_20230128000501.ECRAC' SCOPE=MEMORY SID='ecrac1'; ALTER SYSTEM SET service_names='SYS$SYS.KUPC$C_1_20230128000501.ECRAC','ecrac1','SYS$SYS.KUPC$S_1_20230128000501.ECRAC' SCOPE=MEMORY SID='ecrac1'; Sat Jan 28 00:05:02 2023 DM00 started with pid=645, OS id=141291, job IMPCBSMID.SYS_EXPORT_SCHEMA_01 Sat Jan 28 00:05:03 2023 ALTER SYSTEM SET service_names='SYS$SYS.KUPC$S_1_20230128000501.ECRAC','ecrac1' SCOPE=MEMORY SID='ecrac1'; ALTER SYSTEM SET service_names='ecrac1' SCOPE=MEMORY SID='ecrac1'; Sat Jan 28 00:05:06 2023 ALTER SYSTEM SET service_names='ecrac1','SYS$SYS.KUPC$C_1_20230128000506.ECRAC' SCOPE=MEMORY SID='ecrac1'; ALTER SYSTEM SET service_names='SYS$SYS.KUPC$C_1_20230128000506.ECRAC','ecrac1','SYS$SYS.KUPC$S_1_20230128000506.ECRAC' SCOPE=MEMORY SID='ecrac1'; Sat Jan 28 00:05:07 2023 DM00 started with pid=326, OS id=141607, job IMPCBSMID.SYS_EXPORT_SCHEMA_01 Sat Jan 28 00:05:08 2023 DW00 started with pid=133, OS id=141707, wid=1, job IMPCBSMID.SYS_EXPORT_SCHEMA_01 Sat Jan 28 00:10:04 2023 ALTER SYSTEM SET service_names='SYS$SYS.KUPC$S_1_20230128000506.ECRAC','SYS$SYS.KUPC$C_1_20230128000506.ECRAC','ecrac1','SYS$SYS.KUPC$C_1_20230128001004.ECRAC' SCOPE=MEMORY SID='ecrac1'; ALTER SYSTEM SET service_names='SYS$SYS.KUPC$C_1_20230128001004.ECRAC','SYS$SYS.KUPC$S_1_20230128000506.ECRAC','SYS$SYS.KUPC$C_1_20230128000506.ECRAC','ecrac1','SYS$SYS.KUPC$S_1_20230128001004.ECRAC' SCOPE=MEMORY SID='ecrac1';
理论上service_names包含了实例名即可正常连接，其他名字是系统默认追加进去，逻辑备份结束后会自动去掉，但该问题现象感觉跟此相关。对比其他rac环境上的alert日志发现也是如此记录，未发现异常点；对比前几天的备份点均如此，无异常。初步判断该问题为数据库bug导致，查看mos文档：Bug 14320415 - The value of service name is sometimes removed during expdp/impdp in RAC - superseded (Doc ID 14320415.8)
事后，在和客户的交流沟通中，有聊到这套数据库实例大概有5、6年时间没有重启了，Oracle官方在原文中描述，数据库实例在长期未重启过的情况下，如果注册service_names次数达到65536上限后，就会触发此bug导致注册失败，进而会引发连接问题。
解决方案
针对该案例，MOS文档原文中也给出了几种解决方案，最快捷的方式是重启数据库实例，不过需要额外申请停机窗口；其次，可以选择安装该BUG对应的补丁，甚至可以升级至大版本，比如12C或者19C，不过这类动作影响较大，停机时间更久。综上，最后用户通过重启实例解决，对于每年定期重启数据库或者服务器的用户，也不太可能碰到类似的问题。详情可以参考：Connection With Net Service Name Sometimes Fails With ORA-12514 During DataPump Export/Import In RAC (Doc ID 2302731.1)
