记一次数据库事故-ORA-15038

来源：这里教程网时间：2026-03-03 15:30:54 作者：

背景：客户有两套较大的数据库(一套为10T（A数据库）, 一套为20T（B数据库）)想迁移存储，版本均为12.2.0.1, 采用了ASM管理磁盘，单实例。操作系统分别为linux 6, linux 7。客户想将B数据库的存储迁移到较差的存储，作为只读用途。腾空B数据库的存储之后，将B数据库占用的性能较好的存储分配给A数据库，再将A数据库的数据迁移到新分配的性能较好的存储。 A数据库的存储为较好存储与较差存储的混搭，需要迁移两次（第一次迁移腾出存储空间，第二次迁移到新回收的存储）。这里总共就有3次迁移。因为数据库较大与备份所需磁盘空间的问题，两个数据库均没有备份。 4月17日，客户告诉我，领导想要下周二能将两个库迁移好（我将将要写迁移方案）。这样就完全没有时间写方案。同时两个数据库需要同时迁移。于是我们负责存储的同事赶过来，给两个主机划好了存储。我做好了多路径映射并创建了ASM磁盘组。同时停库，开始做迁移。 4月18日晚上，A数据库数据文件拷贝完成。4月19日调整OCR，日志文件，临时文件等。在此期间，B数据库通过RMAN做COPY已经完成了接近16T左右, 还剩余大致4T数据文件。A数据库调整完成后，为了验证修改正确与否，重启了操作系统。操作系统起来之后，发现数据库起不来，同时新挂载的ASM磁盘组一直是offline。尝试手动mount, 报错: ORA-15032: not all alterations performed ORA-15038: disk '/dev/mapper/newdisk01' mismatch on 'Time Stamp' with target disk group [2434962992] [2434985720] ORA-15038: disk '/dev/mapper/newdisk02' mismatch on 'Time Stamp' with target disk group [2434962992] [2434985720] ORA-15038: disk '/dev/mapper/newdisk03' mismatch on 'Time Stamp' with target disk group [2434962992] [2434985720] ORA-15038: disk '/dev/mapper/newdisk04' mismatch on 'Time Stamp' with target disk group [2434962992] [2434985720] ORA-15038: di 通过谷歌，MOS搜索，可能是磁盘给其他ASM实例使用过，或者是多路径配置的问题。尝试修改asm_diskstring，尝试手动mount，这次错误变了： WARNING: Disk Group NEWDATA containing spfile for this instance is not mounted ORA-15032: not all alterations performed ORA-15038: disk '/dev/mapper/3600c0ff0003af211de24985e01000000' mismatch on 'Time Stamp' with target disk group [2434985720] [2434962992] 可以看到提示的磁盘变了。这个盘我并未加入ASM磁盘组，为什么也提示这个。我推测可能与原asm_diskstring='/dev/mapper/*'有关。虽然盘并未加入磁盘组，ASM实例启动的时候，还是去扫描了磁盘头。通过kfed, amdu等工具扫描ASM磁盘，都没有问题。同时， amdu可以读出数据文件。这样就没有丢失数据的风险（这里已经被吓尿了）。根据MOS文档， Doc ID 2643105.1, 也有类似的症状，一个没有加入到ASM磁盘组的磁盘，阻止了磁盘组的正常启动，同时也是发生在主机重启之后： SQL> alter diskgroup ACFS mount; alter diskgroup ACFS mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15017: diskgroup "ACFS" cannot be mounted ORA-15040: diskgroup is incomplete ORA-15038: disk '/dev/<asmdevices>/asm2' mismatch on 'Time Stamp' with target disk group [2026420570] [2026683750] ORA-15038: disk '/dev/<asmdevices>/asm1' mismatch on 'Time Stamp' with target disk group [2026420570] [2026683750] ORA-15038: disk '/dev/<asmdevices>/asm0' mismatch on 'Time Stamp' with target disk group [2026420570] [2026683750]

A Disk at OS level has this diskgroup information in its header incorrectly:

$GI_HOME/bin:>  kfed read <OS ASM device path>

kfdhdb.dskname: ACFS_4 ; 0x028: length=11
kfdhdb.grpname: ACFS ; 0x048: length=6                                          >>>>>>>>>>>>>>>>>>>Here
kfdhdb.fgname: ACFS_4 ; 0x068: length=11

Disks at OS level should not be updated manually after they have been added to a diskgroup. This will cause diskgroup to fail to mount. In this case, the bad disk was only on 1 node in the cluster. The Diskgroup mounted successfully on the other nodes.

ASM tried to mount the diskgroup with the bad OS disk:

SQL> ALTER DISKGROUP ACFS MOUNT /* asm agent *//* {1:52280:1676} */
NOTE: cache registered group ACFS number=1 incarn=0x47016be4
NOTE: cache began mount (not first) of group ACFS number=1 incarn=0x47016be4
NOTE: Assigning number (1,4) to disk (/dev/<asmdevices>/asm04)                   >>>>>>>>>>>>>>>>>>Here


However, only 3 disks at OS level are associated with this diskgroup in the ASM:

0 30 CLOSED MEMBER ONLINE NORMAL 245760 0 0 /dev/<asmdevices>/asm0
0 31 CLOSED MEMBER ONLINE NORMAL 245760 0 0 /dev/<asmdevices>/asm1
0 32 CLOSED MEMBER ONLINE NORMAL 245760 0 0 /dev/<asmdevices>/asm2
The device, /dev/<asmdevices>/asm04, is not listed.

    MOS给出的方案是dd掉问题磁盘的磁盘头。确实，我在dd磁盘头之后，手动mount磁盘组成功。但是悲剧的是，B数据库的RMAN Copy脚本出错了。

   RMAN-03009: 位于 04/19/2020 19:13:28 的 c1 通道上的 backup 命令失败ORA-19502: 文件 "+NEWDATA/TESTDB/DATAFILE/tbs_pdata.697.1038107047", 块编号 165696 (块大小=16384) 上出现写入错误ORA-15079: ASM 文件已关闭ORA-15079: ASM 文件已关闭


ASM日志：NOTE: AMDU dump of disk group NEWDATA initiated at /u01/app/grid/diag/asm/+asm/+ASM/traceErrors in file /u01/app/grid/diag/asm/+asm/+ASM/trace/+ASM_arb0_29651.trc  (incident=14697):ORA-15335: ASM metadata corruption detected in disk group 'NEWDATA'ORA-15130: diskgroup "NEWDATA" is being dismountedORA-15066: offlining disk "NEWDATA_0011" in group "NEWDATA" may result in a data lossORA-15196: invalid ASM block header [kfc.c:29757] [endian_kfbh] [2147483659] [1] [0 != 1]ORA-15196: invalid ASM block header [kfc.c:29757] [endian_kfbh] [2147483659] [1] [0 != 1]
在B数据库主机上确认，dd掉的那个磁盘正好是B数据库新加入的那个磁盘组的一个成员。无法，只得让存储工程师将这个盘从A数据库主机的映射断开，同时检查有无其它磁盘映射了多个主机。B库的迁移，只得重新创建磁盘组重新开始。
教训：1. 这种很急，比较复杂的迁移，多半要出事。要学会拒绝。           2. 存储映射好之后，应当确认两个主机之间没有映射同一个盘。           3. 必须做好备份。恢复慢是一码事，但是至少数据不会丢失，担惊受怕的程度能少点