19c 打补丁遇到节点2 OPATCHAUTO-68061 报错

遇到了两次OPATCHAUTO-68061 报错

按照正常的打补丁流程：
1. 分别在node1、node2上，升级GI OPatch
2. 分别在node1、node2上，升级Oracle OPatch
3. 分别在node1、node2上，给GI 打补丁
4. 分别在node1、node2上，给Oracle打补丁

结果就在成功在node1上给Oracle打补丁之后，去node2打的时候报错了。离成功就差一丢丢的时候。

第一次 OPATCHAUTO-68061 的报错信息

root@coredb2-adg soft]# /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto apply /u01/soft/31305339 -oh /u01/app/oracle/product/19.0.0/db_1

OPatchauto session is initiated at Fri Sep 11 17:29:55 2020

System initialization log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchautodb/systemconfig2020-09-11_05-29-59PM.log.

Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-09-11_05-30-12PM.log
The id for this session is Q3Y5

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1


Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1


Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Failed while applying binary patches on home /u01/app/oracle/product/19.0.0/db_1

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : coredb2-adg->/u01/app/oracle/product/19.0.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.0.0/db_1, host: coredb2-adg.
Command failed:  /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto  apply /u01/soft/31305339 -oh /u01/app/oracle/product/19.0.0/db_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.0.0/db_1/oraInst.loc -jre /u01/app/oracle/product/19.0.0/db_1/OPatch/jre -persistresult /u01/app/oracle/product/19.0.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_coredb2-adg_rac_2.ser -analyzedresult /u01/app/oracle/product/19.0.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_coredb2-adg_rac_2.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /u01/soft/31305339/31281355
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-09-11_17-30-35PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Fri Sep 11 17:31:52 2020
Time taken to complete the session 1 minute, 57 seconds

 opatchauto failed with error code 42

看到了很明显的(Permission denied) 字样。

解决办法

给oui-patch.xml文件777 的权限。

[root@base3db2-adg db_1]# cd /u01/app/oraInventory/ContentsXML/
[root@base3db2-adg ContentsXML]# ll
total 20
-rw-rw---- 1 grid   oinstall 300 Sep 11 15:52 comps.xml
-rw-rw---- 1 grid   oinstall 557 Sep 11 15:52 inventory.xml
-rw-rw---- 1 grid   oinstall 292 Sep 11 15:52 libs.xml
-rw-r--r-- 1 grid   oinstall 174 Sep 11 14:47 oui-patch.xml
-rw-rw---- 1 oracle oinstall 174 Sep 11 14:47 oui-patch.xml.back
[root@base3db2-adg ContentsXML]# chmod 777 oui-patch.xml

然后可用以下命令，继续打补丁：

[root@base3db2-adg ContentsXML]# /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto resume

第二次 OPATCHAUTO-68061 的报错信息

没过一会儿又报错了，错误号还是 68061，但是前面输出的信息变了：

root@base3db2-adg ContentsXML]# /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto resume

OPatchauto session is initiated at Fri Sep 11 16:18:05 2020
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-09-11_04-18-07PM.log
Resuming existing session with id LIPB

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verification failed on home /u01/app/oracle/product/19.0.0/db_1

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : base3db2-adg->/u01/app/oracle/product/19.0.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.0.0/db_1, host: base3db2-adg.
Command failed:  /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto  apply /u01/soft/31305339 -oh /u01/app/oracle/product/19.0.0/db_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.0.0/db_1/oraInst.loc -jre /u01/app/oracle/product/19.0.0/db_1/OPatch/jre -persistresult /u01/app/oracle/product/19.0.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_base3db2-adg_rac_2.ser -analyze -online -prepare_home
Command failure output: 
==Following patches FAILED in analysis for apply:

Patch: /u01/soft/31305339/31305087
Log: 
Reason: Failed during listing in Analysis: java.lang.Exception: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/31281355 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31281355/etc/config/inventory" does not exist.


Patch: /u01/soft/31305339/31281355
Log: 
Reason: Failed during listing in Analysis: java.lang.Exception: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/31281355 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31281355/etc/config/inventory" does not exist. 

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Fri Sep 11 16:18:18 2020
Time taken to complete the session 0 minute, 13 seconds

 opatchauto failed with error code 42

报错说ORACLE_HOME/inventory/oneoffs/31281355 is corrupted
corrupted！ corrupted ！！ 居然损坏了！？ 奇怪

解决办法

既然node1 打好了补丁，node2 一直在说 ‘31281355’ 损坏。
那么。。emmm。。简单粗暴的去node1 ”借用一下” 31281355

• 看下node1的，emm 有

[oracle@base3db1:/home/oracle]$ cd /u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/
[oracle@base3db1:/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs]$ ls
29517242  29585399  31281355  31305087

• 传输给node2

[oracle@base3db1-adg:/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs]$ scp -r 31281355 db2-adg:`pwd`
inventory.xml                                                                                                                                                            100%  276KB  95.3MB/s   00:00    
actions.xml                                                                                                                                                              100% 1614KB 136.0MB/s   00:00    

• 然后node2 继续打，使用 OPatchAuto resume :

[root@base3db2-adg oneoffs]# /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto resume

OPatchauto session is initiated at Fri Sep 11 16:20:59 2020
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-09-11_04-21-01PM.log
Resuming existing session with id LIPB

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1


Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1


Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/db_1


Performing postpatch operation on home /u01/app/oracle/product/19.0.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1


Preparing home /u01/app/oracle/product/19.0.0/db_1 after database service restarted
No step execution required.........
 

Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:base3db2-adg
RAC Home:/u01/app/oracle/product/19.0.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/soft/31305339/31281355
Reason: This patch is already been applied, so not going to apply again.


==Following patches were SUCCESSFULLY applied:

Patch: /u01/soft/31305339/31305087
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-09-11_16-21-22PM_1.log



OPatchauto session completed at Fri Sep 11 16:21:52 2020
Time taken to complete the session 0 minute, 53 seconds

最终再一次开心的开到了 completed 。

所以给19c打补丁的时候，为了避免报错，那就提前给node2的oracle打补丁之前，该改权限的改权限，改传文件的传文件，提前做。然后打补丁！ 一条命令顺顺利利的