Sunday, April 12, 2015

eBS r12 opatch issue on a cloned environment


My eBS r12 production environment is 12.1.X with a 11.2.0.3 database and using an RMAN catalog server for backup. I recently applied the CPUJan2015 to the production server after of course testing the on GOLD,GOLD and UAT no issues were detected.

When I refreshed the GOLD environment with the PROD clone I got a login page asking me to “contact the system administrator (myself)”

And error
ERROR at line 1:
ORA-29516: Aurora assertion failure: Assertion failure at eox.c:359
Uncaught exception Root of all Java exceptions: java.lang.UnsatisfiedLinkError sun.net.PortConfig.getLower0
ORA-06512: at "SYS.DBMS_JAVA", line 793
ORA-06512: at line 2

My first thought was the GUEST account issue and when I run the below SQL I got

SQL>select FND_WEB_SEC.VALIDATE_LOGIN('GUEST','ORACLE') from dual;
FND_WEB_SEC.VALIDATE_LOGIN('GUEST','ORACLE')
--------------------------------------------------------------------------------
N
1 row selected.
Surprised at the output I checked on all other environments and the output was a “Y”
So I embarked on trying to reset the GUEST password using the API format below in vain in.
[oracle@servernode applmgr]$ java oracle.apps.fnd.security.AdminAppServer APPS/"APPS Password" UPDATE GUEST_USER_PWD=GUEST/"Guest User Password" DB_HOST="Host_Name" DB_PORT="PortNumber" DB_NAME="SID"

• Check the log of FNDCPASS for any error for GUEST user if there is no error for GUEST user then do the next step

• Compile JSP
[oracle@servernode appl]$ cd $FND_TOP/patch/115/bin
[oracle@servernode bin]$ ojspCompile.pl --compile --flush

• Set GUEST_USER_PWD password in $FND_SECURE/.dbc to GUEST/ORACLE (in my case. could be different in your case)

• Bounced the Middle Tier Services and Retested the issue.

The problem was still not solved and I was back where I was started. I then realized that on the production environment where I have applied the same patches everything was running fine

Scouring the internet I landed on a blog which said

“Sometimes an application running in the Oracle JVM will fail with a java permissions error having the following format: java.sql.SQLException: ORA-29532:Java call terminated by uncaught Java exception:”

And OJVM was one of the components I had patched so I checked the OraInventory on the source the output showed the patch was well applied and to my surprise the with the output below

[oracle@SINJFMISAR002 19393542]$ opatch lsinventory -all
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.


Oracle Home : /a01/oracle/GOLD/db/tech_st/11.2.0
Central Inventory : /a01/oracle/oraInventory
from : /a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.3.0
Log file location : /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/opatch2015-03-29_13-45-48PM_1.log

List of Homes on this system:

Home name= GOLD_TOOLS__a04_applmgr_GOLD_apps_tech_st_10_1_2, Location= "/a04/applmgr/GOLD/apps/tech_st/10.1.2"
Home name= GOLD_WEBOH__a04_applmgr_GOLD_apps_tech_st_10_1_3, Location= "/a04/applmgr/GOLD/apps/tech_st/10.1.3"
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
Oracle Home dir. path does not exist in Central Inventory
Oracle Home is a symbolic link
Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73

1.1. Re-create Inventory
The inventory needs to be present for the cloned database.

As Oracle
- Shutdown Oracle
. $ORACLE_HOME/$CONTEXT_NAME.env
export TMP=$HOME/tmp
cd $ORACLE_HOME/appsutil/clone
./ouicli.pl

chmod 775 /a01/oracle/oraInventory/ContentsXML
cd /a01/oracle/oraInventory/ContentsXML
chmod 775 *


- As applmgr
- Shutdown application
export TMP=$HOME/tmp

chmod +r 775 /a01/oracle/oraInventory

. $INST_TOP/ora/10.1.2/$CONTEXT_NAME.env
cd $ORACLE_HOME/appsutil/clone
./ouicli.pl

. $INST_TOP/ora/10.1.3/$CONTEXT_NAME.env
cd $ORACLE_HOME/appsutil/clone
./ouicli.pl

Validated by running “opatch lsinventory” for both oracle and applmgr user.
So I decided to repair the inventory in the below chronology

Made changes to the following file:
/a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
Running OUI CLI home cloning from within OracleHomeCloner:
/a01/oracle/GOLD/db/tech_st/11.2.0/oui/bin/runInstaller -clone -silent -waitForCompletion -invPtrLoc /etc/oraInst.loc ORACLE_HOME=/ a01/oracle/GOLD/db/tech_st/11.2.0 ORACLE_BASE=/a01/oracle/GOLD/db/tech_st/11.2.0 ORACLE_HOME_NAME=GOLD_DB__a01_oracle_GOLD_db_tech_st_11_ 2_0 oracle_install_OSDBA=dba oracle_install_OSOPER=dba -noConfig -nowait
Finished OUI CLI cloning for s_db_oh with return code: 0Sat Apr 11 21:43:43 2015
[oracle@Servername clone]$ chmod 775 /a01/oracle/oraInventory/ContentsXML

[oracle@Servername clone]$ cd /a01/oracle/oraInventory/ContentsXML

[oracle@Servername ContentsXML]$ chmod 775 *

[oracle@Servername ContentsXML]$ ls

comps.xml inventory.xml inventory.xml.270315 libs.xml
[oracle@Servername ContentsXML]$ opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.
Oracle Home : /a01/oracle/GOLD/db/tech_st/11.2.0
Central Inventory : /a01/oracle/oraInventory
from : /a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.3.0
Log file location : /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/opatch2015-04-11_21-52-56PM_1.log

Lsinventory Output file location : /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/lsinv/lsinventory2015-04-11_21-52-56PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Database 11g 11.2.0.3.0
Oracle Database 11g Examples 11.2.0.3.0
There are 2 products installed in this Oracle Home.

Interim patches (6) :

Patch 19121548 : applied on Tue Nov 11 11:07:01 CET 2014
Unique Patch ID: 17940820
Patch description: "Database Patch Set Update : 11.2.0.3.12 (19121548)"
Created on 10 Oct 2014, 06:27:11 hrs PST8PDT
Sub-patch 18522512; "Database Patch Set Update : 11.2.0.3.11 (18522512)"
Sub-patch 18031683; "Database Patch Set Update : 11.2.0.3.10 (18031683)"
Sub-patch 17540582; "Database Patch Set Update : 11.2.0.3.9 (17540582)"
Sub-patch 16902043; "Database Patch Set Update : 11.2.0.3.8 (16902043)"
Sub-patch 16619892; "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch 16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch 14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch 14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch 13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch 13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch 13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Bugs fixed:
13593999, 10350832, 19433746, 14138130, 12919564, 14198511, 13561951
1xxxxxxxxx xxxxxx xxxxxxx xxxxxxxxxxxxxxx
12821418, 13502183, 13705338, 14237793, 16794239, 13554409, 15862024
13103913, 13645917, 12772404
Patch 13366268 : applied on Tue Jun 18 09:26:46 CEST 2013
Unique Patch ID: 14394336
Created on 21 Dec 2011, 08:14:06 hrs PST8PDT
Bugs fixed:
13366268
Patch 13258936 : applied on Tue Jun 18 09:26:18 CEST 2013
Unique Patch ID: 14465000
Created on 5 Dec 2011, 02:41:32 hrs PST8PDT
Bugs fixed:
13258936
Patch 12985184 : applied on Tue Jun 18 09:24:33 CEST 2013
Unique Patch ID: 14296756
Created on 30 Nov 2011, 15:18:06 hrs PST8PDT
Bugs fixed:
12985184
Patch 12960302 : applied on Tue Jun 18 09:23:53 CEST 2013
Unique Patch ID: 14254250
Created on 10 Nov 2011, 02:40:18 hrs PST8PDT
Bugs fixed:
12960302
Patch 4247037 : applied on Tue Jun 18 09:21:05 CEST 2013
Unique Patch ID: 14211388
Created on 17 SEP 2009, 09:52:00 hrs US/Pacific
Bugs fixed:
4247037
--------------------------------------------------------------------------------

OPatch succeeded.

I then proceeded to re-apply the CPUJan2015


Since CpuJan15 and later, Oracle JavaVM Component Database PSU is a new Patch Set Update for Database versions 11.1.0.7, 11.2.0.3, 11.2.0.4, 12.1.0.1 and 12.1.0.2. This PSU addresses Oracle JavaVM CPU program security vulnerabilities.

CPU Patch for Application

Follow below procedure to apply CPU patch to e-business suite application.
Copy/FTP both the patches to /u01/One-Off/cpuJan15/ directory.

The patches need to be applied using application owner. applmgr
1.2. Shutdown Application Service.

 Log into unix using any ssh client such as PUTTY
Login> applmgr 
Pwd> ****** 

 Stop the Application services
$> cd $HOME/bin 
$> {ENV}_APPS.stop 

 Check that unix process for applmgr are stopped
$> ps -ef |grep applmgr |egrep -v 'ps|bash' 

Once all the applmgr processes have gone, move to next step.
1.3. Enable Maintenance Mode.

Enable the maintenance mode in application by running adadmin tool

$ adadmin
- Option 5 (Change Maintenance Mode)
- Option 1 (Enable Maintenance Mode)
- Enter
- Option 3 (Return to Main Menu)
- Option 6 (Exit AD Administration)
1.4. Take Pre Invalid List.

Take a snapshot of invalid objects, which will be compared later after patch.

$ cd /u01/One-Off/cpuJan15/
$ sqlplus apps/$APPS_PWD @/home/applmgr/scripts/objinval > objinval_{ENV}.pre







1.5. Apply Patch 19393542.

This patch is opatch and needs to be applied as Oracle user on 11.2.0.3 Home
Shutdown database and listener
Run below command to verify any patch conflict regarding this patch.
$ cd /u01/One-Off/cpuJan15/19393542
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./

Apply patch using opatch.
$ cd $PATCH_DIRECTORY/19393542
$ opatch apply
Output should look like this
[oracle@Servername 19393542]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.4 Copyright (c) 2012, Oracle Corporation. All rights reserved. Oracle Home : /a01/oracle/GOLD/db/tech_st/11.2.0
Central Inventory : /a01/oracle/oraInventory
from : /a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.3.0
Log file location : /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/19393542_Mar_12_2015_14_27_57/apply2015-03-12_14-27-57PM_1.log Applying interim patch '19393542' to OH '/a01/oracle/GOLD/db/tech_st/11.2.0' Verifying environment and performing prerequisite checks...
All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/a01/oracle/GOLD/db/tech_st/11.2.0') Is the local system ready for patching? [y|n] y
User Responded with: Y
Backing up files... Patching component oracle.rdbms, 11.2.0.3.0... Verifying the update...
Patch 19393542 successfully applied
Log file location: /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/19393542_Mar_12_2015_14_27_57/apply2015-03-12_14-27-57PM_1.log

OPatch succeeded.
[oracle@Servername 19393542]$

Review the log file under $ORACLE_HOME/cfgtoollogs/opatch directory for any errors.

Bring up database and listener.

1.1. Apply Patch 19873049.
As applmgr.
$ cd /u01/One-Off/cpuJan15/19873049
$ $AD_TOP/bin/admsi.pl -patch_top=/u01/One-Off/cpuJan15/19873049 -appspass=$APPS_PWD

[applmgr@Servername One-off]$ $AD_TOP/bin/admsi.pl -patch_top=/home/applmgr/One-off/19873049 -appspass=$APPS_PWD
Gathering Information..
Logfile for this session is located at admsi.log
Generating installation instructions for patch 19873049..
Updating database....
install_19873049.html generated successfully
install_19873049.txt generated successfully


As oracle

$ cd $ORACLE_HOME/appsutil/admin
$ cp -pr /a04/applmgr/GOLD/apps/apps_st/appl/admin/adgrants.sql .
$ sqlplus /nolog
SQL> connect / as sysdba
SQL> @adgrants.sql APPS
Connected.
------ adgrants.sql started at 2015-03-12 14:57:05 ---
Creating PL/SQL profiler objects.
------- profload.sql started at 2015-03-12 14:57:05 ---
Package created.
Grant succeeded.
Synonym created.
Library created.
Package body created.
Testing for correct installation
SYS.DBMS_PROFILER successfully loaded.
PL/SQL procedure successfully completed.
---- profload.sql completed at 2015-03-12 14:57:07 ---
----proftab.sql started at 2015-03-12 14:57:07 ---
Table dropped.
Table dropped.
Table dropped.
Sequence dropped.
Table created.
Comment created.
Table created.
Comment created.
Table created.
Comment created.
Sequence created.
----- profltab.sql completed at 2015-03-12 14:57:07 ---
Installing Hierarchical Profiler.
Table dropped.
Table dropped.
Table dropped.
Sequence dropped.
Table created.
Comment created.
Table created.
Comment created.
Table created.
Comment created.
Sequence created.
Package created.
Synonym created.
Grant succeeded.
Grant succeeded.
Grant succeeded.
Grant succeeded.
Start of Creating AD_JAR context
PL/SQL procedure successfully completed.
End of Creating AD_JAR context
Creating PL/SQL Package AD_DBMS_METADATA.
Package created.
Package body created.
End of Creating PL/SQL Package AD_DBMS_METADATA.
PL/SQL procedure successfully completed.
Commit complete.

As the application owner --- applmgr

$ adpatch

[applmgr@Servername 19873049]$ $AD_TOP/bin/admsi.pl -patch_top=/u01/One-Off/cpuJan15/19873049 -appspass=$APPS_PWD
Gathering Information..
Logfile for this session is located at admsi.log
Generating installation instructions for patch 19873049..
Updating database....
install_19873049.html generated successfully
install_19873049.txt generated successfully

Review the log file adpatch_19873049.log under $APPL_TOP/admin/{ENV}/log directory for any errors.
*If there there is a failure with ADFIXUSER.sql its because the patch has delivered a new version of adgrants so coppy and re-run it should run successfuly

#######################################################
ATTENTION: All workers either have failed or are waiting:
FAILED: file ADFIXUSER.sql on worker 1.
Perform the adgrants copy and re-run again.and restart the worker
#######################################################


$ $AD_TOP/bin/admsi.pl --done

Gathering Information..
Enter the patch_top : /u01/ebs_patches/FamilyPack/19278976
Enter the APPS user password:
Logfile for this session is located at admsi.log
Updating database for completion..
Done.

[applmgr@Servername 19873049]$ $AD_TOP/bin/admsi.pl --done
Gathering Information..
Enter the patch_top: /u01/One-Off/cpuJan15/19873049
Enter the APPS user password:
Logfile for this session is located at admsi.log
Updating database for completion...Done.
[applmgr@Servename 19873049]$
1.2. Run Autoconfig.
Apply autoconfig step after patch.

$ $INST_TOP/admin/scripts/adautocfg.sh appspass=$APPS_PWD
2. CPU Patch for Database
Follow below procedure to apply CPU patch to all database including RMAN server.
The patch needs to be applied as the database owner Oracle.

2.1. Stop Database Service.
 Log into unix using any ssh client such as PUTTY
Login> oracle 
Pwd> ****** 


 Stop the Application services
$> cd $HOME/bin 
$> {ENV}_DB.stop 

 Check that unix process for oracle are stopped
$> ps -ef |grep oracle |egrep -v 'ps|bash' 
2.2. Verify Patch List.
Run below command to verify current PSU patch list.
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ opatch lsinventory -bugs_fixed |grep -i "PATCH SET UPDATE"

The output will show as below
19121548 19121548 Tue Nov 11 11:07:01 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.12 (INCLUDES CP
18522512 18522512 Tue Nov 11 10:21:00 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.11 (INCLUDES CP
18031683 18031683 Tue Jun 24 13:33:56 CEST 2014 DATABASE PATCH SET UPDATE 11.2.0.3.10 (INCLUDES CP
17540582 17540582 Mon Mar 17 13:40:22 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.9 (INCLUDES CPU
16902043 16902043 Mon Mar 17 13:40:09 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.8 (INCLUDES CPU
16619892 16619892 Mon Sep 30 10:24:39 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.7 (INCLUDES CPU
16056266 16056266 Mon Sep 30 10:24:12 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.6 (INCLUDES CPU
14727310 14727310 Mon Sep 30 10:24:00 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.5 (INCLUDES CPU
14275605 14275605 Mon Sep 30 10:23:10 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.4 (INCLUDES CPU
13923374 13923374 Mon Sep 30 10:23:01 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.3 (INCLUDES
13696216 13696216 Mon Sep 30 10:22:29 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.2 (INCLUDES
13343438 13343438 Mon Sep 30 10:21:22 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.1
2.3. Verify Patch Conflict.
Run below command to verify any patch conflict regarding this patch.

$ cd /u01/One-Off/cpuJan15/20132646/19769496
$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./
[oracle@SINJFMISAR003 19769496]$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /a01/oracle/GOLD/db/tech_st/11.2.0
Central Inventory: /a01/oracle/oraInventory
from : /a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.3.0
Log file location: /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/opatch2015-03-13_12-24-20PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

This should give no patch conflict.

Prereq "checkConflictAgainstOHWithDetail" passed.
2.4. How to correct inventory if corrupt
At times the Inventory location can become corrupt or might have changed due to architecture due to different file systems etc. this will cause Opatch to error out with error code 104 and can hamper the database patch application.

To fix a corrupted oracle inventory implicates at times moving it to a different location or editing Contents of the XML and changing the ORACLE_HOME_NAME

[oracle@SINJFMISAR002 bin]$ ./runInstaller -silent -invPtrLoc "/a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc" -attachHome ORACLE_HOME="/a01/oracle/GOLD/db/tech_st/11.2.0" ORACLE_HOME_NAME="Ora11g_home1"
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 17407 MB Passed
The inventory pointer is located at /a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
The inventory is located at /a01/oracle/oraInventory
'AttachHome' was successful.
2.5. Apply Patch 19769496
Run below command to verify any patch conflict regarding this patch.
$ cd /u01/One-Off/cpuJan15/20132646/19769496
$ opatch apply
Warning:
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ins_precomp.mk:19: warning: overriding commands for target `pcscfg.cfg'
/a01/oracle/GOLD/db/tech_st/11.2.0/precomp/lib/env_precomp.mk:2160: warning: ignoring old commands for target `pcscfg.cfg'
/a01/oracle/GOLD/db/tech_st/11.2.0/precomp/lib/ins_precomp.mk:19: warning: overriding commands for target `pcscfg.cfg'
/a01/oracle/GOLD/db/tech_st/11.2.0/precomp/lib/env_precomp.mk:2160: warning: ignoring old commands for target `pcscfg.cfg'

Solution: Ignore - Opatch warning: overriding commands for target xxxx (Doc ID 1448337.1)
2.6. Verify Patch List.
Run below command to verify current PSU patch list.
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ opatch lsinventory -bugs_fixed |grep -i "PATCH SET UPDATE"

The output will show as below
19769496 19769496 Tue Jan 27 15:51:30 CET 2015 DATABASE PATCH SET UPDATE 11.2.0.3.13 (INCLUDES CP
19121548 19121548 Tue Nov 11 11:07:01 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.12 (INCLUDES CP
18522512 18522512 Tue Nov 11 10:21:00 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.11 (INCLUDES CP
18031683 18031683 Tue Jun 24 13:33:56 CEST 2014 DATABASE PATCH SET UPDATE 11.2.0.3.10 (INCLUDES CP
17540582 17540582 Mon Mar 17 13:40:22 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.9 (INCLUDES CPU
16902043 16902043 Mon Mar 17 13:40:09 CET 2014 DATABASE PATCH SET UPDATE 11.2.0.3.8 (INCLUDES CPU
16619892 16619892 Mon Sep 30 10:24:39 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.7 (INCLUDES CPU
16056266 16056266 Mon Sep 30 10:24:12 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.6 (INCLUDES CPU
14727310 14727310 Mon Sep 30 10:24:00 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.5 (INCLUDES CPU
14275605 14275605 Mon Sep 30 10:23:10 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.4 (INCLUDES CPU
13923374 13923374 Mon Sep 30 10:23:01 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.3 (INCLUDES
13696216 13696216 Mon Sep 30 10:22:29 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.2 (INCLUDES
13343438 13343438 Mon Sep 30 10:21:22 CEST 2013 DATABASE PATCH SET UPDATE 11.2.0.3.1
2.7. Load Modified SQL Files.
Load modified sql files into the database.
$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT
Review the log files under $ORACLE_HOME/cfgtoollogs/catbundle
2.8. Verify The Patch.
Load modified sql files into the database.

$ sqlplus /nolog
SQL> connect /as sysdba
Connected.
SQL> set lines 200
col ACTION format a15
col NAMESPACE format a10
col version format a10
col COMMENTS format a25
col BUNDLE_SERIES format a13
col action_time format a30
select * from sys.registry$history;

The output is shown below.

ACTION_TIME ACTION NAMESPACE VERSION ID COMMENTS BUNDLE_SERIES
------------------------------ --------------- ---------- ---------- ---------- ------------
18-JUN-13 12.51.49.442929 PM VIEW INVALIDATE 8289601 view invalidation
18-JUN-13 12.51.52.381437 PM UPGRADE SERVER 11.2.0.3.0 Upgraded from 11.1.0.7.0
18-JUN-13 12.57.57.918181 PM APPLY SERVER 11.2.0.3 0 Patchset 11.2.0.2.0 PSU
23-JAN-08 04.07.02.182324 PM CPU SERVER 10.2.0.3.0 6079591 CPUJul2007
26-NOV-08 11.39.09.067225 PM UPGRADE SERVER 11.1.0.6.0 Upgraded from 10.2.0.3.0
22-JAN-09 04.34.49.320298 AM UPGRADE SERVER 11.1.0.7.0 Upgraded from 11.1.0.6.0
17-MAR-14 01.49.21.350997 PM APPLY SERVER 11.2.0.3 9 PSU 11.2.0.3.9 PSU
24-JUN-14 01.35.47.421582 PM APPLY SERVER 11.2.0.3 10 PSU 11.2.0.3.10 PSU
11-NOV-14 10.24.04.404791 AM APPLY SERVER 11.2.0.3 11 PSU 11.2.0.3.11 PSU
11-NOV-14 11.09.18.638470 AM APPLY SERVER 11.2.0.3 12 PSU 11.2.0.3.12 PSU
27-JAN-15 03.56.56.396273 PM APPLY SERVER 11.2.0.3 13 PSU 11.2.0.3.13 PSU
2.9. Upgrade RMAN Catalog
Upgrade the catalog for rman.

Note: ONLY ONCE DONE IN FOR PROD as only PROD has RMAN:

$ rman catalog username/password@alias
RMAN> UPGRADE CATALOG;
3. OJVM Patch for Database
Follow below procedure to apply OJVM patch to all database including RMAN server.

The patches need to be applied using as the database owner (oracle).

3.1. Shutdown Database
Shutdown the database for applying OJVM patch
3.2. Verify Patch Conflict.
Run below command to verify any patch conflict regarding this patch.

$ cd /u01/One-Off/cpuJan15/20132646/19877443
$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.
PREREQ session

Oracle Home : /a01/oracle/GOLD/db/tech_st/11.2.0
Central Inventory : /a01/oracle/oraInventory
from : /a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.3.0
Log file location : /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/opatch2015-03-29_16-21-02PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

This should give no patch conflict.

Prereq "checkConflictAgainstOHWithDetail" passed.
3.3. Apply Patch 19877443

Run below command to verify any patch conflict regarding this patch.

$ cd /u01/One-Off/cpuJan15/20132646/19877443
[oracle@Servername 19877443]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.


Oracle Home : /a01/oracle/GOLD/db/tech_st/11.2.0
Central Inventory : /a01/oracle/oraInventory
from : /a01/oracle/GOLD/db/tech_st/11.2.0/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.3.0
Log file location : /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/19877443_Mar_29_2015_16_21_50/apply2015-03-29_16-21-50PM_1.log

Applying interim patch '19877443' to OH '/a01/oracle/GOLD/db/tech_st/11.2.0'
Verifying environment and performing prerequisite checks...
Patch 19877443: Optional component(s) missing : [ oracle.sqlj, 11.2.0.3.0 ]
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/a01/oracle/GOLD/db/tech_st/11.2.0')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Patching component oracle.javavm.server, 11.2.0.3.0...
Patching component oracle.rdbms, 11.2.0.3.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.3.0...
Patching component oracle.sqlj.companion, 11.2.0.3.0...
Patching component oracle.dbjava.ic, 11.2.0.3.0...
Patching component oracle.dbjava.jdbc, 11.2.0.3.0...
Verifying the update...
--------------------------------------------------------------------------------
***********************************************************************************
***********************************************************************************
** ATTENTION **
** **
** It is critical and essential to complete the post installation (if you applying ** the patch) or post deinstallation (if you are doing rollback of the patch) as
** per Readme to have a working and functional Oracle JavaVM Component. **
** Failure to follow the instructions could lead to inconsistencies in the database. **
***********************************************************************************
***********************************************************************************
--------------------------------------------------------------------------------
Patch 19877443 successfully applied
Log file location: /a01/oracle/GOLD/db/tech_st/11.2.0/cfgtoollogs/opatch/19877443_Mar_29_2015_16_21_50/apply2015-03-29_16-21-50PM_1.log

OPatch succeeded.

3.4. Verify Patch List.
Run below command to verify current PSU patch list.

$ opatch lsinventory -bugs_fixed |grep -i "ORACLE JAVAVM COMPONENT 11.2"

The output will show as below
[oracle@SINJFMISAR002 19877443]$ opatch lsinventory -bugs_fixed |grep -i "ORACLE JAVAVM COMPONENT 11.2"
19852361 19877443 Sun Mar 29 16:22:52 CEST 2015 ORACLE JAVAVM COMPONENT 11.2.0.3.1 DATABASE PSU -
19877443 19877443 Sun Mar 29 16:22:52 CEST 2015 ORACLE JAVAVM COMPONENT 11.2.0.3.2 DATABASE PSU (J

3.5. Run PostInstall SQL File.
Load modified sql files into the database.

$ cd $ORACLE_HOME/sqlpatch/19877443
$ sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @postinstall.sql
SQL> QUIT
3.6. Run Compile.
Load modified sql files into the database.

$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp
SQL> QUIT

3.7. Start Database Listener
 Log into unix using any ssh client such as PUTTY
Login> oracle 
Pwd> ****** 
 Start the database listener
$> cd $HOME/bin 
$> lsnrctl start {ENV} 
3.8. Take Post Invalid List.

Take a snapshot of invalid objects, which will be compared later after patch.
$ cd /u01/One-Off/cpuJan15/
$ sqlplus apps/$APPS_PWD @/home/applmgr/scripts/objinval > objinval_{ENV}.post
4. Post Patch Steps.
4.1. Disable Maintenance Mode
Disable the maintenance mode in application by running adamin tool
$ adadmin
- Option 5 (Change Maintenance Mode)
- Option 2 (Disable Maintenance Mode)
- Enter
- Option 3 (Return to Main Menu)
- Option 6 (Exit AD Administration)
Change Maintenance Mode
----------------------------------------
Maintenance Mode is currently: [Enabled].
Maintenance mode should normally be enabled when patching
Oracle Applications and disabled when users are logged on
to the system. See the Oracle Applications Maintenance
Utilities manual for more information about maintenance mode.
Please select an option:
1. Enable Maintenance Mode
2. Disable Maintenance Mode
3. Return to Main Menu
Enter your choice [3] : 2
sqlplus -s &un_apps/***** @/a04/applmgr/GOLD/apps/apps_st/appl/ad/12.0.0/patch/115/sql/adsetmmd.sql DISABLE

Successfully disabled Maintenance Mode.

4.2. Start Application Service.
 Log into unix using any ssh client such as PUTTY
Login> applmgr 
Pwd> ****** 
 Stop the Application services
$> cd $HOME/bin 
$> {ENV}_APPS.start 
 Check that unix process for applmgr are started
$> ps -ef |grep applmgr 
4.3. Validation
Validate application by logging into application and running Active User Report.
5. APPENDIX