Upgrading Oracle 12c (12.1.0.2) Grid Infrastructure and Database to 19c

In this guide, we will upgrade Oracle 12c (12.1.0.2) GI and DB to 19c on an Oracle Linux 7.


For the purpose of this guide, note the following:


-To make following this guide easier, put all your zip files in /usr/software/oracle.

-GI stands for Grid Infrastructure, also known as Oracle Restart.

-DB Home owner is Linux OS user “oracle”.

-12c DB Home: /oracle/oraBase/oracle/12.1.0.2/database

-19c DB Home: /oracle/oraBase/oracle/19.0.0/database

-PROD is my database SID. Change it to match your database.

-+ASM is my ASM SID. Change it to match your ASM.

-GI Home owner is Linux OS user “grid”.

-12c GI Home: /oracle/oraBase/12.1.0.2/grid

-19c GI Home: /oracle/oraBase/19.0.0/grid

-On certain occasions, I will display the “su” command to let you know what user needs to execute the command.

-Commands that are prefixed with the “#” prompt must be ran by a user with sudo or root privileges.

-As a safety precaution, ALWAYS make a backup of the database and the oracle software before you start.


1. Prerequisites and Preparations


1.1 Download the following software:

-19.3 GI and DB from https://www.oracle.com/database/technologies/oracle19c-linux-downloads.html

-The latest version of oracle-database-preinstall-19c for Linux 7 from https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.

-The latest OPatch from Oracle Support. We will be using OPatch Version 12.2.0.1.28 in this exercise.

-The latest Oracle’s Database Pre-Upgrade Utility.

-The latest Autonomous Health Framework.

-The latest Cluster Verification Utility. We will be using the Jan 2022 Patch 30839369 in this exercise.

-Download the latest 12.1.0.2 GI/DB Patchsets.

 

1.2 Check the installed patches in your DB Home. If they are older than 12.1.0.2.190416 (April 2019), you will need to download and apply Patch 28553832 for your database. Or alternatively, as we will be doing here, apply the latest GI/DB Patchsets for 12.1.0.2.


# su - oracle
$ $ORACLE_HOME/OPatch/opatch lsinventory|grep -A3 "^Patch.*applied"
Patch  21948354     : applied on Thu Dec 08 15:13:21 PST 2016
Unique Patch ID:  19553095
Patch description:  "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
   Created on 20 Dec 2015, 23:39:33 hrs PST8PDT
--
Patch  21948344     : applied on Thu Dec 08 15:12:55 PST 2016
Unique Patch ID:  19577088
Patch description:  "OCW Patch Set Update : 12.1.0.2.160119 (21948344)"
   Created on 13 Jan 2016, 10:24:27 hrs PST8PDT

1.3 Check Patches in GI Home.


# su - grid
$ $ORACLE_HOME/OPatch/opatch lsinventory|grep -A3 "^Patch.*applied"
Patch  21948354     : applied on Thu Dec 08 15:08:00 PST 2016
Unique Patch ID:  19553095
Patch description:  "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
   Created on 20 Dec 2015, 23:39:33 hrs PST8PDT
--
Patch  21948344     : applied on Thu Dec 08 15:07:26 PST 2016
Unique Patch ID:  19577088
Patch description:  "OCW Patch Set Update : 12.1.0.2.160119 (21948344)"
   Created on 13 Jan 2016, 10:24:27 hrs PST8PDT
--
Patch  21948341     : applied on Thu Dec 08 15:06:52 PST 2016
Unique Patch ID:  19577084
Patch description:  "ACFS Patch Set Update : 12.1.0.2.160119 (21948341)"
   Created on 2 Dec 2015, 05:52:43 hrs PST8PDT
--
Patch  21436941     : applied on Thu Dec 08 15:06:04 PST 2016
Unique Patch ID:  19245012
Patch description:  "WLM Patch Set Update: 12.1.0.2.5 (21436941)"
   Created on 13 Aug 2015, 04:00:40 hrs PST8PDT

1.4 Check current GI software and release version.


# su - grid
$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.1.0.2.0]
$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]

1.5 Verify all objects in all containers are valid before proceeding.

Note: In my database, I have 3 containers: CDB$ROOT (root container), PDB$SEED (seed database), PROD1PDB (pluggable database).


# su - oracle
$ sqlplus / as sysdba

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PROD1PDB                       READ WRITE NO

SQL> select count(*), status from dba_objects group by status;

  COUNT(*) STATUS
---------- ---------------------
       228 INVALID
     86350 VALID

If you have any invalid objects, recompile them.


SQL> @?/rdbms/admin/utlrp.sql

SQL> select count(*), status from dba_objects group by status;

  COUNT(*) STATUS
---------- ---------------
     86589 VALID

Check PROD1PDB.


SQL> alter session set container=PROD1PDB;

SQL> select count(*), status from dba_objects group by status;

  COUNT(*) STATUS
---------- ---------------
       305 INVALID
     87070 VALID

If you have any invalid objects, recompile them.


SQL> @?/rdbms/admin/utlrp.sql

SQL> select count(*), status from dba_objects group by status;
  COUNT(*) STATUS
---------- ---------------
     87375 VALID

Check PDB$SEED.


SQL> alter session set container=PDB$SEED;

Session altered.

SQL> select count(*), status from dba_objects group by status;

  COUNT(*) STATUS
---------- ---------------
     86058 VALID

The seed database is read only and should not have any invalid objects. But if it does have invalid objects and needs to be recompiled, we have to make it read-write, recompile the objects in it, then make it read-only again.


SQL> alter session set "_oracle_script" = true;

Session altered.

SQL> alter pluggable database pdb$seed close;

Pluggable database altered.

SQL> alter pluggable database pdb$seed open read write;

Pluggable database altered.

SQL> @?/rdbms/admin/utlrp.sql

SQL> alter pluggable database pdb$seed close;

Pluggable database altered.

SQL> alter pluggable database pdb$seed open read only;

Pluggable database altered.


2. Extracting GI/DB Software and Installing Prerequisites


2.1 Create 19c GI Home directory and extract GI software.


# mkdir -p /oracle/oraBase/19.0.0/grid
# chown -R grid.oinstall /oracle/oraBase/19.0.0
# chmod 775 /oracle/oraBase/19.0.0

# su - grid
$ unzip -q /usr/software/oracle/LINUX.X64_193000_grid_home.zip

2.2 Create 19c DB Home directory and extract DB software.


# mkdir -p /oracle/oraBase/oracle/19.0.0/database
# chown -R oracle.oinstall /oracle/oraBase/oracle/19.0.0
# chmod -R 775 /oracle/oraBase/oracle/19.0.0

# su - oracle
$ cd /oracle/oraBase/oracle/19.0.0/database
$ unzip -q /usr/software/oracle/LINUX.X64_193000_db_home.zip

2.3 Remove old OPatch and extract the latest OPatch in GI and DB Homes for both 12c and 19c.


WARNING: To avoid accidental deletions, always take extra care with the “rm -rf” command.


2.3.1 GI Homes


# su - grid
$ rm -rf /oracle/oraBase/12.1.0.2/grid/OPatch/*
$ unzip -q /usr/software/oracle/p6880880_210000_Linux-x86-64.zip -d /oracle/oraBase/12.1.0.2/grid
$ /oracle/oraBase/12.1.0.2/grid/OPatch/opatch version
OPatch Version: 12.2.0.1.28

OPatch succeeded.

$ rm -rf /oracle/oraBase/19.0.0/grid/OPatch/*
$ unzip -q /usr/software/oracle/p6880880_210000_Linux-x86-64.zip -d /oracle/oraBase/19.0.0/grid
$ /oracle/oraBase/19.0.0/grid/OPatch/opatch version
OPatch Version: 12.2.0.1.28

OPatch succeeded.

2.3.2 DB Homes.


# su - oracle
$ rm -rf /oracle/oraBase/oracle/12.1.0.2/database/OPatch/*
$ unzip -q /usr/software/oracle/p6880880_210000_Linux-x86-64.zip -d /oracle/oraBase/oracle/12.1.0.2/database
$ /oracle/oraBase/oracle/12.1.0.2/database/OPatch/opatch version
OPatch Version: 12.2.0.1.28

OPatch succeeded.

$ rm -rf /oracle/oraBase/oracle/19.0.0/database/OPatch/*
$ unzip -q /usr/software/oracle/p6880880_210000_Linux-x86-64.zip -d /oracle/oraBase/oracle/19.0.0/database
$ /oracle/oraBase/oracle/19.0.0/database/OPatch/opatch version
OPatch Version: 12.2.0.1.28

OPatch succeeded.

2.4 Install the latest oracle-database-preinstall-19c rpm for your version of linux.


# yum install oracle-database-preinstall-19c-1.0-3.el7.x86_64.rpm -y
Loaded plugins: ulninfo
Examining oracle-database-preinstall-19c-1.0-3.el7.x86_64.rpm: oracle-database-preinstall-19c-1.0-3.el7.x86_64
Marking oracle-database-preinstall-19c-1.0-3.el7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package oracle-database-preinstall-19c.x86_64 0:1.0-3.el7 will be installed
--> Processing Dependency: smartmontools for package: oracle-database-preinstall-19c-1.0-3.el7.x86_64
--> Running transaction check
---> Package smartmontools.x86_64 1:6.5-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================
 Package                             Arch        Version           Repository                                             Size
===============================================================================================================================
Installing:
 oracle-database-preinstall-19c      x86_64      1.0-3.el7         /oracle-database-preinstall-19c-1.0-3.el7.x86_64       76 k
Installing for dependencies:
 smartmontools                       x86_64      1:6.5-1.el7       ol7u6                                                 460 k

Transaction Summary
===============================================================================================================================
Install  1 Package (+1 Dependent package)

Total size: 536 k
Total download size: 460 k
Installed size: 1.8 M
Downloading packages:
smartmontools-6.5-1.el7.x86_64.rpm                                                                      | 460 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:smartmontools-6.5-1.el7.x86_64                                                                            1/2
  Installing : oracle-database-preinstall-19c-1.0-3.el7.x86_64                                                             2/2
  Verifying  : 1:smartmontools-6.5-1.el7.x86_64                                                                            1/2
  Verifying  : oracle-database-preinstall-19c-1.0-3.el7.x86_64                                                             2/2

Installed:
  oracle-database-preinstall-19c.x86_64 0:1.0-3.el7

Dependency Installed:
  smartmontools.x86_64 1:6.5-1.el7

Complete!

Note: For yum to automatically resolve the dependencies, you need internet access to the public oracle linux yum repository. If you don’t have access to the repository, you can set a local yum using the OEL 7 DVD. See Setting Up Local Yum Repository Using Linux DVD or ISO on OEL 6/RHEL 6, the instruction was written for Linux 6 but it will also work for Linux 7.



2.5 Extract Database Pre-Upgrade Utility into the new 19c DB Home.


# su - oracle
$ unzip -oq /usr/software/oracle/preupgrade_19_cbuild_12_lf.zip -d /oracle/oraBase/oracle/19.0.0/database/rdbms/admin

2.6 Install Oracle’s Autonomous Health Framework (AHF)


2.6.1 Create a directory to hold AHF.


# mkdir -p /oratools/AHF
# chmod -R 777 /oratools

2.6.2 Create a directory to hold AHF outputs.


# mkdir -p /var/log/root/orasrv1/ahf
# chmod -R o+rw /var/log/root

2.6.3 Unzip AHF.


# cd /oratools/AHF
# unzip -q /usr/software/oracle/AHF-LINUX_v21.4.1.zip

2.6.4 Install AHF.


# ./ahf_setup -ahf_loc /oratools/AHF -data_dir /var/log/root/orasrv1/ahf

AHF Installer for Platform Linux Architecture x86_64

AHF Installation Log : /tmp/ahf_install_214100_3959_2022_04_05-16_07_06.log

Starting Autonomous Health Framework (AHF) Installation

AHF Version: 21.4.1 Build Date: 202201112133

Removing Standalone AHF : /oratools/AHF/oracle.ahf

AHF Location : /oratools/AHF/oracle.ahf

AHF Data Directory : /var/log/root/orasrv1/ahf/oracle.ahf/data

Do you want to add AHF Notification Email IDs ? [Y]|N : N

Extracting AHF to /oratools/AHF/oracle.ahf

Configuring TFA Services

Discovering Nodes and Oracle Resources

Not generating certificates as GI discovered

Starting TFA Services

.---------------------------------------------------------------------------.
| Host    | Status of TFA | PID  | Port | Version    | Build ID             |
+---------+---------------+------+------+------------+----------------------+
| orasrv1 | RUNNING       | 5405 | 5000 | 21.4.1.0.0 | 21410020220111213353 |
'---------+---------------+------+------+------------+----------------------'

Running TFA Inventory...

Please add world execute permissions for all directories above /var/log/root/orasrv1/ahf to enable TFA Non-Root Access

Adding default users to TFA Access list...

.--------------------------------------------------------------------------.
|                       Summary of AHF Configuration                       |
+-----------------+--------------------------------------------------------+
| Parameter       | Value                                                  |
+-----------------+--------------------------------------------------------+
| AHF Location    | /oratools/AHF/oracle.ahf                               |
| TFA Location    | /oratools/AHF/oracle.ahf/tfa                           |
| Orachk Location | /oratools/AHF/oracle.ahf/orachk                        |
| Data Directory  | /var/log/root/orasrv1/ahf/oracle.ahf/data              |
| Repository      | /var/log/root/orasrv1/ahf/oracle.ahf/data/repository   |
| Diag Directory  | /var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/diag |
'-----------------+--------------------------------------------------------'


Starting orachk scheduler from AHF ...

AHF binaries are available in /oratools/AHF/oracle.ahf/bin

AHF is successfully installed

Do you want AHF to store your My Oracle Support Credentials for Automatic Upload ? Y|[N] : N

Moving /tmp/ahf_install_214100_3959_2022_04_05-16_07_06.log to /var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/diag/ahf/

# tfactl status

.----------------------------------------------------------------------------------------------.
| Host    | Status of TFA | PID  | Port | Version    | Build ID             | Inventory Status |
+---------+---------------+------+------+------------+----------------------+------------------+
| orasrv1 | RUNNING       | 5405 | 5000 | 21.4.1.0.0 | 21410020220111213353 | COMPLETE         |

2.7 Cluster Verification Utility (cvu)


2.7.1 cvu directory is empty so we will need to extract the latest cvu in there.


# cd /oratools/AHF/oracle.ahf/common/cvu/
# ls -l
total 0


# unzip -q /usr/software/oracle/cvupack_linux_ol7_x86_64.zip
# ls -l
total 100
drwxrwxr-x 2 root root 4096 Jan 24 00:36 bin
drwxrwxr-x 4 root root 4096 Jan 24 00:36 clone
drwxrwxr-x 7 root root 4096 Jan 24 00:36 cv
drwxrwxr-x 4 root root 4096 Jan 24 00:36 deinstall
drwxrwxr-x 3 root root 4096 Jan 24 00:36 diagnostics
-r-xr-xr-x 1 root root  852 Jan 24 00:36 env.ora
drwxrwxr-x 6 root root 4096 Jan 24 00:36 install
drwxrwxr-x 3 root root 4096 Jan 24 00:36 jdbc
drwxrwxr-x 7 root root 4096 Jan 24 00:36 jdk
drwxrwxr-x 2 root root 4096 Jan 24 00:36 jlib
drwxrwxr-x 3 root root 4096 Jan 24 00:36 ldap
drwxrwxr-x 2 root root 4096 Jan 24 00:36 lib
drwxrwxr-x 6 root root 4096 Jan 24 00:36 network
drwxrwxr-x 3 root root 4096 Jan 24 00:36 nls
drwxrwxr-x 4 root root 4096 Jan 24 00:36 oracore
drwxrwxr-x 3 root root 4096 Jan 24 00:36 oss
drwxrwxr-x 8 root root 4096 Jan 24 00:36 oui
drwxrwxr-x 4 root root 4096 Jan 24 00:36 rdbms
-rw------- 1 root root 4679 Jan 31 20:40 readme.txt
drwxrwxr-x 3 root root 4096 Jan 24 00:36 srvm
drwxrwxr-x 3 root root 4096 Jan 24 00:36 suptools
drwxrwxr-x 3 root root 4096 Jan 24 00:36 ucp
drwxrwxr-x 2 root root 4096 Jan 24 00:36 utl
drwxrwxr-x 3 root root 4096 Jan 24 00:36 xdk

2.7.2 Install cvuqdisk


# yum install /oratools/AHF/oracle.ahf/common/cvu/cv/rpm/cvuqdisk-1.0.10-1.rpm
Loaded plugins: ulninfo
Examining /oratools/AHF/oracle.ahf/common/cvu/cv/rpm/cvuqdisk-1.0.10-1.rpm: cvuqdisk-1.0.10-1.x86_64
Marking /oratools/AHF/oracle.ahf/common/cvu/cv/rpm/cvuqdisk-1.0.10-1.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package cvuqdisk.x86_64 0:1.0.10-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================
 Package                     Arch                      Version                     Repository                             Size
===============================================================================================================================
Installing:
 cvuqdisk                    x86_64                    1.0.10-1                    /cvuqdisk-1.0.10-1                     22 k

Transaction Summary
===============================================================================================================================
Install  1 Package

Total size: 22 k
Installed size: 22 k
Is this ok [y/d/N]: N
Exiting on user command
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx.2022-04-05.16-29.k_0JC3.yumtx
# yum install /oratools/AHF/oracle.ahf/common/cvu/cv/rpm/cvuqdisk-1.0.10-1.rpm -y
Loaded plugins: ulninfo
Examining /oratools/AHF/oracle.ahf/common/cvu/cv/rpm/cvuqdisk-1.0.10-1.rpm: cvuqdisk-1.0.10-1.x86_64
Marking /oratools/AHF/oracle.ahf/common/cvu/cv/rpm/cvuqdisk-1.0.10-1.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package cvuqdisk.x86_64 0:1.0.10-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================
 Package                     Arch                      Version                     Repository                             Size
===============================================================================================================================
Installing:
 cvuqdisk                    x86_64                    1.0.10-1                    /cvuqdisk-1.0.10-1                     22 k

Transaction Summary
===============================================================================================================================
Install  1 Package

Total size: 22 k
Installed size: 22 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : cvuqdisk-1.0.10-1.x86_64                                                                                    1/1
  Verifying  : cvuqdisk-1.0.10-1.x86_64                                                                                    1/1

Installed:
  cvuqdisk.x86_64 0:1.0.10-1

Complete!

2.8 Run orachk to check upgrade readiness for 12c Home.


# cd /oratools/AHF/oracle.ahf/orachk
# ./orachk -u -o pre
Enter upgrade target version (valid versions are 11.2.0.4.0, 12.1.0.2.0, 12.2.0.1.0, 18.0.0.0.0, 19.0.0.0.0, 21.0.0.0.0) 3 tries left:-  19.0.0.0.0

Clusterware stack is running from /oracle/oraBase/12.1.0.2/grid. Is this the correct Clusterware Home?[y/n][y] y

List of running databases

1. PROD
2. None of above

Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1].
.  .
.  .

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

.  .  . . . .
.  .  . . . .  .  .  .  .  .  .  .  .  .  .  .  .
-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status
-------------------------------------------------------------------------------------------------------
  Host Name       CRS Installed       ASM HOME  RDBMS Installed    CRS UP    ASM UP  RDBMS UP    DB Instance Name
-------------------------------------------------------------------------------------------------------
    orasrv1                    Yes           No          Yes          Yes      Yes      Yes                PROD
-------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------------------
                                       Installed components summary
---------------------------------------------------------------------------------------------------------------------------------
             GI_HOME                              ORACLE_HOME                                               Database Names
---------------------------------------------------------------------------------------------------------------------------------
/oracle/oraBase/12.1.0.2/grid - 12.1.0.2.0        /oracle/oraBase/oracle/12.1.0.2/database - 12.1.0.2.0                         PROD

---------------------------------------------------------------------------------------------------------------------------------
. . .

Copying plug-ins

. .
.  .  .  .  .  .



Checking Oracle Grid Infrastructure and RDBMS upgrade readiness to version 190000


.

Collections and audit checks log file is
/var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/orachk/user_root/output/orachk_orasrv1_PROD_040522_172645/log/orachk.log

============================================================
                Node name - orasrv1
============================================================
. . . . . .


 Collecting - /proc/cmdline
 Collecting - /proc/modules
 Collecting - CPU Information
 Collecting - Clusterware OCR healthcheck
 Collecting - DiskFree Information
 Collecting - DiskMount Information
 Collecting - Kernel parameters
 Collecting - Maximum number of semaphore sets on system
 Collecting - Maximum number of semaphores on system
 Collecting - Maximum number of semaphores per semaphore set
 Collecting - Memory Information
 Collecting - NUMA Configuration
 Collecting - Network Interface Configuration
 Collecting - Network Performance
 Collecting - Network Service Switch
 Collecting - OS Packages
 Collecting - OS version
 Collecting - Patches for Grid Infrastructure
 Collecting - Patches for RDBMS Home
 Collecting - number of semaphore operations per semop system call

Starting to run root collections in background

 Collecting - slabinfo
 Collecting - Root user limits
 Collecting - Disk Information
 Collecting - Grid Infastructure user shell limits configuration
 Collecting - umask setting for GI owner


Data collections completed. Checking best practices on orasrv1.
------------------------------------------------------------



 INFO =>     Take Note for pre-11g Clients connecting to latest database version
 INFO =>     Unattended Installation and Configuration Options
 INFO =>     Patching Notes by Release
 WARNING =>  Linux swap configuration does not meet recommendation
 WARNING =>  Cluster name does not adhere to RFC 952 Department of Defense internet host table specification
 WARNING =>  Oracle patch 19855835 is not applied on RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Pre-requisite Oracle Patch 21255373 (cssd : duplicate response in group data update) does not exist in RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Oracle Patch 21626377 is not applied on RDBMS home /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Pre-requisite Oracle Patch 21255373 (cssd : duplicate response in group data update) does not exist in GRID_HOME /oracle/oraBase/12.1.0.2/grid
 WARNING =>  Oracle Patch 21780146 is not applied on RDBMS home /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Oracle patch 20898997 is not applied on RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Oracle patch 20348910 is not applied on RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Oracle patch 21856522 is not applied on RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Oracle patch 20958816 is not applied on RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  RECOMMENDATION: Apply Latest PSU/RU to source GRID_HOME and RDBMS_HOME prior to upgrade
 INFO =>     Review results of the Database AutoUpgrade utility for PROD
 INFO =>     Public interface is not configured in OCR
 INFO =>     Minimizing Planned Downtime for Upgrades and Patching
 INFO =>     Upgrade Related References
 INFO =>     Capture performance baseline, backup important configuration files and stop all batch, cron, DBMS_JOBS and DBMS_SCHEDULER jobs
 INFO =>     Database feature usage statistics for PROD
Best Practice checking completed. Checking recommended patches on orasrv1
--------------------------------------------------------------------------------
Collecting patch inventory on CRS_HOME /oracle/oraBase/12.1.0.2/grid
Collecting patch inventory on ORACLE_HOME /oracle/oraBase/oracle/12.1.0.2/database
--------------------------------------------------------------------------------
1 Recommended CRS patches for 121020 from /oracle/oraBase/12.1.0.2/grid on orasrv1
--------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description
--------------------------------------------------------------------------------
26983807 no          no    /oracle/oraBase/oracle/12.1.0.2/database DBWLM Patch Set Update 12.1.0.2.180116                    
32758932 no          no    /oracle/oraBase/oracle/12.1.0.2/database OCW Patch Set Update 12.1.0.2.210720                      
33112931 no          no    /oracle/oraBase/oracle/12.1.0.2/database ACFS Patch Set Update 12.1.0.2.210720                      
33128590 no          no    /oracle/oraBase/oracle/12.1.0.2/database Database Patch Set Update 12.1.0.2.211019                  
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
1 Recommended RDBMS patches for 121020 from /oracle/oraBase/oracle/12.1.0.2/database on orasrv1
--------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description
--------------------------------------------------------------------------------
32758932 no               merge                OCW Patch Set Update 12.1.0.2.210720
33128590 no               merge                Database Patch Set Update 12.1.0.2.211019
--------------------------------------------------------------------------------

A Cluster Verification Utility (CVU) Exception occurred while running CVU on node orasrv1. orachk is skipping CVU results from node orasrv1.

------------------------------------------------------------
Detailed report (html) -  /var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/orachk/user_root/output/orachk_orasrv1_PROD_040522_172645/orachk_orasrv1_PROD_040522_172645.html



UPLOAD [if required] - /var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/orachk/user_root/output/orachk_orasrv1_PROD_040522_172645.zip

2.9 Applying the recommended fixes for the issues founded by orachk.


2.9.1 Increasing the Linux swap to correct the warning below.


 WARNING =>  Linux swap configuration does not meet recommendation

There are various ways to add swap space. Example below creates a 16G swap file called swapfile1 under /swapfiles in the root directory. You can put in any directory that has sufficient free space.


# mkdir /swapfiles

# dd if=/dev/zero of=/swapfiles/swapfile1 bs=1M count=16000
16000+0 records in
16000+0 records out
16777216000 bytes (17 GB) copied, 65.294 s, 257 MB/s

# mkswap /swapfiles/swapfile1
Setting up swapspace version 1, size = 16383996 KiB
no label, UUID=3644776d-9979-4305-b83b-613ec538d0e1

# swapon /swapfiles/swapfile1

# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       4194300 0       -1
/swapfiles/swapfile1                    file    16383996        0       -2

2.9.2 Per the recommendation from the pre-upgrade check, I will need to install the latest PSU/RU in Oracle 12c GI/DB Home before upgrading.


 WARNING =>  RECOMMENDATION: Apply Latest PSU/RU to source GRID_HOME and RDBMS_HOME prior to upgrade"

For help with installing oracle PSU/RU, see How to Install the Latest Oracle 19c Update Patches – Opatch/OJVM/GI/DB/JDK on Oracle 19c on Linux. You can use the same instructions for patching 12c. You need to follow up to the end of step 3 on that page, which installs the GI/DB RU, to satisfied the pre-upgrade check.


2.9.3 Run the orachk again after the applying the fixes.


# cd /oratools/AHF/oracle.ahf/orachk/
# ./orachk -u -o pre
Enter upgrade target version (valid versions are 11.2.0.4.0, 12.1.0.2.0, 12.2.0.1.0, 18.0.0.0.0, 19.0.0.0.0, 21.0.0.0.0) 3 tries left:-  19.0.0.0.0

Clusterware stack is running from /oracle/oraBase/12.1.0.2/grid. Is this the correct Clusterware Home?[y/n][y]

List of running databases

1. PROD
2. None of above

Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1].
.  .
.  .

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

.  .  . . . .
.  .  . . . .  .  .  .  .  .  .  .  .  .  .  .  .
-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status
-------------------------------------------------------------------------------------------------------
  Host Name       CRS Installed       ASM HOME  RDBMS Installed    CRS UP    ASM UP  RDBMS UP    DB Instance Name
-------------------------------------------------------------------------------------------------------
    orasrv1                    Yes           No          Yes          Yes      Yes      Yes                PROD
-------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------------------
                                       Installed components summary
---------------------------------------------------------------------------------------------------------------------------------
             GI_HOME                              ORACLE_HOME                                               Database Names
---------------------------------------------------------------------------------------------------------------------------------
/oracle/oraBase/12.1.0.2/grid - 12.1.0.2.0        /oracle/oraBase/oracle/12.1.0.2/database - 12.1.0.2.0                         PROD

---------------------------------------------------------------------------------------------------------------------------------
. . .

Copying plug-ins

. .
.  .  .  .  .  .



Checking Oracle Grid Infrastructure and RDBMS upgrade readiness to version 190000


.

Collections and audit checks log file is
/var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/orachk/user_root/output/orachk_orasrv1_PROD_040722_101633/log/orachk.log

============================================================
                Node name - orasrv1
============================================================
. . . . . .


 Collecting - /proc/cmdline
 Collecting - /proc/modules
 Collecting - CPU Information
 Collecting - Clusterware OCR healthcheck
 Collecting - DiskFree Information
 Collecting - DiskMount Information
 Collecting - Kernel parameters
 Collecting - Maximum number of semaphore sets on system
 Collecting - Maximum number of semaphores on system
 Collecting - Maximum number of semaphores per semaphore set
 Collecting - Memory Information
 Collecting - NUMA Configuration
 Collecting - Network Interface Configuration
 Collecting - Network Performance
 Collecting - Network Service Switch
 Collecting - OS Packages
 Collecting - OS version
 Collecting - Patches for Grid Infrastructure
 Collecting - Patches for RDBMS Home
 Collecting - number of semaphore operations per semop system call

Starting to run root collections in background

 Collecting - slabinfo
 Collecting - Grid Infastructure user shell limits configuration
 Collecting - Root user limits
 Collecting - Disk Information
 Collecting - umask setting for GI owner


Data collections completed. Checking best practices on orasrv1.
------------------------------------------------------------



 INFO =>     Take Note for pre-11g Clients connecting to latest database version
 INFO =>     Unattended Installation and Configuration Options
 INFO =>     Patching Notes by Release
 WARNING =>  Cluster name does not adhere to RFC 952 Department of Defense internet host table specification
 WARNING =>  Oracle patch 19855835 is not applied on RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  Oracle patch 20348910 is not applied on RDBMS_HOME /oracle/oraBase/oracle/12.1.0.2/database
 WARNING =>  RECOMMENDATION: Apply Latest PSU/RU to source GRID_HOME and RDBMS_HOME prior to upgrade
 INFO =>     Review results of the Database AutoUpgrade utility for PROD
 INFO =>     Public interface is not configured in OCR
 INFO =>     Minimizing Planned Downtime for Upgrades and Patching
 INFO =>     Upgrade Related References
 INFO =>     Capture performance baseline, backup important configuration files and stop all batch, cron, DBMS_JOBS and DBMS_SCHEDULER jobs
 INFO =>     Database feature usage statistics for PROD
Best Practice checking completed. Checking recommended patches on orasrv1
--------------------------------------------------------------------------------
Collecting patch inventory on CRS_HOME /oracle/oraBase/12.1.0.2/grid
Collecting patch inventory on ORACLE_HOME /oracle/oraBase/oracle/12.1.0.2/database
--------------------------------------------------------------------------------
1 Recommended CRS patches for 121020 from /oracle/oraBase/12.1.0.2/grid on orasrv1
--------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description
--------------------------------------------------------------------------------
 33477199yes         yes   /oracle/oraBase/oracle/12.1.0.2/database DBWLM Patch Set Update 12.1.0.2.180116                    
32758932 yes         yes   /oracle/oraBase/oracle/12.1.0.2/database OCW Patch Set Update 12.1.0.2.210720                      
 33477199yes         yes   /oracle/oraBase/oracle/12.1.0.2/database ACFS Patch Set Update 12.1.0.2.210720                      
33128590 yes         yes   /oracle/oraBase/oracle/12.1.0.2/database Database Patch Set Update 12.1.0.2.211019                  
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
1 Recommended RDBMS patches for 121020 from /oracle/oraBase/oracle/12.1.0.2/database on orasrv1
--------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description
--------------------------------------------------------------------------------
32758932 yes              merge                OCW Patch Set Update 12.1.0.2.210720
33128590 yes              merge                Database Patch Set Update 12.1.0.2.211019
--------------------------------------------------------------------------------

A Cluster Verification Utility (CVU) Exception occurred while running CVU on node orasrv1. orachk is skipping CVU results from node orasrv1.

------------------------------------------------------------
Detailed report (html) -  /var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/orachk/user_root/output/orachk_orasrv1_PROD_040722_101633/orachk_orasrv1_PROD_040722_101633.html



UPLOAD [if required] - /var/log/root/orasrv1/ahf/oracle.ahf/data/orasrv1/orachk/user_root/output/orachk_orasrv1_PROD_040722_101633.zip

2.10 Validating the Grid Infrastructure


# su - grid
Last login: Thu Apr  7 10:46:36 PDT 2022
$ cd /oracle/oraBase/19.0.0/grid
$ ./runcluvfy.sh stage -pre hacfg -fixup -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       27.2341GB (2.8557072E7KB)  8GB (8388608.0KB)         passed
Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       13.4206GB (1.4072552E7KB)  50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       19.625GB (2.0578296E7KB)  16GB (1.6777216E7KB)      passed
Verifying Swap Size ...PASSED
Verifying Free Space: orasrv1:/usr,orasrv1:/etc,orasrv1:/sbin ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              orasrv1       /             23.2471GB     25MB          passed
  /etc              orasrv1       /             23.2471GB     25MB          passed
  /sbin             orasrv1       /             23.2471GB     10MB          passed
Verifying Free Space: orasrv1:/usr,orasrv1:/etc,orasrv1:/sbin ...PASSED
Verifying Free Space: orasrv1:/var ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /var              orasrv1       /var          7.4531GB      5MB           passed
Verifying Free Space: orasrv1:/var ...PASSED
Verifying Free Space: orasrv1:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              orasrv1       /tmp          8.8994GB      1GB           passed
Verifying Free Space: orasrv1:/tmp ...PASSED
Verifying User Existence: grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  orasrv1       passed                    exists(921)

  Verifying Users With Same UID: 921 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: dba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  orasrv1       passed                    exists
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  orasrv1       passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: dba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  orasrv1           yes           yes           yes           passed
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       3.8.13-98.7.1.el7uek.x86_64  4.1.12                    failed
Verifying OS Kernel Version ...FAILED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           256           256           250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           32000         32000         32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           100           100           100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           4398046511104  4398046511104  14621220864   passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           1073741824    1073741824    1073741824    passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           4194304       4194304       262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           262144        262144        262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           1048576       1048576       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           1048576       1048576       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  orasrv1           1             1             1             passed
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       kmod(x86_64)-20-23.0.1.el7  kmod(x86_64)-20-21        passed
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       kmod-libs(x86_64)-20-23.0.1.el7  kmod-libs(x86_64)-20-21   passed
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       binutils-2.27-34.base.0.1.el7  binutils-2.23.52.0.1      passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       libgcc(x86_64)-4.8.5-36.0.1.el7  libgcc(x86_64)-4.8.2      passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       libstdc++(x86_64)-4.8.5-36.0.1.el7  libstdc++(x86_64)-4.8.2   passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       libstdc++-devel(x86_64)-4.8.5-36.0.1.el7  libstdc++-devel(x86_64)-4.8.2  passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       sysstat-10.1.5-17.el7     sysstat-10.1.5            passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       make-3.82-23.el7          make-3.82                 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       glibc(x86_64)-2.17-260.0.9.el7  glibc(x86_64)-2.17        passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       glibc-devel(x86_64)-2.17-260.0.9.el7  glibc-devel(x86_64)-2.17  passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       nfs-utils-1.3.0-0.61.0.1.el7  nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       smartmontools-6.5-1.el7   smartmontools-6.2-4       passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  orasrv1       net-tools-2.0-0.24.20131004git.el7  net-tools-2.0-0.17        passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  orasrv1                               passed
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.


Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Verifying OS Kernel Version ...FAILED
orasrv1: Kernel of proper version is not found on node "orasrv1" [Expected =
"4.1.12" ; Found = "3.8.13-98.7.1.el7uek.x86_64"]


CVU operation performed:      stage -pre hacfg
Date:                         Apr 7, 2022 10:56:06 AM
CVU home:                     /oracle/oraBase/19.0.0/grid/
User:                         grid

I will ignore the message about the kernel as it is not necessary to upgrade the kernel on my system.


3. Upgrade Grid Infrastructure (Oracle Restart) to 19c


3.1 Check the ASM compatibility settings for all your diskgroups. They are required to be at 11.2.0.2.0 or higher.


# su - grid

$ sqlplus / as sysasm

SQL> col name for a10
SQL> col type for a10
SQL> col compatibility for a15
SQL> col database_compatibility for a15
SQL> select name,type,compatibility,database_compatibility from v$asm_diskgroup;

NAME       TYPE       COMPATIBILITY   DATABASE_COMPAT
---------- ---------- --------------- ---------------
FRA        EXTERN     10.1.0.0.0      10.1.0.0.0
DATA       EXTERN     10.1.0.0.0      10.1.0.0.0

3.2 The settings above are too low. We will set both ASM and database compatibilities to be the same level at 12.1.0.0.0.


SQL> alter diskgroup FRA set attribute 'compatible.asm'='12.1.0.0.0';

Diskgroup altered.

SQL> alter diskgroup DATA set attribute 'compatible.asm'='12.1.0.0.0';

Diskgroup altered.

SQL> alter diskgroup FRA set attribute 'compatible.rdbms'='12.1.0.0.0';

Diskgroup altered.

SQL> alter diskgroup DATA set attribute 'compatible.rdbms'='12.1.0.0.0';

Diskgroup altered.
SQL> select name,type,compatibility,database_compatibility from v$asm_diskgroup;

NAME       TYPE       COMPATIBILITY   DATABASE_COMPAT
---------- ---------- --------------- ---------------
FRA        EXTERN     12.1.0.0.0      12.1.0.0.0
DATA       EXTERN     12.1.0.0.0      12.1.0.0.0

3.3 Check the status of DB services.


# su - oracle
$ srvctl status service -d PROD
Service ACEV is running
Service ADS is running
Service CVIEW is running
Service JMS is running
Service WFM is running

3.4 Stop the database.


$ srvctl stop database -d PROD

3.5 Execute the gridSetup.sh from the 19c GI Home


# su - grid
$ cd /oracle/oraBase/19.0.0/grid/
$ ./gridSetup.sh -silent -ignorePrereq -responseFile /home/grid/19c_grid_upgrade.rsp
Launching Oracle Grid Infrastructure Setup Wizard...

The response file for this session can be found at:
 /oracle/oraBase/19.0.0/grid/install/response/grid_2022-04-07_02-06-49PM.rsp

You can find the log of this install session at:
 /oracle/oraInventory/logs/GridSetupActions2022-04-07_02-06-49PM/gridSetupActions2022-04-07_02-06-49PM.log

As a root user, execute the following script(s):
        1. /oracle/oraBase/19.0.0/grid/rootupgrade.sh

Execute /oracle/oraBase/19.0.0/grid/rootupgrade.sh on the following nodes:
[orasrv1]


Successfully Setup Software.
As install user, execute the following command to complete the configuration.
        /oracle/oraBase/19.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /home/grid/19c_grid_upgrade.rsp [-silent]

Sample Response File: 19c_grid_upgrade_rsp.txt (611 downloads )

Note: To use the sample response file, change the values (ie. INVENTORY_LOCATION, ORACLE_BASE, etc.) to match your environment. Rename it to 19c_grid_upgrade.rsp. I had to add .txt to the end of the file name so WordPress would allow me to upload it.


3.5.1 As root, run rootupgrade.sh:


[root@orasrv1 oracle]# /oracle/oraBase/19.0.0/grid/rootupgrade.sh
Check /oracle/oraBase/19.0.0/grid/install/root_orasrv1_2022-04-07_14-11-41-993082143.log for the output of root script

3.5.2 As grid, run gridSetup.sh:


[root@orasrv1 oracle]# su - grid
Last login: Thu Apr  7 14:15:42 PDT 2022 on pts/1
[grid@orasrv1 ~]$ /oracle/oraBase/19.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /home/grid/19c_grid_upgrade.rsp -silent
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the logs of this session at:
/oracle/oraInventory/logs/GridSetupActions2022-04-07_02-40-23PM

You can find the log of this install session at:
 /oracle/oraInventory/logs/UpdateNodeList2022-04-07_02-40-23PM.log
You can find the log of this install session at:
 /oracle/oraInventory/logs/UpdateNodeList2022-04-07_02-40-23PM.log
Successfully Configured Software.

3.6 Our GI software should now be at 19c. Let’s verify.


3.6.1 Update your environment settings for ORACLE_HOME variable to point to 19c GI Home.


# su - grid
$ grep ORACLE_HOME .bash_profile
ORACLE_HOME=/oracle/oraBase/19.0.0/grid

$ env|grep ORA
ORACLE_SID=+ASM
ORACLE_BASE=/oracle/oraBase/grid
ORACLE_HOME=/oracle/oraBase/19.0.0/grid

3.6.2 Verify that the listener.ora file was copied to the new GI Home as part of the upgrade. If not, you can copy it from the 12c GI Home to the 19c GI Home.


$ ls -l /oracle/oraBase/19.0.0/grid/network/admin/listener.ora
-rw------- 1 grid oinstall 675 Apr  7 14:12 /oracle/oraBase/19.0.0/grid/network/admin/listener.ora

3.6.3 Check listener status


$ lsnrctl status

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 07-APR-2022 14:50:01

Copyright (c) 1991, 2019, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=orasrv1)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                07-APR-2022 23:15:31
Uptime                    0 days 0 hr. 34 min. 30 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/oraBase/19.0.0/grid/network/admin/listener.ora
Listener Log File         /oracle/oraBase/grid/diag/tnslsnr/orasrv1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=orasrv1)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=orasrv1)(PORT=1522)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
  Instance "+ASM", status READY, has 1 handler(s) for this service...
Service "+ASM_FRA" has 1 instance(s).
  Instance "+ASM", status READY, has 1 handler(s) for this service...
Service "PROD" has 1 instance(s).
  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

3.6.4 Check GI Software Version


$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [19.0.0.0.0]
$  crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

3.6.5 Check Oracle HAS Services


$ /oracle/oraBase/19.0.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       orasrv1                  STABLE
ora.FRA.dg
               ONLINE  ONLINE       orasrv1                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       orasrv1                  STABLE
ora.asm
               ONLINE  ONLINE       orasrv1                  Started,STABLE
ora.ons
               OFFLINE OFFLINE      orasrv1                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       orasrv1                  STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       orasrv1                  STABLE
ora.prod.acev.svc
      1        OFFLINE OFFLINE                               STABLE
ora.prod.ads.svc
      1        OFFLINE OFFLINE                               STABLE
ora.prod.cview.svc
      1        OFFLINE OFFLINE                               STABLE
ora.prod.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.prod.jms.svc
      1        OFFLINE OFFLINE                               STABLE
ora.prod.wfm.svc
      1        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------

3.6.6 Check Patches in GI Home


$ /oracle/oraBase/19.0.0/grid/OPatch/opatch lsinventory|grep -A3 "^Patch.*applied"
Patch  29585399     : applied on Thu Apr 18 00:36:24 PDT 2019
Unique Patch ID:  22840393
Patch description:  "OCW RELEASE UPDATE 19.3.0.0.0 (29585399)"
   Created on 9 Apr 2019, 19:12:47 hrs PST8PDT
--
Patch  29517247     : applied on Thu Apr 18 00:36:02 PDT 2019
Unique Patch ID:  22840392
Patch description:  "ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)"
   Created on 1 Apr 2019, 15:08:20 hrs PST8PDT
--
Patch  29517242     : applied on Thu Apr 18 00:35:39 PDT 2019
Unique Patch ID:  22862832
Patch description:  "Database Release Update : 19.3.0.0.190416 (29517242)"
   Created on 17 Apr 2019, 23:27:10 hrs PST8PDT
--
Patch  29401763     : applied on Thu Apr 18 00:35:16 PDT 2019
Unique Patch ID:  22759421
Patch description:  "TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)"
   Created on 11 Apr 2019, 22:26:25 hrs PST8PDT

4. Install Oracle Database 19c


4.1 Verify the database is down.


[root@orasrv1 oracle]# su - oracle

[oracle@orasrv1 ~]$ srvctl status database -d PROD
Database is not running.

4.1.1 If it is up, stop it.

Note: Change PROD to your database SID.


[oracle@orasrv1 ~]$ srvctl stop database -d PROD

4.2 Create a racdba group and add oracle and grid users to it. This is needed to satisfied the setting in my response file.


# groupadd -g 54330 racdba
# usermod -a -G racdba oracle
# usermod -a -G racdba grid

# id oracle
uid=901(oracle) gid=900(oinstall) groups=900(oinstall),901(dba),902(oper),903(backupdba),904(dgdba),905(kmdba),921(asmdba),923(asmadmin),54330(racdba)
# id grid
uid=921(grid) gid=900(oinstall) groups=900(oinstall),901(dba),921(asmdba),922(asmoper),923(asmadmin),54330(racdba)

4.3 Execute runInstaller from the 19c DB Home.


# su - oracle
$ cd /oracle/oraBase/oracle/19.0.0/database
$ ./runInstaller -ignorePrereq -waitforcompletion -silent -responseFile /home/oracle/19c_db_ee.rsp
Launching Oracle Database Setup Wizard...

The response file for this session can be found at:
 /oracle/oraBase/oracle/19.0.0/database/install/response/db_2022-04-07_03-13-05PM.rsp

You can find the log of this install session at:
 /oracle/oraInventory/logs/InstallActions2022-04-07_03-13-05PM/installActions2022-04-07_03-13-05PM.log

As a root user, execute the following script(s):
        1. /oracle/oraBase/oracle/19.0.0/database/root.sh

Execute /oracle/oraBase/oracle/19.0.0/database/root.sh on the following nodes:
[orasrv1]


Successfully Setup Software.

Sample Response File: 19c_db_ee_rsp.txt (607 downloads )

Note: To use the sample response file, change the values (ie. INVENTORY_LOCATION, ORACLE_BASE, etc.) to match your environment. Rename it to 19c_db_ee.rsp.


4.3.1 As root, run


# /oracle/oraBase/oracle/19.0.0/database/root.sh
Check /oracle/oraBase/oracle/19.0.0/database/install/root_orasrv1_2022-04-07_15-16-57-303083128.log for the output of root script

5. Upgrade Oracle Database to 19c


5.1 Verify ORACLE_HOME is currently set to 12c.


# su - oracle
$ echo $ORACLE_HOME
/oracle/oraBase/oracle/12.1.0.2/database

5.2 Start the database.


$ srvctl start db -d PROD

5.3 Run the preupgrade.jar from the 19c DB Home.


$ /oracle/oraBase/oracle/19.0.0/database/jdk/bin/java -jar /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/preupgrade.jar TERMINAL TEXT
Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 12 on 2022-04-07T15:23:22

Upgrade-To version: 19.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  PROD
     Container Name:  CDB$ROOT
       Container ID:  1
            Version:  12.1.0.2.0
     DB Patch Level:  DATABASE PATCH SET UPDATE 12.1.0.2.220118
         Compatible:  12.1.0.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  18
  Database log mode:  NOARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  JServer JAVA Virtual Machine           [to be upgraded]  VALID
  Oracle XDK for Java                    [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  OLAP Analytic Workspace                [to be upgraded]  VALID
  Oracle Label Security                  [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID
  Oracle Java Packages                   [to be upgraded]  VALID
  Oracle Multimedia                      [to be upgraded]  VALID
  Oracle OLAP API                        [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  1.  (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

      The database contains 1263 objects in the recycle bin.

      The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
  2.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 12.1.0.2
      Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
  3.  Here are ALL the components in this database registry:

      Component Current     Current     Original    Previous    Component
      CID       Version     Status      Version     Version     Schema
      --------- ----------- ----------- ----------- ----------- -----------
      APS       12.1.0.2.0  VALID                               SYS
      CATALOG   12.1.0.2.0  VALID                               SYS
      CATJAVA   12.1.0.2.0  VALID                               SYS
      CATPROC   12.1.0.2.0  VALID                               SYS
      CONTEXT   12.1.0.2.0  VALID                               CTXSYS
      JAVAVM    12.1.0.2.0  VALID                               SYS
      OLS       12.1.0.2.0  VALID                               LBACSYS
      ORDIM     12.1.0.2.0  VALID       12.1.0.2.0  12.1.0.2.0  ORDSYS
      OWM       12.1.0.2.0  VALID                               WMSYS
      RAC       12.1.0.2.0  OPTION OFF                          SYS
      XDB       12.1.0.2.0  VALID                               XDB
      XML       12.1.0.2.0  VALID                               SYS
      XOQ       12.1.0.2.0  VALID       12.1.0.2.0  12.1.0.2.0  SYS

      Review the information before upgrading.

  4.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      SYSAUX                            1090 MB      1138 MB
      SYSTEM                             730 MB      1046 MB

      Minimum tablespace sizes for upgrade are estimates.

  5.  No action needed.

      Using default parallel upgrade options, this CDB with 2 PDBs will first
      upgrade the CDB$ROOT, and then upgrade at most 2 PDBs at a time using 2
      parallel processes per PDB.

      The number of PDBs upgraded in parallel and the number of parallel
      processes per PDB can be adjusted as described in Database Upgrade Guide.

  6.  Check the Oracle Backup and Recovery User's Guide for information on how
      to manage an RMAN recovery catalog schema.

      If you are using a version of the recovery catalog schema that is older
      than that required by the RMAN client version, then you must upgrade the
      catalog schema.

      It is good practice to have the catalog schema the same or higher version
      than the RMAN client version you are using.

  7.  Here is a count of invalid objects by Oracle-maintained users:

      Oracle-Maintained User Name                 Number of INVALID Objects
      ---------------------------                 -------------------------
      None                                        None

      Review the information before upgrading.

  8.  Here is a count of invalid objects by Application users:

      Application User Name                       Number of INVALID Objects
      ---------------------------                 -------------------------
      None                                        None

      Review the information before upgrading.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PROD container CDB$ROOT
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/preupgrade_fixups.sq
    l

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  9.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 18 and the target 19 release
      ships with time zone file version 32.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 19 Oracle Database
      Globalization Support Guide.

  10. (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  11. Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 12.1.0.2
      Oracle Database SQL Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PROD container CDB$ROOT
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postupgrade_fixups.s
    ql


Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 12 on 2022-04-07T15:23:36

Upgrade-To version: 19.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  PROD
     Container Name:  PDB$SEED
       Container ID:  2
            Version:  12.1.0.2.0
     DB Patch Level:  DATABASE PATCH SET UPDATE 12.1.0.2.220118
         Compatible:  12.1.0.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  18
  Database log mode:  NOARCHIVELOG
           Readonly:  TRUE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  JServer JAVA Virtual Machine           [to be upgraded]  VALID
  Oracle XDK for Java                    [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  OLAP Analytic Workspace                [to be upgraded]  VALID
  Oracle Label Security                  [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID
  Oracle Java Packages                   [to be upgraded]  VALID
  Oracle Multimedia                      [to be upgraded]  VALID
  Oracle OLAP API                        [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  1.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 12.1.0.2
      Oracle Database SQL Tuning Guide.

  2.  (AUTOFIXUP) Gather statistics on fixed objects prior to the upgrade using
      the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      None of the fixed object tables have had stats collected.

      Gathering statistics on fixed objects, if none have been gathered yet, is
      recommended prior to upgrading.

      For information on managing optimizer statistics, refer to the 12.1.0.2
      Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
  3.  Here are ALL the components in this database registry:

      Component Current     Current     Original    Previous    Component
      CID       Version     Status      Version     Version     Schema
      --------- ----------- ----------- ----------- ----------- -----------
      APS       12.1.0.2.0  VALID                               SYS
      CATALOG   12.1.0.2.0  VALID                               SYS
      CATJAVA   12.1.0.2.0  VALID                               SYS
      CATPROC   12.1.0.2.0  VALID                               SYS
      CONTEXT   12.1.0.2.0  VALID                               CTXSYS
      JAVAVM    12.1.0.2.0  VALID                               SYS
      OLS       12.1.0.2.0  VALID                               LBACSYS
      ORDIM     12.1.0.2.0  VALID       12.1.0.2.0  12.1.0.2.0  ORDSYS
      OWM       12.1.0.2.0  VALID                               WMSYS
      RAC       12.1.0.2.0  OPTION OFF                          SYS
      XDB       12.1.0.2.0  VALID                               XDB
      XML       12.1.0.2.0  VALID                               SYS
      XOQ       12.1.0.2.0  VALID       12.1.0.2.0  12.1.0.2.0  SYS

      Review the information before upgrading.

  4.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      SYSAUX                             285 MB       500 MB
      SYSTEM                             250 MB       568 MB

      Minimum tablespace sizes for upgrade are estimates.

  5.  Here is a count of invalid objects by Oracle-maintained users:

      Oracle-Maintained User Name                 Number of INVALID Objects
      ---------------------------                 -------------------------
      None                                        None

      Review the information before upgrading.

  6.  Here is a count of invalid objects by Application users:

      Application User Name                       Number of INVALID Objects
      ---------------------------                 -------------------------
      None                                        None

      Review the information before upgrading.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PROD container PDB$SEED
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/preupgrade_fixups.sq
    l

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  7.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 18 and the target 19 release
      ships with time zone file version 32.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 19 Oracle Database
      Globalization Support Guide.

  8.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  9.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 12.1.0.2
      Oracle Database SQL Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PROD container PDB$SEED
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postupgrade_fixups.s
    ql


Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 12 on 2022-04-07T15:23:35

Upgrade-To version: 19.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  PROD
     Container Name:  PROD1PDB
       Container ID:  3
            Version:  12.1.0.2.0
     DB Patch Level:  DATABASE PATCH SET UPDATE 12.1.0.2.220118
         Compatible:  12.1.0.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  18
  Database log mode:  NOARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  JServer JAVA Virtual Machine           [to be upgraded]  VALID
  Oracle XDK for Java                    [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  OLAP Analytic Workspace                [to be upgraded]  VALID
  Oracle Label Security                  [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID
  Oracle Java Packages                   [to be upgraded]  VALID
  Oracle Multimedia                      [to be upgraded]  VALID
  Oracle OLAP API                        [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  1.  (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

      The database contains 399 objects in the recycle bin.

      The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
  2.  If an auto-login Oracle Transparent Data Encryption (TDE) Keystore is
      correctly set up, no action needs to be taken. Otherwise, before starting
      up the database in upgrade mode in the new Oracle Database Oracle Home,
      either open the TDE Keystore, or ensure that an auto-login TDE Keystore
      is configured for the system.  If errors are seen while the database
      upgrade is running because the TDE Keystore is closed, then open the TDE
      Keystore and resume the upgrade (see "catctl.pl -R").  It may be
      necessary to open the TDE Keystore as the upgrade progresses in a non-CDB
      or CDB (e.g., in CDB$ROOT and PDB) if no auto-login TDE Keystore has been
      configured, as the upgrade process can shutdown and startup the
      database.

      The database is using TDE.  The database upgrade process can involve
      operations (certain SQL statements and/or database startups) that need to
      access the encryption key.

      If Oracle Transparent Data Encryption (TDE) is in use, the database
      system must have access to its Oracle Transparent Data Encryption Master
      Encryption Key during database upgrade.  For more information on
      configuring Transparent Data Encryption, refer to the 12.1 Oracle
      Database Advanced Security Guide, Section 3: Configuring Transparent Data
      Encryption.

  3.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 12.1.0.2
      Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
  4.  Here are ALL the components in this database registry:

      Component Current     Current     Original    Previous    Component
      CID       Version     Status      Version     Version     Schema
      --------- ----------- ----------- ----------- ----------- -----------
      APS       12.1.0.2.0  VALID                               SYS
      CATALOG   12.1.0.2.0  VALID                               SYS
      CATJAVA   12.1.0.2.0  VALID                               SYS
      CATPROC   12.1.0.2.0  VALID                               SYS
      CONTEXT   12.1.0.2.0  VALID                               CTXSYS
      JAVAVM    12.1.0.2.0  VALID                               SYS
      OLS       12.1.0.2.0  VALID                               LBACSYS
      ORDIM     12.1.0.2.0  VALID       12.1.0.2.0  12.1.0.2.0  ORDSYS
      OWM       12.1.0.2.0  VALID                               WMSYS
      RAC       12.1.0.2.0  OPTION OFF                          SYS
      XDB       12.1.0.2.0  VALID                               XDB
      XML       12.1.0.2.0  VALID                               SYS
      XOQ       12.1.0.2.0  VALID       12.1.0.2.0  12.1.0.2.0  SYS

      Review the information before upgrading.

  5.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      SYSAUX                             565 MB       583 MB
      SYSTEM                             270 MB       592 MB

      Minimum tablespace sizes for upgrade are estimates.

  6.  Here is a count of invalid objects by Oracle-maintained users:

      Oracle-Maintained User Name                 Number of INVALID Objects
      ---------------------------                 -------------------------
      None                                        None

      Review the information before upgrading.

  7.  Here is a count of invalid objects by Application users:

      Application User Name                       Number of INVALID Objects
      ---------------------------                 -------------------------
      None                                        None

      Review the information before upgrading.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PROD container PROD1PDB
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/preupgrade_fixups.sq
    l

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  8.  (AUTOFIXUP) If you use the -T option for the database upgrade, then run
      $ORACLE_HOME/rdbms/admin/utluptabdata.sql after the upgrade is complete,
      to VALIDATE and UPGRADE any user tables affected by changes to
      Oracle-Maintained types.

      There are user tables dependent on Oracle-Maintained object types.

      If the -T option is used to set user tablespaces to READ ONLY during the
      upgrade, user tables in those tablespaces, that are dependent on
      Oracle-Maintained types, will not be automatically upgraded. If a type is
      evolved during the upgrade, any dependent tables need to be re-validated
      and upgraded to the latest type version AFTER the database upgrade
      completes.

  9.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 18 and the target 19 release
      ships with time zone file version 32.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 19 Oracle Database
      Globalization Support Guide.

  10. Recreate directory objects to remove any symbolic links from directory
      paths.  To identify paths that contain symbolic links before upgrading,
      use OS commands like UNIX file or WINDOWS dir.  After upgrading, run
      $ORACLE_HOME/rdbms/admin/utldirsymlink.sql to identify directory objects
      with symbolic links in the path.

      Found 4 user directory objects to be checked: PDB_DATA_PUMP_DIR1,
      PDB_DATA_PUMP_DIR2, PDB_DATA_PUMP_LOG, UTL_FILE_DIR.

      Starting in Release 18c, symbolic links are not allowed in directory
      object paths used with BFILE data types, the UTL_FILE package, or
      external tables.

  11. (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  12. Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 12.1.0.2
      Oracle Database SQL Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database PROD container PROD1PDB
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following from within the container

    SQL>@/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postupgrade_fixups.s
    ql


==================
PREUPGRADE SUMMARY
==================
  /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/preupgrade.log
  /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/preupgrade_fixups.sql
  /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postupgrade_fixups.sql

Execute fixup scripts across the entire CDB:

Before upgrade:

1. Execute preupgrade fixups with the below command
$ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/rdbms/admin $ORACLE_HOME/rdbms/admin/catcon.pl -l /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/ -b preup_PROD /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/preupgrade_fixups.sql

2. Review logs under /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/

After the upgrade:

1. Execute postupgrade fixups with the below command
$ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/rdbms/admin $ORACLE_HOME/rdbms/admin/catcon.pl -l /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/ -b postup_PROD /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postupgrade_fixups.sql

2. Review logs under /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/

Preupgrade complete: 2022-04-07T15:23:40

6. Upgrading the database to 19c


6.1 Execute preupgrade fixups as user oracle.


Note: This will update all the containers.


$ $ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/rdbms/admin $ORACLE_HOME/rdbms/admin/catcon.pl -l /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/ -b preup_PROD /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/preupgrade_fixups.sql
catcon: ALL catcon-related output will be written to /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade//preup_PROD_catcon_29488.lst
catcon: See /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade//preup_PROD*.log files for output generated by scripts
catcon: See /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade//preup_PROD_*.lst files for spool files, if any
catcon.pl: completed successfully

Review logs under /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/


6.2 Prepping and Upgrading the database


6.2.1 Stop the database


# su - oracle
$ srvctl stop database -d PROD

6.2.2 Copy the config files from the old to the new Oracle DB Home.


$ cp -p /oracle/oraBase/oracle/12.1.0.2/database/network/admin/*.ora /oracle/oraBase/oracle/19.0.0/database/network/admin/

$ ls -l /oracle/oraBase/oracle/19.0.0/database/network/admin/*.ora
-rw-r--r-- 1 oracle oinstall 348 May  3  2018 /oracle/oraBase/oracle/19.0.0/database/network/admin/sqlnet.ora
-rwxr-xr-x 1 oracle oinstall 349 Dec  8  2016 /oracle/oraBase/oracle/19.0.0/database/network/admin/tnsnames.ora

6.2.3 Copy orapw* and init* and spfile* from the old to the new home.


$ cp -p /oracle/oraBase/oracle/12.1.0.2/database/dbs/init* /oracle/oraBase/oracle/19.0.0/database/dbs/
$ cp -p /oracle/oraBase/oracle/12.1.0.2/database/dbs/orapw* /oracle/oraBase/oracle/19.0.0/database/dbs/
$ ls -l /oracle/oraBase/oracle/19.0.0/database/dbs/
total 16
-rw-r--r-- 1 oracle oinstall 2992 Feb  2  2012 init.ora
-rw-r--r-- 1 oracle oinstall   35 Apr 11  2018 initPROD.ora
-rw-r----- 1 oracle oinstall 7680 Apr 11  2018 orapwPROD

6.2.4 Update your environment settings for ORACLE_HOME variable.


$ grep ORACLE_HOME .bash_profile
ORACLE_HOME=/oracle/oraBase/oracle/19.0.0/database; export ORACLE_HOME

$ echo $ORACLE_HOME
/oracle/oraBase/oracle/19.0.0/database

6.2.5 Update oracle to use the new software to start up the database.


$ srvctl upgrade database -d PROD -o /oracle/oraBase/oracle/19.0.0/database

6.2.6 Start the database in upgrade mode.


[oracle@orasrv1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Apr 7 16:08:25 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup upgrade;
ORACLE instance started.

Total System Global Area 3221225152 bytes
Fixed Size                  9141952 bytes
Variable Size            1006632960 bytes
Database Buffers         2181038080 bytes
Redo Buffers               24412160 bytes
Database mounted.
Database opened.

6.2.6.1 Put all the pluggable database into upgrade mode.


SQL> alter pluggable database all open upgrade force;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MIGRATE    YES
         3 PROD1PDB                       MIGRATE    YES

6.2.7 As user oracle, upgrade the database.


$ /oracle/oraBase/oracle/19.0.0/database/bin/dbupgrade

Argument list for [/oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catctl.pl]
For Oracle internal use only A = 0
Run in                       c = 0
Do not run in                C = 0
Input Directory              d = 0
Echo OFF                     e = 1
Simulate                     E = 0
Forced cleanup               F = 0
Log Id                       i = 0
Child Process                I = 0
Log Dir                      l = 0
Priority List Name           L = 0
Upgrade Mode active          M = 0
SQL Process Count            n = 0
SQL PDB Process Count        N = 0
Open Mode Normal             o = 0
Start Phase                  p = 0
End Phase                    P = 0
Reverse Order                r = 0
AutoUpgrade Resume           R = 0
Script                       s = 0
Serial Run                   S = 0
RO User Tablespaces          T = 0
Display Phases               y = 0
Debug catcon.pm              z = 0
Debug catctl.pl              Z = 0

catctl.pl VERSION: [19.0.0.0.0]
           STATUS: [Production]
            BUILD: [RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417]


/oracle/oraBase/oracle/19.0.0/database/rdbms/admin/orahome = [/oracle/oraBase/oracle/19.0.0/database]
/oracle/oraBase/oracle/19.0.0/database/bin/orabasehome = [/oracle/oraBase/oracle/19.0.0/database]
catctlGetOraBaseLogDir = [/oracle/oraBase/oracle/19.0.0/database]

Analyzing file /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catupgrd.sql

Log file directory = [/tmp/cfgtoollogs/upgrade20220407161307]

catcon::set_log_file_base_path: ALL catcon-related output will be written to [/tmp/cfgtoollogs/upgrade20220407161307/catupgrd_catcon_11528.lst]

catcon::set_log_file_base_path: catcon: See [/tmp/cfgtoollogs/upgrade20220407161307/catupgrd*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/tmp/cfgtoollogs/upgrade20220407161307/catupgrd_*.lst] files for spool files, if any


Number of Cpus        = 4
Database Name         = PROD
DataBase Version      = 12.1.0.2.0
catcon::set_log_file_base_path: ALL catcon-related output will be written to [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdcdbroot_catcon_11528.lst]

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdcdbroot*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdcdbroot_*.lst] files for spool files, if any


Log file directory = [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322]

Parallel SQL Process Count (PDB)      = 2
Parallel SQL Process Count (CDB$ROOT) = 4
Concurrent PDB Upgrades               = 2
Generated PDB Inclusion:[PDB$SEED PROD1PDB]
Components in [CDB$ROOT]
    Installed [APS CATALOG CATJAVA CATPROC CONTEXT JAVAVM OLS ORDIM OWM XDB XML XOQ]
Not Installed [APEX DV EM MGW ODM RAC SDO WK]

------------------------------------------------------
Phases [0-107]         Start Time:[2022_04_07 16:13:49]
Container Lists Inclusion:[CDB$ROOT] Exclusion:[NONE]
------------------------------------------------------
***********   Executing Change Scripts   ***********
Serial   Phase #:0    [CDB$ROOT] Files:1    Time: 49s
***************   Catalog Core SQL   ***************
Serial   Phase #:1    [CDB$ROOT] Files:5    Time: 85s
Restart  Phase #:2    [CDB$ROOT] Files:1    Time: 3s
***********   Catalog Tables and Views   ***********
Parallel Phase #:3    [CDB$ROOT] Files:19   Time: 33s
Restart  Phase #:4    [CDB$ROOT] Files:1    Time: 4s
*************   Catalog Final Scripts   ************
Serial   Phase #:5    [CDB$ROOT] Files:7    Time: 34s
*****************   Catproc Start   ****************
Serial   Phase #:6    [CDB$ROOT] Files:1    Time: 23s
*****************   Catproc Types   ****************
Serial   Phase #:7    [CDB$ROOT] Files:2    Time: 19s
Restart  Phase #:8    [CDB$ROOT] Files:1    Time: 2s
****************   Catproc Tables   ****************
Parallel Phase #:9    [CDB$ROOT] Files:67   Time: 34s
Restart  Phase #:10   [CDB$ROOT] Files:1    Time: 4s
*************   Catproc Package Specs   ************
Serial   Phase #:11   [CDB$ROOT] Files:1    Time: 92s
Restart  Phase #:12   [CDB$ROOT] Files:1    Time: 3s
**************   Catproc Procedures   **************
Parallel Phase #:13   [CDB$ROOT] Files:94   Time: 15s
Restart  Phase #:14   [CDB$ROOT] Files:1    Time: 4s
Parallel Phase #:15   [CDB$ROOT] Files:120  Time: 18s
Restart  Phase #:16   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:17   [CDB$ROOT] Files:22   Time: 8s
Restart  Phase #:18   [CDB$ROOT] Files:1    Time: 4s
*****************   Catproc Views   ****************
Parallel Phase #:19   [CDB$ROOT] Files:32   Time: 23s
Restart  Phase #:20   [CDB$ROOT] Files:1    Time: 4s
Serial   Phase #:21   [CDB$ROOT] Files:3    Time: 18s
Restart  Phase #:22   [CDB$ROOT] Files:1    Time: 4s
Parallel Phase #:23   [CDB$ROOT] Files:25   Time: 119s
Restart  Phase #:24   [CDB$ROOT] Files:1    Time: 4s
Parallel Phase #:25   [CDB$ROOT] Files:12   Time: 81s
Restart  Phase #:26   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:27   [CDB$ROOT] Files:1    Time: 0s
Serial   Phase #:28   [CDB$ROOT] Files:3    Time: 10s
Serial   Phase #:29   [CDB$ROOT] Files:1    Time: 0s
Restart  Phase #:30   [CDB$ROOT] Files:1    Time: 1s
***************   Catproc CDB Views   **************
Serial   Phase #:31   [CDB$ROOT] Files:1    Time: 6s
Restart  Phase #:32   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:34   [CDB$ROOT] Files:1    Time: 0s
*****************   Catproc PLBs   *****************
Serial   Phase #:35   [CDB$ROOT] Files:293  Time: 32s
Serial   Phase #:36   [CDB$ROOT] Files:1    Time: 0s
Restart  Phase #:37   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:38   [CDB$ROOT] Files:6    Time: 10s
Restart  Phase #:39   [CDB$ROOT] Files:1    Time: 4s
***************   Catproc DataPump   ***************
Serial   Phase #:40   [CDB$ROOT] Files:3    Time: 48s
Restart  Phase #:41   [CDB$ROOT] Files:1    Time: 4s
******************   Catproc SQL   *****************
Parallel Phase #:42   [CDB$ROOT] Files:13   Time: 81s
Restart  Phase #:43   [CDB$ROOT] Files:1    Time: 2s
Parallel Phase #:44   [CDB$ROOT] Files:11   Time: 13s
Restart  Phase #:45   [CDB$ROOT] Files:1    Time: 3s
Parallel Phase #:46   [CDB$ROOT] Files:3    Time: 9s
Restart  Phase #:47   [CDB$ROOT] Files:1    Time: 3s
*************   Final Catproc scripts   ************
Serial   Phase #:48   [CDB$ROOT] Files:1    Time: 14s
Restart  Phase #:49   [CDB$ROOT] Files:1    Time: 3s
**************   Final RDBMS scripts   *************
Serial   Phase #:50   [CDB$ROOT] Files:1    Time: 25s
************   Upgrade Component Start   ***********
Serial   Phase #:51   [CDB$ROOT] Files:1    Time: 6s
Restart  Phase #:52   [CDB$ROOT] Files:1    Time: 3s
**********   Upgrading Java and non-Java   *********
Serial   Phase #:53   [CDB$ROOT] Files:2    Time: 343s
*****************   Upgrading XDB   ****************
Restart  Phase #:54   [CDB$ROOT] Files:1    Time: 2s
Serial   Phase #:56   [CDB$ROOT] Files:3    Time: 16s
Serial   Phase #:57   [CDB$ROOT] Files:3    Time: 9s
Parallel Phase #:58   [CDB$ROOT] Files:10   Time: 12s
Parallel Phase #:59   [CDB$ROOT] Files:25   Time: 13s
Serial   Phase #:60   [CDB$ROOT] Files:4    Time: 15s
Serial   Phase #:61   [CDB$ROOT] Files:1    Time: 0s
Serial   Phase #:62   [CDB$ROOT] Files:32   Time: 11s
Serial   Phase #:63   [CDB$ROOT] Files:1    Time: 0s
Parallel Phase #:64   [CDB$ROOT] Files:6    Time: 13s
Serial   Phase #:65   [CDB$ROOT] Files:2    Time: 21s
Serial   Phase #:66   [CDB$ROOT] Files:3    Time: 39s
****************   Upgrading ORDIM   ***************
Restart  Phase #:67   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:69   [CDB$ROOT] Files:1    Time: 9s
Parallel Phase #:70   [CDB$ROOT] Files:2    Time: 27s
Restart  Phase #:71   [CDB$ROOT] Files:1    Time: 3s
Parallel Phase #:72   [CDB$ROOT] Files:2    Time: 100s
Serial   Phase #:73   [CDB$ROOT] Files:2    Time: 8s
*****************   Upgrading SDO   ****************
Restart  Phase #:74   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:76   [CDB$ROOT] Files:1    Time: 8s
Serial   Phase #:77   [CDB$ROOT] Files:2    Time: 7s
Restart  Phase #:78   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:79   [CDB$ROOT] Files:1    Time: 7s
Restart  Phase #:80   [CDB$ROOT] Files:1    Time: 3s
Parallel Phase #:81   [CDB$ROOT] Files:3    Time: 7s
Restart  Phase #:82   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:83   [CDB$ROOT] Files:1    Time: 7s
Restart  Phase #:84   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:85   [CDB$ROOT] Files:1    Time: 7s
Restart  Phase #:86   [CDB$ROOT] Files:1    Time: 2s
Parallel Phase #:87   [CDB$ROOT] Files:4    Time: 7s
Restart  Phase #:88   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:89   [CDB$ROOT] Files:1    Time: 7s
Restart  Phase #:90   [CDB$ROOT] Files:1    Time: 4s
Serial   Phase #:91   [CDB$ROOT] Files:2    Time: 7s
Restart  Phase #:92   [CDB$ROOT] Files:1    Time: 3s
Serial   Phase #:93   [CDB$ROOT] Files:1    Time: 6s
Restart  Phase #:94   [CDB$ROOT] Files:1    Time: 4s
*******   Upgrading ODM, WK, EXF, RUL, XOQ   *******
Serial   Phase #:95   [CDB$ROOT] Files:1    Time: 17s
Restart  Phase #:96   [CDB$ROOT] Files:1    Time: 2s
***********   Final Component scripts    ***********
Serial   Phase #:97   [CDB$ROOT] Files:1    Time: 8s
*************   Final Upgrade scripts   ************
Serial   Phase #:98   [CDB$ROOT] Files:1    Time: 155s
*******************   Migration   ******************
Serial   Phase #:99   [CDB$ROOT] Files:1    Time: 31s
***   End PDB Application Upgrade Pre-Shutdown   ***
Serial   Phase #:100  [CDB$ROOT] Files:1    Time: 6s
Serial   Phase #:101  [CDB$ROOT] Files:1    Time: 6s
Serial   Phase #:102  [CDB$ROOT] Files:1    Time: 67s
*****************   Post Upgrade   *****************
Serial   Phase #:103  [CDB$ROOT] Files:1    Time: 32s
****************   Summary report   ****************
Serial   Phase #:104  [CDB$ROOT] Files:1    Time: 7s
***   End PDB Application Upgrade Post-Shutdown   **
Serial   Phase #:105  [CDB$ROOT] Files:1    Time: 7s
Serial   Phase #:106  [CDB$ROOT] Files:1    Time: 5s
Serial   Phase #:107  [CDB$ROOT] Files:1     Time: 58s

------------------------------------------------------
Phases [0-107]         End Time:[2022_04_07 16:49:12]
Container Lists Inclusion:[CDB$ROOT] Exclusion:[NONE]
------------------------------------------------------

Start processing of PDBs (PDB$SEED)
[/oracle/oraBase/oracle/19.0.0/database/perl/bin/perl /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catctl.pl -I -i pdb_seed -n 2 -c 'PDB$SEED' -l /oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322 /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catupgrd.sql]

Start processing of PDBs (PROD1PDB)
[/oracle/oraBase/oracle/19.0.0/database/perl/bin/perl /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catctl.pl -I -i prod1pdb -n 2 -c 'PROD1PDB' -l /oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322 /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catupgrd.sql]

Argument list for [/oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catctl.pl]
For Oracle internal use only A = 0
Run in                       c = PDB$SEED
Do not run in                C = 0
Input Directory              d = 0
Echo OFF                     e = 1
Simulate                     E = 0
Forced cleanup               F = 0
Log Id                       i = pdb_seed
Child Process                I = 1
Log Dir                      l = /oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322
Priority List Name           L = 0
Upgrade Mode active          M = 0
SQL Process Count            n = 2
SQL PDB Process Count        N = 0
Open Mode Normal             o = 0
Start Phase                  p = 0
End Phase                    P = 0
Reverse Order                r = 0
AutoUpgrade Resume           R = 0
Script                       s = 0
Serial Run                   S = 0
RO User Tablespaces          T = 0
Display Phases               y = 0
Debug catcon.pm              z = 0
Debug catctl.pl              Z = 0

catctl.pl VERSION: [19.0.0.0.0]
           STATUS: [Production]
            BUILD: [RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417]


/oracle/oraBase/oracle/19.0.0/database/rdbms/admin/orahome = [/oracle/oraBase/oracle/19.0.0/database]

Argument list for [/oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catctl.pl]
For Oracle internal use only A = 0
Run in                       c = PROD1PDB
Do not run in                C = 0
Input Directory              d = 0
Echo OFF                     e = 1
Simulate                     E = 0
Forced cleanup               F = 0
Log Id                       i = prod1pdb
Child Process                I = 1
Log Dir                      l = /oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322
Priority List Name           L = 0
Upgrade Mode active          M = 0
SQL Process Count            n = 2
SQL PDB Process Count        N = 0
Open Mode Normal             o = 0
Start Phase                  p = 0
End Phase                    P = 0
Reverse Order                r = 0
AutoUpgrade Resume           R = 0
Script                       s = 0
Serial Run                   S = 0
RO User Tablespaces          T = 0
Display Phases               y = 0
Debug catcon.pm              z = 0
Debug catctl.pl              Z = 0

catctl.pl VERSION: [19.0.0.0.0]
           STATUS: [Production]
            BUILD: [RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417]


/oracle/oraBase/oracle/19.0.0/database/bin/orabasehome = [/oracle/oraBase/oracle/19.0.0/database]
catctlGetOraBaseLogDir = [/oracle/oraBase/oracle/19.0.0/database]

Analyzing file /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catupgrd.sql

Log file directory = [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322]

catcon::set_log_file_base_path: ALL catcon-related output will be written to [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdpdb_seed_catcon_4374.lst]

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdpdb_seed*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdpdb_seed_*.lst] files for spool files, if any

/oracle/oraBase/oracle/19.0.0/database/rdbms/admin/orahome = [/oracle/oraBase/oracle/19.0.0/database]
/oracle/oraBase/oracle/19.0.0/database/bin/orabasehome = [/oracle/oraBase/oracle/19.0.0/database]
catctlGetOraBaseLogDir = [/oracle/oraBase/oracle/19.0.0/database]

Analyzing file /oracle/oraBase/oracle/19.0.0/database/rdbms/admin/catupgrd.sql

Log file directory = [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322]

catcon::set_log_file_base_path: ALL catcon-related output will be written to [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdprod1pdb_catcon_4376.lst]

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdprod1pdb*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdprod1pdb_*.lst] files for spool files, if any


Number of Cpus        = 4

Number of Cpus        = 4
Database Name         = PROD
Database Name         = PROD
DataBase Version      = 19.0.0.0.0
DataBase Version      = 19.0.0.0.0
PDB$SEED Open Mode = [MIGRATE]
Generated PDB Inclusion:[PDB$SEED]
CDB$ROOT  Open Mode = [OPEN]
Components in [PDB$SEED]
    Installed [APS CATALOG CATJAVA CATPROC CONTEXT JAVAVM OLS ORDIM OWM XDB XML XOQ]
Not Installed [APEX DV EM MGW ODM RAC SDO WK]
PROD1PDB Open Mode = [MIGRATE]
Generated PDB Inclusion:[PROD1PDB]
CDB$ROOT  Open Mode = [OPEN]

------------------------------------------------------
Phases [0-107]         Start Time:[2022_04_07 16:49:54]
Container Lists Inclusion:[PDB$SEED] Exclusion:[NONE]
------------------------------------------------------
Components in [PROD1PDB]
    Installed [APS CATALOG CATJAVA CATPROC CONTEXT JAVAVM OLS ORDIM OWM XDB XML XOQ]
Not Installed [APEX DV EM MGW ODM RAC SDO WK]
***********   Executing Change Scripts   ***********
Serial   Phase #:0    [PDB$SEED] Files:1
------------------------------------------------------
Phases [0-107]         Start Time:[2022_04_07 16:50:04]
Container Lists Inclusion:[PROD1PDB] Exclusion:[NONE]
------------------------------------------------------
***********   Executing Change Scripts   ***********
Serial   Phase #:0    [PROD1PDB] Files:1    Time: 79s
***************   Catalog Core SQL   ***************
Serial   Phase #:1    [PDB$SEED] Files:5    Time: 76s
***************   Catalog Core SQL   ***************
Serial   Phase #:1    [PROD1PDB] Files:5    Time: 106s
Restart  Phase #:2    [PROD1PDB] Files:1    Time: 113s
Restart  Phase #:2    [PDB$SEED] Files:1    Time: 1s
***********   Catalog Tables and Views   ***********
Parallel Phase #:3    [PDB$SEED] Files:19    Time: 1s
***********   Catalog Tables and Views   ***********
Parallel Phase #:3    [PROD1PDB] Files:19   Time: 52s
Restart  Phase #:4    [PDB$SEED] Files:1   Time: 52s
Restart  Phase #:4    [PROD1PDB] Files:1    Time: 2s
*************   Catalog Final Scripts   ************
Serial   Phase #:5    [PDB$SEED] Files:7    Time: 2s
*************   Catalog Final Scripts   ************
Serial   Phase #:5    [PROD1PDB] Files:7    Time: 46s
*****************   Catproc Start   ****************
Serial   Phase #:6    [PDB$SEED] Files:1    Time: 46s
*****************   Catproc Start   ****************
Serial   Phase #:6    [PROD1PDB] Files:1    Time: 25s
*****************   Catproc Types   ****************
Serial   Phase #:7    [PDB$SEED] Files:2    Time: 25s
*****************   Catproc Types   ****************
Serial   Phase #:7    [PROD1PDB] Files:2    Time: 20s
Restart  Phase #:8    [PDB$SEED] Files:1    Time: 20s
Restart  Phase #:8    [PROD1PDB] Files:1    Time: 2s
****************   Catproc Tables   ****************
Parallel Phase #:9    [PROD1PDB] Files:67    Time: 2s
****************   Catproc Tables   ****************
Parallel Phase #:9    [PDB$SEED] Files:67   Time: 50s
Restart  Phase #:10   [PDB$SEED] Files:1   Time: 50s
Restart  Phase #:10   [PROD1PDB] Files:1    Time: 1s
*************   Catproc Package Specs   ************
Serial   Phase #:11   [PDB$SEED] Files:1    Time: 1s
*************   Catproc Package Specs   ************
Serial   Phase #:11   [PROD1PDB] Files:1    Time: 108s
Restart  Phase #:12   [PDB$SEED] Files:1    Time: 108s
Restart  Phase #:12   [PROD1PDB] Files:1    Time: 0s
**************   Catproc Procedures   **************
Parallel Phase #:13   [PDB$SEED] Files:94    Time: 1s
**************   Catproc Procedures   **************
Parallel Phase #:13   [PROD1PDB] Files:94   Time: 14s
Restart  Phase #:14   [PDB$SEED] Files:1   Time: 13s
Restart  Phase #:14   [PROD1PDB] Files:1    Time: 1s
Parallel Phase #:15   [PDB$SEED] Files:120    Time: 1s
Parallel Phase #:15   [PROD1PDB] Files:120  Time: 17s
Restart  Phase #:16   [PDB$SEED] Files:1  Time: 17s
Restart  Phase #:16   [PROD1PDB] Files:1    Time: 0s
Serial   Phase #:17   [PROD1PDB] Files:22    Time: 1s
Serial   Phase #:17   [PDB$SEED] Files:22   Time: 9s
Restart  Phase #:18   [PROD1PDB] Files:1   Time: 8s
Restart  Phase #:18   [PDB$SEED] Files:1    Time: 1s
*****************   Catproc Views   ****************
Parallel Phase #:19   [PROD1PDB] Files:32    Time: 2s
*****************   Catproc Views   ****************
Parallel Phase #:19   [PDB$SEED] Files:32   Time: 34s
Restart  Phase #:20   [PROD1PDB] Files:1   Time: 33s
Restart  Phase #:20   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:21   [PROD1PDB] Files:3    Time: 1s
Serial   Phase #:21   [PDB$SEED] Files:3    Time: 22s
Restart  Phase #:22   [PDB$SEED] Files:1    Time: 22s
Restart  Phase #:22   [PROD1PDB] Files:1    Time: 0s
Parallel Phase #:23   [PDB$SEED] Files:25    Time: 1s
Parallel Phase #:23   [PROD1PDB] Files:25   Time: 292s
Restart  Phase #:24   [PROD1PDB] Files:1   Time: 293s
Restart  Phase #:24   [PDB$SEED] Files:1    Time: 1s
Parallel Phase #:25   [PROD1PDB] Files:12    Time: 2s
Parallel Phase #:25   [PDB$SEED] Files:12   Time: 91s
Restart  Phase #:26   [PDB$SEED] Files:1   Time: 92s
Restart  Phase #:26   [PROD1PDB] Files:1    Time: 2s
Serial   Phase #:27   [PDB$SEED] Files:1    Time: 0s
Serial   Phase #:28   [PDB$SEED] Files:3    Time: 2s
Serial   Phase #:27   [PROD1PDB] Files:1    Time: 0s
Serial   Phase #:28   [PROD1PDB] Files:3    Time: 19s
Serial   Phase #:29   [PROD1PDB] Files:1    Time: 0s
Restart  Phase #:30   [PROD1PDB] Files:1    Time: 19s
Serial   Phase #:29   [PDB$SEED] Files:1    Time: 0s
Restart  Phase #:30   [PDB$SEED] Files:1    Time: 1s
***************   Catproc CDB Views   **************
Serial   Phase #:31   [PDB$SEED] Files:1    Time: 1s
***************   Catproc CDB Views   **************
Serial   Phase #:31   [PROD1PDB] Files:1    Time: 6s
Restart  Phase #:32   [PDB$SEED] Files:1    Time: 6s
Restart  Phase #:32   [PROD1PDB] Files:1    Time: 2s
Serial   Phase #:34   [PDB$SEED] Files:1    Time: 0s
*****************   Catproc PLBs   *****************
Serial   Phase #:35   [PDB$SEED] Files:293    Time: 2s
Serial   Phase #:34   [PROD1PDB] Files:1    Time: 0s
*****************   Catproc PLBs   *****************
Serial   Phase #:35   [PROD1PDB] Files:293  Time: 111s
Serial   Phase #:36   [PROD1PDB] Files:1    Time: 0s
Restart  Phase #:37   [PROD1PDB] Files:1  Time: 111s
Serial   Phase #:36   [PDB$SEED] Files:1    Time: 0s
Restart  Phase #:37   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:38   [PROD1PDB] Files:6    Time: 2s
Serial   Phase #:38   [PDB$SEED] Files:6    Time: 10s
Restart  Phase #:39   [PROD1PDB] Files:1    Time: 9s
Restart  Phase #:39   [PDB$SEED] Files:1    Time: 1s
***************   Catproc DataPump   ***************
Serial   Phase #:40   [PROD1PDB] Files:3    Time: 2s
***************   Catproc DataPump   ***************
Serial   Phase #:40   [PDB$SEED] Files:3    Time: 40s
Restart  Phase #:41   [PDB$SEED] Files:1    Time: 42s
Restart  Phase #:41   [PROD1PDB] Files:1    Time: 2s
******************   Catproc SQL   *****************
Parallel Phase #:42   [PDB$SEED] Files:13    Time: 2s
******************   Catproc SQL   *****************
Parallel Phase #:42   [PROD1PDB] Files:13   Time: 88s
Restart  Phase #:43   [PROD1PDB] Files:1   Time: 89s
Restart  Phase #:43   [PDB$SEED] Files:1    Time: 2s
Parallel Phase #:44   [PDB$SEED] Files:11    Time: 2s
Parallel Phase #:44   [PROD1PDB] Files:11   Time: 10s
Restart  Phase #:45   [PDB$SEED] Files:1   Time: 11s
Restart  Phase #:45   [PROD1PDB] Files:1    Time: 2s
Parallel Phase #:46   [PDB$SEED] Files:3    Time: 1s
Parallel Phase #:46   [PROD1PDB] Files:3    Time: 7s
Restart  Phase #:47   [PROD1PDB] Files:1    Time: 7s
Restart  Phase #:47   [PDB$SEED] Files:1    Time: 1s
*************   Final Catproc scripts   ************
Serial   Phase #:48   [PROD1PDB] Files:1    Time: 2s
*************   Final Catproc scripts   ************
Serial   Phase #:48   [PDB$SEED] Files:1    Time: 18s
Restart  Phase #:49   [PROD1PDB] Files:1    Time: 17s
Restart  Phase #:49   [PDB$SEED] Files:1    Time: 1s
**************   Final RDBMS scripts   *************
Serial   Phase #:50   [PROD1PDB] Files:1    Time: 2s
**************   Final RDBMS scripts   *************
Serial   Phase #:50   [PDB$SEED] Files:1    Time: 32s
************   Upgrade Component Start   ***********
Serial   Phase #:51   [PROD1PDB] Files:1    Time: 31s
************   Upgrade Component Start   ***********
Serial   Phase #:51   [PDB$SEED] Files:1    Time: 6s
Restart  Phase #:52   [PROD1PDB] Files:1    Time: 6s
Restart  Phase #:52   [PDB$SEED] Files:1    Time: 2s
**********   Upgrading Java and non-Java   *********
Serial   Phase #:53   [PDB$SEED] Files:2    Time: 2s
**********   Upgrading Java and non-Java   *********
Serial   Phase #:53   [PROD1PDB] Files:2    Time: 259s
*****************   Upgrading XDB   ****************
Restart  Phase #:54   [PDB$SEED] Files:1    Time: 0s
Serial   Phase #:56   [PDB$SEED] Files:3    Time: 259s
*****************   Upgrading XDB   ****************
Restart  Phase #:54   [PROD1PDB] Files:1    Time: 1s
Serial   Phase #:56   [PROD1PDB] Files:3    Time: 16s
Serial   Phase #:57   [PDB$SEED] Files:3    Time: 17s
Serial   Phase #:57   [PROD1PDB] Files:3    Time: 10s
Parallel Phase #:58   [PDB$SEED] Files:10    Time: 9s
Parallel Phase #:58   [PROD1PDB] Files:10   Time: 10s
Parallel Phase #:59   [PDB$SEED] Files:25   Time: 10s
Parallel Phase #:59   [PROD1PDB] Files:25   Time: 13s
Serial   Phase #:60   [PDB$SEED] Files:4   Time: 13s
Serial   Phase #:60   [PROD1PDB] Files:4    Time: 22s
   Time: 23s
Serial   Phase #:61   [PROD1PDB] Files:1 Serial   Phase #:61   [PDB$SEED] Files:1    Time: 0s
   Time: 0s
Serial   Phase #:62   [PROD1PDB] Files:32 Serial   Phase #:62   [PDB$SEED] Files:32   Time: 14s
Serial   Phase #:63   [PROD1PDB] Files:1    Time: 0s
Parallel Phase #:64   [PROD1PDB] Files:6   Time: 14s
Serial   Phase #:63   [PDB$SEED] Files:1    Time: 0s
Parallel Phase #:64   [PDB$SEED] Files:6    Time: 14s
Serial   Phase #:65   [PROD1PDB] Files:2    Time: 14s
Serial   Phase #:65   [PDB$SEED] Files:2    Time: 21s
Serial   Phase #:66   [PDB$SEED] Files:3    Time: 21s
Serial   Phase #:66   [PROD1PDB] Files:3    Time: 36s
****************   Upgrading ORDIM   ***************
Restart  Phase #:67   [PDB$SEED] Files:1    Time: 36s
****************   Upgrading ORDIM   ***************
Restart  Phase #:67   [PROD1PDB] Files:1    Time: 1s
Serial   Phase #:69   [PDB$SEED] Files:1    Time: 2s
Serial   Phase #:69   [PROD1PDB] Files:1    Time: 9s
Parallel Phase #:70   [PDB$SEED] Files:2    Time: 9s
Parallel Phase #:70   [PROD1PDB] Files:2    Time: 31s
Restart  Phase #:71   [PROD1PDB] Files:1    Time: 32s
Restart  Phase #:71   [PDB$SEED] Files:1    Time: 1s
Parallel Phase #:72   [PDB$SEED] Files:2    Time: 1s
Parallel Phase #:72   [PROD1PDB] Files:2    Time: 92s
Serial   Phase #:73   [PROD1PDB] Files:2    Time: 92s
Serial   Phase #:73   [PDB$SEED] Files:2    Time: 9s
*****************   Upgrading SDO   ****************
Restart  Phase #:74   [PROD1PDB] Files:1    Time: 9s
*****************   Upgrading SDO   ****************
Restart  Phase #:74   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:76   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:76   [PROD1PDB] Files:1    Time: 8s
Serial   Phase #:77   [PDB$SEED] Files:2    Time: 9s
Serial   Phase #:77   [PROD1PDB] Files:2    Time: 9s
Restart  Phase #:78   [PDB$SEED] Files:1    Time: 8s
Restart  Phase #:78   [PROD1PDB] Files:1    Time: 0s
Serial   Phase #:79   [PROD1PDB] Files:1    Time: 1s
Serial   Phase #:79   [PDB$SEED] Files:1    Time: 9s
Restart  Phase #:80   [PROD1PDB] Files:1    Time: 8s
Restart  Phase #:80   [PDB$SEED] Files:1    Time: 1s
Parallel Phase #:81   [PROD1PDB] Files:3    Time: 2s
Parallel Phase #:81   [PDB$SEED] Files:3    Time: 10s
Restart  Phase #:82   [PROD1PDB] Files:1    Time: 10s
Restart  Phase #:82   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:83   [PROD1PDB] Files:1    Time: 1s
Serial   Phase #:83   [PDB$SEED] Files:1    Time: 9s
Restart  Phase #:84   [PROD1PDB] Files:1    Time: 9s
Restart  Phase #:84   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:85   [PROD1PDB] Files:1    Time: 1s
Serial   Phase #:85   [PDB$SEED] Files:1    Time: 9s
Restart  Phase #:86   [PROD1PDB] Files:1    Time: 9s
Restart  Phase #:86   [PDB$SEED] Files:1    Time: 1s
Parallel Phase #:87   [PROD1PDB] Files:4    Time: 1s
Parallel Phase #:87   [PDB$SEED] Files:4    Time: 10s
Restart  Phase #:88   [PROD1PDB] Files:1    Time: 10s
Restart  Phase #:88   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:89   [PROD1PDB] Files:1    Time: 1s
Serial   Phase #:89   [PDB$SEED] Files:1    Time: 9s
Restart  Phase #:90   [PROD1PDB] Files:1    Time: 9s
Restart  Phase #:90   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:91   [PROD1PDB] Files:2    Time: 1s
Serial   Phase #:91   [PDB$SEED] Files:2    Time: 9s
Restart  Phase #:92   [PROD1PDB] Files:1    Time: 9s
Restart  Phase #:92   [PDB$SEED] Files:1    Time: 1s
Serial   Phase #:93   [PROD1PDB] Files:1    Time: 1s
Serial   Phase #:93   [PDB$SEED] Files:1    Time: 7s
Restart  Phase #:94   [PROD1PDB] Files:1    Time: 7s
Restart  Phase #:94   [PDB$SEED] Files:1    Time: 2s
*******   Upgrading ODM, WK, EXF, RUL, XOQ   *******
Serial   Phase #:95   [PROD1PDB] Files:1    Time: 1s
*******   Upgrading ODM, WK, EXF, RUL, XOQ   *******
Serial   Phase #:95   [PDB$SEED] Files:1    Time: 18s
Restart  Phase #:96   [PDB$SEED] Files:1    Time: 18s
Restart  Phase #:96   [PROD1PDB] Files:1    Time: 1s
***********   Final Component scripts    ***********
Serial   Phase #:97   [PDB$SEED] Files:1    Time: 2s
***********   Final Component scripts    ***********
Serial   Phase #:97   [PROD1PDB] Files:1    Time: 8s
*************   Final Upgrade scripts   ************
Serial   Phase #:98   [PDB$SEED] Files:1    Time: 7s
*************   Final Upgrade scripts   ************
Serial   Phase #:98   [PROD1PDB] Files:1    Time: 147s
*******************   Migration   ******************
Serial   Phase #:99   [PDB$SEED] Files:1    Time: 147s
*******************   Migration   ******************
Serial   Phase #:99   [PROD1PDB] Files:1    Time: 30s
***   End PDB Application Upgrade Pre-Shutdown   ***
Serial   Phase #:100  [PDB$SEED] Files:1    Time: 30s
***   End PDB Application Upgrade Pre-Shutdown   ***
Serial   Phase #:100  [PROD1PDB] Files:1    Time: 6s
Serial   Phase #:101  [PDB$SEED] Files:1    Time: 6s
Serial   Phase #:101  [PROD1PDB] Files:1    Time: 6s
Serial   Phase #:102  [PDB$SEED] Files:1    Time: 7s
*****************   Post Upgrade   *****************
Serial   Phase #:103  [PDB$SEED] Files:1    Time: 17s
Serial   Phase #:102  [PROD1PDB] Files:1    Time: 8s
*****************   Post Upgrade   *****************
Serial   Phase #:103  [PROD1PDB] Files:1    Time: 34s
****************   Summary report   ****************
Serial   Phase #:104  [PROD1PDB] Files:1    Time: 7s
***   End PDB Application Upgrade Post-Shutdown   **
Serial   Phase #:105  [PROD1PDB] Files:1    Time: 6s
Serial   Phase #:106  [PROD1PDB] Files:1    Time: 13s
Serial   Phase #:107  [PROD1PDB] Files:1     Time: 0s

------------------------------------------------------
Phases [0-107]         End Time:[2022_04_07 17:28:19]
Container Lists Inclusion:[PROD1PDB] Exclusion:[NONE]
------------------------------------------------------

Grand Total Time: 2295s [PROD1PDB]

 LOG FILES: (/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdprod1pdb*.log)

Upgrade Summary Report Located in:
/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/upg_summary.log

   Time: 373s
****************   Summary report   ****************
Serial   Phase #:104  [PDB$SEED] Files:1    Time: 6s
***   End PDB Application Upgrade Post-Shutdown   **
Serial   Phase #:105  [PDB$SEED] Files:1    Time: 6s
Serial   Phase #:106  [PDB$SEED] Files:1    Time: 5s
Serial   Phase #:107  [PDB$SEED] Files:1     Time: 0s

------------------------------------------------------
Phases [0-107]         End Time:[2022_04_07 17:33:37]
Container Lists Inclusion:[PDB$SEED] Exclusion:[NONE]
------------------------------------------------------

Grand Total Time: 2630s [PDB$SEED]

 LOG FILES: (/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdpdb_seed*.log)

Upgrade Summary Report Located in:
/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/upg_summary.log

     Time: 2122s For CDB$ROOT
     Time: 2676s For PDB(s)

Grand Total Time: 4798s

 LOG FILES: (/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/catupgrdcdbroot*.log)

Upgrade Summary Report Located in:
/oracle/oraBase/oracle/19.0.0/database/cfgtoollogs/PROD/upgrade20220407161322/upg_summary.log

Grand Total Upgrade Time:    [0d:1h:19m:58s]

6.2.8 Review the log files and the upgrade summary report.


6.2.9 Execute postupgrade fixups with the below command.


Note: This will update all the containers.


6.2.10 Verify your database is open.


[oracle@orasrv1 ~]$ sqlplus / as sysdba
SQL> select open_mode from v$database;

OPEN_MODE
------------------------------------------------------------
READ WRITE

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PROD1PDB                       MOUNTED

SQL> alter pluggable database all open read write;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PROD1PDB                       READ WRITE NO

6.2.11 Run the postupgrade_fixups.sql script. This script will update all the containers.


[root@orasrv1 ~]# su - oracle
Last login: Fri Apr  8 10:03:58 PDT 2022 on pts/0
[oracle@orasrv1 ~]$ $ORACLE_HOME/perl/bin/perl -I$ORACLE_HOME/perl/lib -I$ORACLE_HOME/rdbms/admin $ORACLE_HOME/rdbms/admin/catcon.pl -l /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/ -b postup_PROD /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postupgrade_fixups.sql
catcon::set_log_file_base_path: ALL catcon-related output will be written to [/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postup_PROD_catcon_6382.lst]

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postup_PROD*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/postup_PROD_*.lst] files for spool files, if any

catcon.pl: completed successfully

6.2.12 Review logs under /oracle/oraBase/oracle/cfgtoollogs/PROD/preupgrade/


7. Upgrade the DB Timezone


We are almost done with the upgrade. One thing left to do is to upgrade the DB timezone. Go here for instructions – Upgrading the Oracle Database Timezone File



Next Step


Go here to learn how to install the latest patches on your new 19c GI/DB Homes – How to Install the Latest Oracle 19c Update Patches – Opatch/OJVM/GI/DB/JDK on Oracle 19c on Linux