Thursday, May 26, 2022

java.sql.SQLException: Non supported character set (add orai18n.jar in your classpath) - Oracle Enterprise Manager 13.5 upgrade/install

The repository of EM Cloud Control was historically situated in the database of different from AL32UTF8 characterset. The upgrade procedure of EM is stucking into the error in the title. Had the character set of repository database been AL32UTF8, this error would have never been happen (but I found this is a tricky way because encrypted EM key is kept in the repository and it depends on the id (guid ?) of database as well).

The following steps was made to resolve.

1. Unzip em13500_linux64.bin into temporary catalog

% unzip -q /u02/install/13.5/em13500_linux64.bin -d /u02/install/13.5/unzip/

2. Unzip EM installer, put the file orai18n.jar from existing 19c oracle home and make some modifications :

% cd /u02/install/13.5/unzip/Disk1/stage/Components/oracle.sysman.em.installer/13.5.0.0.0/DataFiles

% mkdir unzip_fg1

% unzip -q filegroup1.jar -d unzip_fg1

% cd unzip_fg1/oui/em

% cp $ORACLE_HOME/jlib/orai18n.jar .

% cp -p ../../../compDef.xml ../../../compDef.xml.orig

% vi ../../../compDef.xml 

Put the line describing orai18n.jar into compDef.xml

...

             <file size="1661488" dest="orai18n.jar"/>
...

% cd ../.. ; pwd

/u02/install/13.5/unzip/Disk1/stage/Components/oracle.sysman.em.installer/13.5.0.0.0/DataFiles/unzip_fg1

% cp -p ../filegroup1.jar ../filegroup1.jar.orig

% zip -r ../filegroup1.jar *

% cd /u02/install/13.5/unzip/Disk1/install

% cp -p oraparam.ini oraparam.ini.orig

% cp -p linux64/oraparam.ini linux64/oraparam.ini.orig

Modify oraparam.ini to include the new jar file in the classpath :

...

EXT_CLASSPATH=oui/em/orai18n.jar,...

...

% cd /u02/install/13.5

% cp -p em13500_linux64.bin em13500_linux64.bin.orig

% cd unzip

% zip -u ../em13500_linux64.bin Disk1/stage/Components/oracle.sysman.em.installer/13.5.0.0.0/compDef.xml

% zip -u ../em13500_linux64.bin Disk1/stage/Components/oracle.sysman.em.installer/13.5.0.0.0/DataFiles/filegroup1.jar

% zip -u ../em13500_linux64.bin Disk1/install/oraparam.ini

% zip -u ../em13500_linux64.bin Disk1/install/linux64/oraparam.ini

3. That's it ! Run the installation or upgrade 😀


Wednesday, May 25, 2022

ORA-29548: Java system class reported: release of Java system classes in the database... - datapatch - rollback OJVM patches on PDB

I created PDB (version 19.10) via database link (from PDB version 19.13).

I opened it - and there were a lot of violations in PDB opening (patch mismatches etc.)

I run datapatch -verbose -pdb ... to rollback all the patches from 19.13 and implement patches from 19.10 home.

The output was :

Validating logfiles...done                                                                     
Patch 33192694 rollback (pdb AAA): WITH ERRORS                
 logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/33192694/24442454/33192694_rollback_CDBR_AAA_2022May25_15_18_42.log (errors)
 -> Error at line 55: script javavm/install/jvmpsupdi.sql         
     - SP2-0310: unable to open file "/u01/app/oracle/product/db_1910/javavm/install/jvmpsupdii.sql"
Patch 29213893 rollback (pdb AAA): SUCCESS                                
 logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/29213893/24384541/29213893_rollback_CDBR_AAA_2022May25_15_18_43.log (no errors)
Patch 33430073 rollback (pdb AAA): WITH ERRORS                
 logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/33430073/24509446/33430073_rollback_CDBR_AAA_2022May25_15_18_46.log (errors)
 -> Error at line 106017: script rdbms/admin/dpload.sql                  
     - ORA-65047: Object SYS.KU$_COLLIST_VIEW is invalid or compiled with errors in root.
     - ORA-06512: at "SYS.RECOMP_CATMETA_VIEWS", line 35          
     - ORA-06512: at line 8                                                                                                                                                                  
Patch 33192793 rollback (pdb AAA): WITH ERRORS               
 logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/33192793/24467360/33192793_rollback_CDBR_AAA_2022May25_15_29_03.log (errors)
 -> Error at line 217991: script md/admin/catsem.sql                      
     - ORA-29548: Java system class reported: release of Java system classes in the
     - database (19.0.0.0.211019 1.8) does not match that of the oracle executable
     - (19.0.0.0.0 1.8)                                                                         
     - ORA-06512: at "SYS.DBMS_JAVA", line 587                                                                                                                                               
     - ORA-06512: at line 1         



According to the logs, the real cause laid in the absence of $ORACLE_HOME/javavm/install/mpsupdii.sql file, which was in this place in oracle home version 19.13, but it also was in rollback catalog related to rollbacking patch :

 gfind -name "*jvmpsupdii*"
./sqlpatch/33192694/24442454/rollback_files/19.1.0.0.0/javavm/install/jvmpsupdii.sql

 I just put this file to directory ORACLE_HOME/javavm/install and rerun datapatch without errors. Good Luck !


Wednesday, May 11, 2022

In-place upgrade of Oracle Linux 6 or 7 to Oracle Linux 8

OL upgrade from version 6 to 8 requires upgrade version 6 to version 7 first, then you can upgrade version 7 to 8. Next steps describe briefly how to upgrade OL 6.10 (UNL-registered) to 8.5. It doesn't replace the documentation. Before you start, consider the restoration procedure, the upgrade might possibly fail.

1. Enable additional repositories.

Install additional package end enable new repository system :

# yum install oracle-softwarecollection-release-el6

# /usr/bin/ol_yum_configure.sh

I enabled all disabled repositores, but it's not required. 

# yum-config-manager --enable \*

If you receive the error :

 Traceback (most recent call last):
 File "/usr/bin/yum-config-manager", line 192, in <module>
   repo.cfg.options, repo.iteritems, repo.optionobj,
AttributeError: 'RhnRepo' object has no attribute 'cfg'

then disable rhn plugin in file /etc/yum/pluginconf.d/rhnplugin.conf.

Check enabled repositories :

# yum repolist

2. Update packages of Linux 6 to the last available versions.

# yum update

You may want to modify proxy settings (/etc/sysconfig/rhn/up2date) or resolve dependency issues. Reboot the system. You might also disable any network mounted filesystems in /etc/fstab. Change mount options for the local file system to defaults, just to be on the safe side.

3. Install additional packages required for upgrade to OL 7 and check possible issues before upgrade :

# yum install openscap redhat-upgrade-tool preupgrade-assistant preupgrade-assistant-el6toel7 preupgrade-assistant-el6toel7-data-0 preupgrade-assistant-tools preupgrade-assistant-ui

# preupg

4. Copy iso of OL7.6 into local disk, mount it and run upgrade to 7.6. redhat-upgrade-tool-cli works only with OL7.6 iso, so you have to upgrade to 7.6 first, then upgrade to 7.9 (the last release of OL7) aftermath. Copy the content of iso into local directory, you can use rsync -av for it. Open session on the server console, do not do the upgrade over the network.

# mount -o loop /install/OracleLinux-R7-U6-Server-x86_64-dvd.iso /mnt/temp/

# mkdir /install/ol7r6 ; cd /mnt/temp ; rsync -av ./* /install/ol7r6

# redhat-upgrade-tool-cli --force --iso=/install/OracleLinux-R7-U6-Server-x86_64-dvd.iso 

--debuglog=/tmp/upgrade.log --cleanup-post

I used --force option on my own risk. I had only two failed prerequisites related to GNOME and KDE desktop environment, so I decided to ignore them forcibly. 

I had to reboot the server at this stage also because upgrade procedure gave the error when it restarted the network service as preliminary action. The upgrade changes network card names from ethX to netX after reboot, but it actually used new names before (mandatory before upgrade) reboot. After running again redhat-upgrade-tool-cli, when script's run successfully, go further :

                                                          [  OK  ]
Finished. Reboot to start upgrade.

5. Reboot to start upgrade to 7.6

At this phase system boots into temporary upgrade shell and upgrade is being performed, and system is rebooted again into new kernel.

6. Upgrade GRUB boot loader to GRUB2.

Please use documentation link to do : upgrade_grub_to_grub2

7. Install last updates (patches) to 7.9 release and reboot :

# yum-config-manager --enable \*

# yum upgrade

# reboot

8. Unregister the system from Unbreakable Linux Network (ULN, if registered)  and reboot.

9. Add leapp_ol7 repository and install leapp utility (or modify oracle-linux-ol7-.repo.rpmnew to enable ol7_leapp repository) :

# reboot

# cat /etc/yum.repos.d/leapp.repo
[ol7_leapp]
name=Leapp Upgrade Utilities for Oracle Linux $releasever ($basearch)
baseurl=https://yum.oracle.com/repo/OracleLinux/OL7/leapp/x86_64/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
 

# yum install -y leapp --enablerepo=ol7_leapp,ol7_latest

All steps about using leapp for upgrade OL7 to OL8 described in detail in the document Performing Systems Upgrade With Leapp.

10. Grant root login over ssh explicitly (PermitRoolLogin=yes in sshd_config) and perform preupgrade :

# leapp preupgrade --oraclelinux

11. Analyze file /var/log/leapp/leapp-report.txt for issues/errors. As for me - I received the following issues :

============================================================
                    UPGRADE INHIBITED  ============================================================                                                                                                                                  

Upgrade has been inhibited due to the following problems:
   1. Inhibitor: Confirm only RDMA UEKR6 is available on OL8.
   2. Inhibitor: Confirm kernel upgrade to UEKR6 for RDMA?
   3. Inhibitor: Missing required answers in the answer file
   4. Inhibitor: Missing required answers in the answer file
   5. Inhibitor: Missing required answers in the answer file
Consult the pre-upgrade report for details and possible remediation.

============================================================
                    UPGRADE INHIBITED
============================================================     

12. To fix inhibitor 1, I'd enabled appropriate repositories :

# yum-config-manager --enable ol8_UEKR6,ol8_UEKR6_RDMA

I'd changed link of baseurl in repository file to publicly available site yum.oracle.com :

[ol8_UEKR6]
name=Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux $releasever ($basearch)
baseurl=https://yum.oracle.com/repo/OracleLinux/OL8/UEKR6/x86_64/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

[ol8_UEKR6_RDMA]
name=Oracle Linux 8 UEK6 RDMA ($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL8/UEKR6/RDMA/x86_64/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

Then I entered answeres into answerfile and repeated preupgrade procedure :

# cat /var/log/leapp/answerfile
[confirm_RDMA_UEKR6_only_on_OL8]
# Title:              None
# Reason:             Confirmation
# =================== confirm_RDMA_UEKR6_only_on_OL8.confirm ==================
# Label:              Confirm RDMA UEKR6 on OL8 only? If no, the upgrade process will be interrupted.
# Description:        Confirm only RDMA UEKR6 is available on OL8.
# Type:               bool
# Default:            None
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
confirm = True

[remove_pam_pkcs11_module_check]
# Title:              None
# Reason:             Confirmation
# =================== remove_pam_pkcs11_module_check.confirm ==================
# Label:              Disable pam_pkcs11 module in PAM configuration? If no, the upgrade process will be interrupted.
# Description:        PAM module pam_pkcs11 is no longer available in OL-8 since it was replaced by SSSD.
# Type:               bool
# Default:            None
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
confirm = True

[RDMA_upgrade_kernel_to_UEKR6]
# Title:              None
# Reason:             Confirmation
# ==================== RDMA_upgrade_kernel_to_UEKR6.confirm ===================
# Label:              Confirm kernel upgrade to UEKR6 for RDMA? If no, the upgrade process will be interrupted.
# Description:        Confirm upgrade kernel to UEKR6 for RDMA.
# Type:               bool
# Default:            None
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
confirm = True

# leapp preupgrade --oraclelinux

At this point you may want to update proxy settings as well. You should do it in several places actually (not in one place, as you possible did when upgrading to OL7) :

- /etc/yum.repos.d/leapp-upgrade-repos-ol8.repo. Here you must add proxy entries for everyone enabled OL8 repository

[ol8_UEKR6]
name=Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux $releasever ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL8/UEKR6/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
proxy=http://127.0.0.1:3128

[ol8_appstream]
name=Oracle Linux 8 Application Stream ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL8/appstream/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
proxy=http://127.0.0.1:3128

[ol8_baseos_latest]
name=Oracle Linux 8 BaseOS Latest ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL8/baseos/latest/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
proxy=http://127.0.0.1:3128

- /etc/dnf/dnf.conf

[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True

sslverify=False

proxy=http://127.0.0.1:3128
 

- just in case I modified /etc/yum.conf as well (proxy=...) and created some shell variables for curl, which is used for getting new versions of packages :

export all_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 http_proxy=http://127.0.0.1:3128

Just to point you out : you may need additional free space in /var directory (/var/lib/leapp, in my case there was only 8GB free, but leapp requested 20GB free). Solution - move leapp directory to filesystem with more free space and make a symbolic link to it.

13. After receiving "green" report, one can upgrade the OS :

Debug output written to /var/log/leapp/leapp-preupgrade.log

============================================================
                          REPORT
============================================================                                                                                                                                  

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                      END OF REPORT
============================================================                                                                                                                                  

Answerfile has been generated at /var/log/leapp/answerfile

# leapp upgrade --oraclelinux

Upgrade report should also be "green". Reboot the system. The upgrade procedure will start immediately after running operating system (new entry is created in the boot loader configuration and is used during reboot), and OS will be rebooted again.

13. Verify the upgraded OS.

# cat /etc/*release

# uname -r

14. Register system in Unbreakable Linux Network (if needed) and install last patches. You may need to recreate grub2 configuration (in order to reorder grub menu entries - remove OL7 and add OL8 lines) by :

# grub2-mkconfig -o /boot/grub2/grub.cfg

15. Relink/reinstall/upgrade all the needed software.

16. Enjoy !

P.S. leapp software can be used for upgrade to OL9 as well.