CRSD fails to start with ORA-01017 invalid username/password

The grid alert.log says smth like this :

 2022-06-24 09:01:27.377 [OHASD(34093)]CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hostname'
2022-06-24 09:01:27.835 [CRSD(30213)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /u01/oracle/diag/crs/hostname/crs/trace/crsd.trc.
2022-06-24 09:01:27.842 [CRSD(30213)]CRS-0804: Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage Storage layer error [I
nsufficient quorum to open OCR devices] [0]]. Details at (:CRSD00111:) in /u01/oracle/diag/crs/hostname/crs/trace/crsd.trc

crsd.trc says something like this :

 2022-06-24 09:01:27.834 :  OCRRAW:1: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied


2022-06-24 09:01:27.834*:kgfo.c@1014: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied

2022-06-24 09:01:27.834 :  OCRRAW:1: -- trace dump on error exit --

2022-06-24 09:01:27.834 :  OCRRAW:1: Error [kgfoAl06] in [kgfokge] at kgfo.c:3182

2022-06-24 09:01:27.834 :  OCRRAW:1: ORA-01017: invalid username/password; logon denied


2022-06-24 09:01:27.834 :  OCRRAW:1: Category: 7  

2022-06-24 09:01:27.834 :  OCRRAW:1: DepInfo: 1017

2022-06-24 09:01:27.834 :  OCRRAW:1: -- trace dump end --

 OCRASM:1: SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge

2022-06-24 09:01:27.834 :  OCRASM:1: ASM Error Stack : ORA-01017: invalid username/password; logon denied

2022-06-24 09:01:27.835 :  OCRASM:1: proprasmo: kgfoCheckMount returned [7]
2022-06-24 09:01:27.835 :  OCRASM:1: proprasmo: The ASM instance is down
2022-06-24 09:01:27.835 :  OCRRAW:1: proprioo: Failed to open [+DATAC6/cl_name/OCRFILE/registry.255.1081797515]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2022-06-24 09:01:27.835 :  OCRRAW:1: proprioo: No OCR/OLR devices are usable
 OCRUTL:1: u_fill_errorbuf: Error Info : [Insufficient quorum to open OCR devices]
default:1: u_set_gbl_comp_error: comptype '107' : error '0'
2022-06-24 09:01:27.836 : default:1: clsvactversion:4: Retrieving Active Version from local storage.
2022-06-24 09:01:27.838 : CSSCLNT:1: clssgsgrppubdata: group (ocr_cl_name) not found
2022-06-24 09:01:27.838 :  OCRRAW:1: proprio_repairconf: Failed to retrieve the group public data. CSS ret code [20]
2022-06-24 09:01:27.838 :  OCRRAW:1: proprioo: Failed to auto repair the OCR configuration.
2022-06-24 09:01:27.838 :  OCRRAW:1: proprinit: Could not open raw device
2022-06-24 09:01:27.841 :  OCRAPI:1: a_init:16!: Backend init unsuccessful : [26]
2022-06-24 09:01:27.841 :  OCRAPI:1: estack 'PROC-00026: Error while accessing the physical storage'
2022-06-24 09:01:27.842 :  CRSOCR:1: [    ERROR]  OCR context init failure.  Error: PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR
devices] [0]
2022-06-24 09:01:27.843 :    CRSD:1: [     NONE]  Created alert : (:CRSD00111:) :  Could not init OCR, error: PROC-26: Error while accessing the physical storage Storage layer error [Insuff
icient quorum to open OCR devices] [0]
2022-06-24 09:01:27.843 :    CRSD:1: [    ERROR] [PANIC] CRSD exiting: Could not init OCR, code: 26
2022-06-24 09:01:27.843 :    CRSD:1: [     INFO]  Done.

2022-06-24 09:01:27.834 :  OCRRAW:1: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied


2022-06-24 09:01:27.834*:kgfo.c@1014: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied

2022-06-24 09:01:27.834 :  OCRRAW:1: -- trace dump on error exit --

2022-06-24 09:01:27.834 :  OCRRAW:1: Error [kgfoAl06] in [kgfokge] at kgfo.c:3182

2022-06-24 09:01:27.834 :  OCRRAW:1: ORA-01017: invalid username/password; logon denied


2022-06-24 09:01:27.834 :  OCRRAW:1: Category: 7  

2022-06-24 09:01:27.834 :  OCRRAW:1: DepInfo: 1017

2022-06-24 09:01:27.834 :  OCRRAW:1: -- trace dump end --

 OCRASM:1: SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge

2022-06-24 09:01:27.834 :  OCRASM:1: ASM Error Stack : ORA-01017: invalid username/password; logon denied

2022-06-24 09:01:27.835 :  OCRASM:1: proprasmo: kgfoCheckMount returned [7]
2022-06-24 09:01:27.835 :  OCRASM:1: proprasmo: The ASM instance is down

The cause of this has been not corrupted ASM pwfile or something like that. It was much more easier - accidentally changed default group id of grid software owner. Instead of 

% id -a oracle
uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba)

it was set to

% id -a oracle
uid=1000(oracle) gid=1002(dba) groups=1001(oinstall),1002(dba)

Fixing it via usermod and restarting the server resolved the issue 😀

[FATAL] Error in invoking target 'irman ioracle...' of makefile 'ORACLE_HOME/rdbms/lib/ins_rdbms.mk'. See 'ORA_INVENTORY/logs/InstallActions.../installActions....log' for details

This error occurred during installation of new Oracle RU to new oracle home. The linking phase was going on. The out-of-memory operating system protection mechanism had killed the linking process. 

The main cause of it had been vm.swappiness=0 settings (had been set in sysctl.conf before); in other words, operating system was disabling any swap activity if it could do so. Any values more that 0 resolved the issue (by repeating the installation).

Exception Occured while executiong Apply Operation: ApplySession failed in system modification phase... 'Execution of PRE script failed, with return value = 127'

Today I got the following error installing the RU7 for OEM 13.5 :

$ /u01/app/oem13c/emcc_135/OMSPatcher/omspatcher apply -property_file /u01/app/oem13c/aux/property_file

OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved. 

...

OMSPatcher failed to apply following patch(es) "34003606" "34003614" "34003617" "34003654" "34003678"  to core/plugin Oracle home(s).


Logfile said that the error occured while executing PRE script :

[Jun 3, 2022 10:44:33 AM]    NApplyReport Log file: /u01/app/oem13c/emcc_135/cfgtoollogs/opatch/opatch2022-06-03_10-41-10AM_1.log
[Jun 3, 2022 10:45:30 AM]    Exception Occured while executiong Apply Operation: ApplySession failed in system modification phase... 'Execution of PRE script failed, with return value = 127
'
[Jun 3, 2022 10:45:30 AM]    NApply Logfile: /u01/app/oem13c/emcc_135/cfgtoollogs/opatch/opatch2022-06-03_10-41-10AM_1.log

The main cause of that was the spaces in the path of unpacked RU patch directory. But what to do in this case ? Trying to resume (or reapply) the installation is obvious, but it wouldn't work because of the spaces in the path neither omspatcher rollback would.

But how to overcome spaces-in-the-path issue ? Go to the OMS oracle home and fix it manually 😉 :

1. unzip/move the patch to the new location without spaces in the path.

2. edit some files in the OMS oracle home

$ cd $ORACLE_HOME/.omspatcher_storage

Edit the files :

- .patch_failure.properties

- restoreSteps.txt

- file under the execution_txt variable inside .patch_failure.properties

3. run omspatcher resume command and Bob is your uncle !

Good Luck !