Unable to start CRS after ASM group mounted temporary on different ASM instance

There was a situation when I had to mount disk group of one ASM instance on another to simplify moving of 15 TB database. What I did :

1. Prepared new CRS and new ASM instance with single disk group (NORMAL redundancy). CRS places OCR and voting files on this group.

2. Stopped newly created CRS.

3. On ASM instance contained the database I changed asm_diskstring parameter to get access to newly asm disk group.

4. Mounted newly created ASM disk group on ASM instance with database.

5. Backed up database via RMAN to new ASM disk group.

6. Dismounted newly created ASM disk group on ASM instance with database and changed asm_diskstring parameter back to the value it had before.

7. Started CRS by crsctl start crs

And nothing happened... :)

The process list looked like this one :

# ps -u oracle
 PID TTY         TIME CMD
62224 ?           0:07 diskmon
61638 ?           0:04 oraagent.bin
61687 ?           0:04 evmd.bin
61955 ?           0:08 gipcd.bin
61924 ?           0:03 evmlogger.bin
64258 pts/9       0:00 less
64665 pts/9       0:00 bash
46926 ?           0:04 gpnpd.bin
11263 ?           0:04 ocssd.bin

The ocssd.trc log file said 'no voting files found' :

2021-08-28 23:29:45.985 :    CSSD:29: [     INFO] clssnmvDiskVerify: Successful discovery of 0 disks
2021-08-28 23:29:45.985 :    CSSD:29: [     INFO] clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2021-08-28 23:29:45.985 :    CSSD:29: [     INFO] clssnmvFindInitialConfigs: No voting files found
2021-08-28 23:29:45.986 :    CSSD:29: [     INFO] (:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
2021-08-28 23:29:46.877 :    CSSD:11: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization

It turned out that voting files was removed by foreign ASM instance :)

2021-08-28T19:47:39.007588+03:00
SUCCESS: alter diskgroup datac6 mount  
2021-08-28T19:47:41.312845+03:00
NOTE: Attempting voting file refresh on diskgroup DATAC6
NOTE: Refresh completed on diskgroup DATAC6. No voting file found.
NOTE: Voting file relocation is required in diskgroup DATAC6
NOTE: Attempting voting file relocation on diskgroup DATAC6
NOTE: voting file deletion (replicated) on grp 3 disk AAA
NOTE: voting file deletion on grp 3 disk AAA
NOTE: voting file deletion (replicated) on grp 3 disk BBB
NOTE: voting file deletion on grp 3 disk BBB
NOTE: voting file deletion (replicated) on grp 3 disk CCC
NOTE: voting file deletion on grp 3 disk CCC
NOTE: No voting file found on diskgroup DATAC6

To resolve this, I needed to create new voting files. 

1. Stop CRS

# crsctl stop crs -f

If command failed or run too long without response, kill ocssd.bin and gpnpd.bin processes by hand. Repeat to confirm the stop 

# crsctl stop crs -f
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.

Also check the process list too.

2. Start CRS in init mode :

# crsctl start crs -excl -nocrs

3. Create new voting files 

 $ crsctl query css votedisk
Located 0 voting disk(s).
$ crsctl replace votedisk +datac6
Successful addition of voting disk 93480b19b7e64f90bf2d5a70e2fdcbfd.
Successful addition of voting disk eca9bf5347fd4f67bf06a2e7c830e342.
Successful addition of voting disk 1f468b939fbe4f91bfc83375c532a6da.
Successfully replaced voting disk group with +datac6.
CRS-4266: Voting file(s) successfully replaced

4. Stop and CRS 

# crsctl stop crs

# crsctl start crs -wait

After these steps the CRS stack worked in the normal mode.

Good Luck !

Yum on Oracle Linux - work through proxy - how ?

Just modify /etc/sysconfig/rhn/up2date file. Set 

enableProxy=1
httpProxy=http://ip_address_of_proxy:port

Use man up2date for details.

How to resize logical volume group (LVM) (example on Oracle Linux VM VirtualBox)

If possible, just add another physical disk into LVM configuration and enlarge logical volume using lvresize command. It's much better and lesser error pruning behavior. The following scratched steps can be used when there are no possibilities to have another physical disk or there is a decision to use existing disk.

0. Identify LV and underlying physical volume to extend (use pvdisplay command)

1. Shutdown VM

2. Increase size of hard disk (CLI or VirtualBox virtual media manager)

3. Start the VM (might be in single user mode)

4. Modify partition table using parted. In other words, remove and recreate new partition with desirable size.

Steps depend on what kind of partition (primary or extended you need to extend). It also depend on are there any partitions behind being resized one. You may drop this partition (which you need to resize) and all the other partitions behind or recreate expended partition and logical partitions completely.

5. Reread modified partition table - reboot or blockdev --rereadpt etc.

6. Extend underlying LVM physical volume (correspond to increased partition) using pvresize command, for example :

# pvresize /dev/sda2

7. Increase size of logical volume 

# lvresize --verbose --extents +100%FREE /dev/vg_db01/lv_root

# lvdisplay

8. Increase filesystem size. In my case it was ext4 (possible to increase online) :

# resize2fs /dev/mapper/vg_db01-lv_root

That's it ! Good Luck !

How to extend ASM SAN disk group in Solaris zone

1. First of all, identify disk's paths to work with. Use sqlplus, kfod or OS utilities to get the information.

Example using sqlplus (GI home) :

% sqlplus / as sysdba

SYS@+ASM> col path for a50

SYS@+ASM> select path,os_mb,total_mb,free_mb from v$asm_disk ;

Example using kfod (GI home) :

% kfod disks=all

2. Next, increase the space on LUNs using storage system software (on HPE 3PAR, it was made by growvv command)

3. Next, on the global zone, use format utility to extend the space on OS side:

For example :

# format /dev/rdsk/c0t60002AC00000000002003CDE000095DDd0

selecting c0t60002AC00000000002003CDE000095DDd0 <3PARdata-VV-3226-2.49TB>
[disk formatted]
Note: capacity in disk label is less than the disk capacity.
Select <partition> <expand> to adjust the label capacity.

FORMAT MENU:
       disk       - select a disk
       type       - select (define) a disk type
       partition  - select (define) a partition table
       current    - describe the current disk
       format     - format and analyze the disk
       repair     - repair a defective sector
       label      - write label to the disk
       analyze    - surface analysis
       defect     - defect list management
       backup     - search for backup labels
       verify     - read and display labels
       inquiry    - show disk ID
       volname    - set 8-character volume name
       !<cmd>     - execute <cmd>, then return
       quit
format> p


PARTITION MENU:
       0      - change `0' partition
       1      - change `1' partition
       2      - change `2' partition
       3      - change `3' partition
       4      - change `4' partition
       5      - change `5' partition
       6      - change `6' partition
       expand - expand label to use the maximum allowed space
       select - select a predefined table
       modify - modify a predefined partition table
       name   - name the current table
       print  - display the current table
       label  - write partition map and label to the disk
       !<cmd> - execute <cmd>, then return
       quit
partition> ex
The expanded capacity is added to the unallocated space.
partition> p

Current partition table (original):
Total disk sectors available: 5343526845 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector          Size          Last Sector
 0        usr    wm                64         2.00TB           4294950847
 1 unassigned    wm                 0            0                0
 2 unassigned    wm                 0            0                0
 3 unassigned    wm                 0            0                0
 4 unassigned    wm                 0            0                0
 5 unassigned    wm                 0            0                0
 6 unassigned    wm                 0            0                0
 8   reserved    wm        5343526879         8.00MB           5343543262

partition> 0
Part      Tag    Flag     First Sector          Size          Last Sector
 0        usr    wm                64         2.00TB           4294950847

Enter partition id tag[usr]:
Enter partition permission flags[wm]:
Enter new starting sector[64]:
Enter partition size[4294950784b, 4294950847e, 2097143mb, 2047gb, 1tb]: $
partition> p
Current partition table (unnamed):
Total disk sectors available: 5343526845 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector          Size          Last Sector
 0        usr    wm                64         2.49TB           5343526877
 1 unassigned    wm                 0            0                0
 2 unassigned    wm                 0            0                0
 3 unassigned    wm                 0            0                0
 4 unassigned    wm                 0            0                0
 5 unassigned    wm                 0            0                0
 6 unassigned    wm                 0            0                0
 8   reserved    wm        5343526879         8.00MB           5343543262

partition> label
Ready to label disk, continue? y

partition> q


FORMAT MENU:
       disk       - select a disk
       type       - select (define) a disk type
       partition  - select (define) a partition table
       current    - describe the current disk
       format     - format and analyze the disk
       repair     - repair a defective sector
       label      - write label to the disk
       analyze    - surface analysis
       defect     - defect list management
       backup     - search for backup labels
       verify     - read and display labels
       inquiry    - show disk ID
       volname    - set 8-character volume name
       !<cmd>     - execute <cmd>, then return
       quit
format> ver

Volume name = <        >
ascii name  = <3PARdata-VV-3226-2.49TB>
bytes/sector    =  512
sectors = 5343543295  
accessible sectors = 5343543262
Part      Tag    Flag     First Sector          Size          Last Sector
 0        usr    wm                64         2.49TB           5343526877
 1 unassigned    wm                 0            0                0
 2 unassigned    wm                 0            0                0
 3 unassigned    wm                 0            0                0
 4 unassigned    wm                 0            0                0
 5 unassigned    wm                 0            0                0
 6 unassigned    wm                 0            0                0
 8   reserved    wm        5343526879         8.00MB           5343543262

format> q

4. Repeat previous step for the rest of LUNs

5. Resize ASM diskgroup 

SYS@+ASM> alter diskgroup disk_group_name resize all ;

You might encounter the error 

ORA-15099, 00000, "disk '%s' is larger than maximum size of %s MBs" 

In my case it was low compatibility.rdbms value (10.1) for the altered diskgroup.

Good Luck !