RMAN hangs with SQL*Net Break/reset To Client

Once I tried to recover the standby database over the network using RMAN, the operation started and lasted infinitely in the first file :

RMAN> recover database noredo from service "connection_name" ;

Starting recover at 22-SEP-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=10 device type=DISK
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service dg-bris122-m8f
destination for restore of datafile 00001: +DATA_RCF_CDE/DB_SA_RCF/DATAFILE/system.265.1049997735

Quick analyze shown than rman session on the remote database was INACTIVE with even "SQL*Net Break/reset To Client". The Database Reference says about this event - "The server sends a break or reset message to the client. The session running on the server waits for a reply from the client.". It worth to mention that operating system was Oracle Solaris 11.4, and the both databases situated inside dedicated database zones with dedicated (non-shared) network interfaces, separated by the firewall though. It looked like the server (remote database) send to the client (rman) break message (out-of-bands break or OOB) and the client host didn't received it because of the firewall or the client host hadn't processed it (or unable to).

So I decided to disable OOB in the sqlnet.ora on the rman side via setting the following in sqlnet.ora.

disable_oob = on

It worked (fortunately or not) :)

What does awk script do ?

Sometimes it's difficult to understand what the awk program has done. For better understanding, it's useful to debug the program (-D option). But there is another brief way to produce the outline of what program did - to produce execution trace of the program with help of '--profile' option. Let's consider an example.

Suppose that we have some input file which actually can contain anything. In my case it was the file contained disassembled function. The fragment of input file is the following : 

  0x000000000cd2e32a <+442>:   test   %eax,%eax
  0x000000000cd2e32c <+444>:   je     0xcd2e33a <ksfd_io+458>
  0x000000000cd2e32e <+446>:   callq  0xcdc40f0 <sltrgftime64>
  0x000000000cd2e333 <+451>:   mov    %rax,-0x98(%rbp)
  0x000000000cd2e33a <+458>:   mov    -0x60(%rbp),%rax
  0x000000000cd2e33e <+462>:   mov    0x88(%rax),%r12d

I would like to display all the callq's between two calls kslwtbctx and kgecrs.

The short onliner can be like this :

gawk '!/callq/{next}/kslwtbctx/,/kgecrs/' input

The result is looking like this :

 > awk '!/callq/{next}/kslwtbctx/,/kgecrs/' input   
  0x000000000cd2e3ac <+572>:   callq  0xc9a3e00 <kslwtbctx>
  0x000000000cd2e3c7 <+599>:   callq  0xce573b0 <skgfrgsz>
  0x000000000cd2e3db <+619>:   callq  0xce418b0 <kghstack_alloc>
  0x000000000cd2e449 <+729>:   callq  0xcd32c80 <ksfd_osdrqfil>
  0x000000000cd2e4a2 <+818>:   callq  0xcd334f0 <ksfd_skgfqio>
  0x000000000cd2e4b4 <+836>:   callq  0xce42580 <kghstack_free>
  0x000000000cd2e4ce <+862>:   callq  0xce4e090 <kgecrs>

What does this short onliner do ? To clarify a behavior, use the profiler :

gawk --profile '!/callq/{next}/kslwtbctx/,/kgecrs/' input

The file awkprof.out is generated (after the execution of awk program) and contains the following :

> cat awkprof.out
       # gawk profile, created Wed Sep  8 16:47:20 2021

       # Rule(s)

 4085  ! /callq/ { # 3879
 3879          next
       }

  206  /kslwtbctx/, /kgecrs/ { # 7
    7          print $0
       }

Here we can see that program is consisted in two steps. The first one is checking the every input line on the regexp pattern /callq/. If it does not contain callq part of the line, the next line from the input is read.

The next pattern block is a range pattern which filters all the lines between line contained kslwtbctx and kgecrs, including first and last matched lines. But the output contains only callq instructions because of the first block checking /callq/ pattern before. Every input line is checked by the first block and then by second anyway.

Hope if was useful ! 

Good Luck !

RMAN output like in sqlplus (command, then its output)

There are several ways to organize output of rman script. 

I prefer 'set echo on' way, because it affords to see the output of each command consequently, although the output of block { } goes only after the entire block, and output line from the previous command can potentially interleave with the output of next command.

Let suppose we've got the following script to test :

set echo on

connect target 'c##ddi/password@orcl2 as sysbackup'

select user from dual ;

exit

You might notice 'set echo on' here, it does the main job to obtain easy to read output. According the documentation it can be useful when the opportunity exists to manipulate standard input and standard output (inside of Unix like operating systems, for example).

You can use several ways of getting the stuff to work (it is not a full list of possibilities) :

$ cat conn2.rman | rman > conn2.rman.out

$ rman @ $(pwd)/conn2.rman  > conn2.rman.out

$ rman < conn2.rman > conn2.rman.out

$ rman @ $(pwd)/conn2.rman | tee conn2.rman.out

The output with 'set echo on' but without 'tee' looks like this:

Recovery Manager: Release 12.1.0.2.0 - Production on Wed Sep 1 17:25:54 2021

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

RMAN> ;
echo set on
connect target *

connected to target database: ORCL2 (DBID=1043166856)

RMAN>  

RMAN> select user from dual ;
using target database control file instead of recovery catalog
USER                           
------------------------------
SYSBACKUP                      


RMAN>  

RMAN> exit

Without 'set echo on' the output looks like 'list of all the commands   --> output of all the commands' (with help of 'tee'):

Recovery Manager: Release 12.1.0.2.0 - Production on Wed Sep 1 23:08:47 2021

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

RMAN> set echo on
2>  
3> connect target *
4>  
5> select user from dual ;
6>  
7> exit
echo set on

connected to target database: ORCL2 (DBID=1043166856)

using target database control file instead of recovery catalog
USER                           
------------------------------
SYSBACKUP                      

Recovery Manager complete.

Another output without 'set echo on' and without 'tee' looks like this:

Recovery Manager: Release 12.1.0.2.0 - Production on Wed Sep 1 17:26:50 2021

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

RMAN>  
connected to target database: ORCL2 (DBID=1043166856)

RMAN>  
RMAN>  
using target database control file instead of recovery catalog
USER                           
------------------------------
SYSBACKUP                      

RMAN>  
RMAN>  

Recovery Manager complete.

P.S. Sometimes we can see the output from connect command trapped into output of select statement.

Good Luck !

rman ORA-01031 RMAN-00554 from command line after connecting AS SYSBACKUP

The way of connecting to database by rman from command line and from the script is a little bit different (the difference is in connection string).

For example, for operating system command line use '"..."' form :

$ rman  target '"c##ddi@orcl2 as sysbackup"' 

For the script use '...' form

RMAN> connect target 'c##ddi@orcl2 as sysbackup'

P.S. For channels configuration use '"..."' form.

Unable to start CRS after ASM group mounted temporary on different ASM instance

There was a situation when I had to mount disk group of one ASM instance on another to simplify moving of 15 TB database. What I did :

1. Prepared new CRS and new ASM instance with single disk group (NORMAL redundancy). CRS places OCR and voting files on this group.

2. Stopped newly created CRS.

3. On ASM instance contained the database I changed asm_diskstring parameter to get access to newly asm disk group.

4. Mounted newly created ASM disk group on ASM instance with database.

5. Backed up database via RMAN to new ASM disk group.

6. Dismounted newly created ASM disk group on ASM instance with database and changed asm_diskstring parameter back to the value it had before.

7. Started CRS by crsctl start crs

And nothing happened... :)

The process list looked like this one :

# ps -u oracle
 PID TTY         TIME CMD
62224 ?           0:07 diskmon
61638 ?           0:04 oraagent.bin
61687 ?           0:04 evmd.bin
61955 ?           0:08 gipcd.bin
61924 ?           0:03 evmlogger.bin
64258 pts/9       0:00 less
64665 pts/9       0:00 bash
46926 ?           0:04 gpnpd.bin
11263 ?           0:04 ocssd.bin

The ocssd.trc log file said 'no voting files found' :

2021-08-28 23:29:45.985 :    CSSD:29: [     INFO] clssnmvDiskVerify: Successful discovery of 0 disks
2021-08-28 23:29:45.985 :    CSSD:29: [     INFO] clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2021-08-28 23:29:45.985 :    CSSD:29: [     INFO] clssnmvFindInitialConfigs: No voting files found
2021-08-28 23:29:45.986 :    CSSD:29: [     INFO] (:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
2021-08-28 23:29:46.877 :    CSSD:11: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization

It turned out that voting files was removed by foreign ASM instance :)

2021-08-28T19:47:39.007588+03:00
SUCCESS: alter diskgroup datac6 mount  
2021-08-28T19:47:41.312845+03:00
NOTE: Attempting voting file refresh on diskgroup DATAC6
NOTE: Refresh completed on diskgroup DATAC6. No voting file found.
NOTE: Voting file relocation is required in diskgroup DATAC6
NOTE: Attempting voting file relocation on diskgroup DATAC6
NOTE: voting file deletion (replicated) on grp 3 disk AAA
NOTE: voting file deletion on grp 3 disk AAA
NOTE: voting file deletion (replicated) on grp 3 disk BBB
NOTE: voting file deletion on grp 3 disk BBB
NOTE: voting file deletion (replicated) on grp 3 disk CCC
NOTE: voting file deletion on grp 3 disk CCC
NOTE: No voting file found on diskgroup DATAC6

To resolve this, I needed to create new voting files. 

1. Stop CRS

# crsctl stop crs -f

If command failed or run too long without response, kill ocssd.bin and gpnpd.bin processes by hand. Repeat to confirm the stop 

# crsctl stop crs -f
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.

Also check the process list too.

2. Start CRS in init mode :

# crsctl start crs -excl -nocrs

3. Create new voting files 

 $ crsctl query css votedisk
Located 0 voting disk(s).
$ crsctl replace votedisk +datac6
Successful addition of voting disk 93480b19b7e64f90bf2d5a70e2fdcbfd.
Successful addition of voting disk eca9bf5347fd4f67bf06a2e7c830e342.
Successful addition of voting disk 1f468b939fbe4f91bfc83375c532a6da.
Successfully replaced voting disk group with +datac6.
CRS-4266: Voting file(s) successfully replaced

4. Stop and CRS 

# crsctl stop crs

# crsctl start crs -wait

After these steps the CRS stack worked in the normal mode.

Good Luck !

Yum on Oracle Linux - work through proxy - how ?

Just modify /etc/sysconfig/rhn/up2date file. Set 

enableProxy=1
httpProxy=http://ip_address_of_proxy:port

Use man up2date for details.

How to resize logical volume group (LVM) (example on Oracle Linux VM VirtualBox)

If possible, just add another physical disk into LVM configuration and enlarge logical volume using lvresize command. It's much better and lesser error pruning behavior. The following scratched steps can be used when there are no possibilities to have another physical disk or there is a decision to use existing disk.

0. Identify LV and underlying physical volume to extend (use pvdisplay command)

1. Shutdown VM

2. Increase size of hard disk (CLI or VirtualBox virtual media manager)

3. Start the VM (might be in single user mode)

4. Modify partition table using parted. In other words, remove and recreate new partition with desirable size.

Steps depend on what kind of partition (primary or extended you need to extend). It also depend on are there any partitions behind being resized one. You may drop this partition (which you need to resize) and all the other partitions behind or recreate expended partition and logical partitions completely.

5. Reread modified partition table - reboot or blockdev --rereadpt etc.

6. Extend underlying LVM physical volume (correspond to increased partition) using pvresize command, for example :

# pvresize /dev/sda2

7. Increase size of logical volume 

# lvresize --verbose --extents +100%FREE /dev/vg_db01/lv_root

# lvdisplay

8. Increase filesystem size. In my case it was ext4 (possible to increase online) :

# resize2fs /dev/mapper/vg_db01-lv_root

That's it ! Good Luck !