sqlplus easy connect could not connect on linux ORA-12504 ORA-01017

When you try to connect to database using sqlplus with collaboration of easy connect feature, you may get a bit confused by not having successfully logged in when you don't enter user's password inline. In other case (with inline password), the following command will be ok :

sqlplus username/password@IP_OR_HOSTNAME:PORT/SERVICE_NAME

and following will not:

sqlplus username@IP_OR_HOSTNAME:PORT/SERVICE_NAME

In such situation you'll get sqlplus breaf help text. The workaround is using single or double quotes with escape symbols :

sqlplus username@\"IP_OR_HOSTNAME:PORT/SERVICE_NAME\"

or

sqlplus username@\'IP_OR_HOSTNAME:PORT/SERVICE_NAME\'

You may catch errors ORA-12504 ORA-01017 if you place quotes in wrong place. For example, you'll get ORA-01017 in the following case :

sqlplus \'username@IP_OR_HOSTNAME:PORT/SERVICE_NAME\'

or  ORA-12504 when you'll not use escape quoting, like in :

sqlplus username@"IP_OR_HOSTNAME:PORT/SERVICE_NAME"

Good luck !

mount.cifs error 13 another case

If you try to mount windows shared resource to your linux folder and get "mount error(13): Permission denied" error, even if username and password are correct and checked 100 times, another credential setting like "domain=" could help you.

1. If your windows in domain and windows user is a local windows user, then use "domain=windows_hostname" credential setting.

2. If your windows in domain and windows user is a domain windows user, then use "domain=windows_domain_name" credential setting.

Good luck !

Oracle offline documentation links always open in new window

First of all, try to renew downloaded documentation, recent versions (at least September 2016) doesn't have such issue. If you can't, the following is for you.

Later offline versions of Oracle 11g and 12c offline documentations use java scripts for generating links. During offline browsing every link opens new window, which is uncomfortable for navigating/reading etc. Playing with styles in the directory

(unzipped_documentation_directory)/dcommon

 didn't bring any result as well as modifying files

(unzipped_documentation_directory)/dcommon/js/common.js
(unzipped_documentation_directory)/dcommon/js/headfoot.js
(unzipped_documentation_directory)/dcommon/js/headfoot_prev.js

 for changing strings containing 'img.style.display = "inline"' to 'img.style.display = "none"' etc.

The following workaround doesn't claim to be a panacea.

For myself I've decided to change firefox setting "browser.link.open_newwindow" to 1 with help of about:config. This setting prevents to open new windows and tabs for all web pages. But it's always possible to use right mouse button or shift/ctrl+left mouse button to open links in new window/tab respectivly, allowing you to make decision whether display page in the current window or not.

The similar possibility exists in other popular browsers.

undefined reference to `nnfyboot'

When installing or relinking Oracle executables, it may be possible to see such error. The solution is :
1. Install addtitional packages (if not installed, 32bit packages such as glibc-32bit,glibc-devel-32bit,gcc-32bit and other required).
2. look through install.log and search messages kind of 'No such file or directory'. Try to fix errors (see item 1) and run all failed commands by hand in console, using oracle account.
For example, in following text

INFO: (if [ "compile" = "compile" ] ; then \
          echo "Building 32bit version of nnfgt.o"; \
          /opt/oracle/product/10gR2/db/bin/gennfgt > nnfgt.c ;\
          gcc -m32  -c nnfgt.c ;\
          rm -f /opt/oracle/product/10gR2/db/lib32/nnfgt.o ;\
          mv nnfgt.o /opt/oracle/product/10gR2/db/lib32/ ;\
          /usr/bin/ar rv /opt/oracle/product/10gR2/db/lib32/libn10.a /opt/oracle/product/10gR2/db/lib32/nnfgt.o ;\
          echo "Building 64bit version of nnfgt.o"; \
          /opt/oracle/product/10gR2/db/bin/gennfgt > nnfgt.c ;\
          gcc  -c nnfg
INFO: t.c ;\
          rm -f /opt/oracle/product/10gR2/db/lib/nnfgt.o ;\
          mv nnfgt.o /opt/oracle/product/10gR2/db/lib/ ;\
          /usr/bin/ar rv /opt/oracle/product/10gR2/db/lib/libn10.a /opt/oracle/product/10gR2/db/lib/nnfgt.o ; fi)

INFO: Building 32bit version of nnfgt.o

INFO: In file included from /usr/include/features.h:371,
                 from /usr/include/sys/types.h:27,
                 from nnfgt.c:7:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory

INFO: mv: cannot stat `nnfgt.o': No such file or directory

INFO: /usr/bin/ar: /opt/oracle/product/10gR2/db/lib32/nnfgt.o: No such file or directory

INFO: Building 64bit version of nnfgt.o

INFO: r - /opt/oracle/product/10gR2/db/lib/nnfgt.o

The file gnu/stubs-32.h, needed to build 32-bit version of nnfgt.o, needed for construction of libclntsh.so, is absent. Install required package glibc-devel-32bit and run by hand above commands.

3. relink executables running relink command.

The active version of Oracle Clusterware is not 10g Release 2

Silent install of 10gR2 database on 12c or 11g clusterware environment fails with following error :

"The active version of Oracle Clusterware is not 10g Release 2"

Flags such as "-ignore..." don't help.

So the solution is to edit file /stage/prereq/db/db_prereq.xml and remove whole PREREQUISITE NAME="Detect10.2CRS" section, or remove its content,related to rules.

After that you'll be able to complete the installation. 

And don't forget to pin cluster nodes with:

/bin/crsctl pin css -n

Hanged or stalled Grid Infrastructure (11g or 12c) installation performing remote operations on Linux

The installation process goes fine, but suddenly is's stalled in the middle (50-60%) for unknown reason. That's becouse java installation process unable to transmit copied data (files and dirs) to oracle processes named ractrans, running on remote nodes in listening mode and basically performing all work for filling up remote oracle homes.

In my case that was becouse of sysctl.conf settings.

The content I had :

net.core.rmem_default = 25165824
net.core.wmem_default = 25165824
net.core.rmem_max = 25165824
net.core.wmem_max = 25165824
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 134217728 134217728 134217728
net.ipv4.tcp_wmem = 134217728 134217728 134217728
net.core.netdev_max_backlog = 300000

The content I had to modify :

net.core.rmem_default = 4194304
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 262144
net.ipv4.tcp_rmem = 4096        87380   174760
net.ipv4.tcp_wmem = 4096        16384   131072
net.core.netdev_max_backlog = 1000

Honestly, I suspect, only net.ipv4.tcp_rmem and net.ipv4.tcp_wmem parameters play main role in this issue. Just in case I decided to set default values for other in the list above.

I.e., decreasing amount of memory per tcp socket and buffer space, slowing ethernet speed, took results.

P.S.

1. Just as surveillance. Oracle Universal Installer creates six (6) process per remote node to sync contents of grid_home. If you perform installation only for two nodes, you could not face with this issue.

2. It generally depends on speed of your public network stack. If you have fast enough switches and adapters, probably you will not bump into it.

3. It also depends on speed your storage (including mounting options). Any measure decreasing speed of copying files, can help.

How to modify interconnect device/settings in 11gR2 Clusterware

Let's imagine the following situation.
You have two interconnect interfaces. Suddenly, all of them have become unaccessible, your rac immediately stopped. How to rule it out ?
Build up new interconnect, using new cards, of course.
But you will not be able to run clusterware services like HAIP and other becouse your new interconnect device name does not consistent with interconnect device names in OLR (Oracle Local Registry on each node) and OCR.
Actions ?

1. Rename you interconnect device by operating system means (modify udev net-persistent-device-names file, for example, and run udevadm trigger command on each node), and rerun your clusterware afterwards.

OR

2. Modify OLR content on each node

$ cd $CRS_HOME/gpnp/$(hostname)/profiles/peer

Here will be signed profile.xml, modifications only allowed with gpnptool ($CRS_HOME/bin) utility.

query profile.2 in some manner like that:
$ gpnptool getpval -p=profile.2 -net
$ gpnptool getpval -p=profile.2 -net2:net_ip
$ gpnptool getpval -p=profile.2 -net2:net_ada

unsign profile.xml to new profile.2 :
$ gpnptool unsign -p=profile.xml -o=profile.2

rename interconnect adapter name in profile.2 :

$ gpnptool edit -p=profile.2 -net2:net_ada=new_if_name -o=profile.2 -ovr

set if needed new IP subnet for interconnect :
$ gpnptool edit -p=profile.2 -net2:net_ip=subnet.0 -o=profile.2 -ovr

sign profile.2 :
$ gpnptool sign -p=profile.2 -o=profile.2.xml.signed -w=/$CRS_HOME/gpnp/$(hostname)/wallets/peer

make a rotation :
$ mv profile.xml profile.xml.orig

$ cp profile.2.xml.signed profile.xml

run clusterware :
# /etc/init.d/ohasd start

After successful running, with help of oifcfg, delete old interconnect data from OCR by
$ oifcfg delif -global old_if_name.

Repeat above steps related to profile.xml on each node