Tuesday, July 28, 2015

Oracle 11g Release 2 (11.2.0.4.0) RAC Configuration on HP-UX Itanium (11.31) Server

This document show steps of installation and setting up 2-Node Oracle 11g Release 2 (11.2.0.4.0) RAC on HP-UX Itanium (11.31) server.

>> 2-Node RAC Initial
Operating system: HPUX Itanium 11.31
Oracle Version: 11.2.0.4.0
Server: ALL RAC Nodes + DNS Server

Grid Infrastructure software (Clusterware + ASM 11.2.0.4.0)
        GRID_HOME=/u01/app/11.2.0.4/grid
        ORACLE_BASE=/u01/app/grid
        ORACLE_HOME=$GRID_HOME
        Owner: grid (Primary Group: oinstall, Secondary Group: dba,asmdba,asmadmin,asmoper)
        Permission: 775
        OCR/Voting Disk: ASM
        Oracle Inventory Location: /u01/app/oraInventory

Oracle Database software (RAC 11.2.0.4.0)
        ORACLE_BASE=/u02/app/oracle
        ORACLE_HOME=/u02/app/oracle/product/11.2.0.4/dbhome_1
        Owner: oracle (Primary Group: oinstall, Secondary Group: dba,oper,asmadmin)
        Permission: 775


>> Installation divided into Eight main parts:
1. Pre-Installation tasks.
2. Installation of oracle 11gR2 Grid Infrastructure (11.2.0.4.0).
3. Installation of Oracle 11gR2 RDBMS Software (11.2.0.4.0).
4. Run ASMCA to create diskgroups.
5. Separate VOTING from OCR diskgroups to VOTING diskgroup.  
6. Run NETCA to create Listener [Additional Port].
7. Run DBCA to create database.
8. Tablespace, Schema, Export & Import and Others Configuration.

>> 1. Pre-Installation Tasks.
1.1. Configure NTP
      [...Configure NTP...]
                     OR

/sbin/init.d/xntpd stop
mv /etc/ntp.conf /etc/ntp.conf.bak

1.2. Configure DNS & Verify DNS from RAC Nodes
>>/etc/resolv.conf (on RAC Nodes)
      search nblitd.com
      nameserver 172.31.xx.xx

>>/etc/resolv.conf (on RAC Nodes)
      service named start
      chkconfig --level 35 named on

1.3. Create OS Groups using the command below. Enter these Command as "root" user.
/usr/sbin/groupadd -g 700 oinstall
/usr/sbin/groupadd -g 701 dba
/usr/sbin/groupadd -g 702 oper
/usr/sbin/groupadd -g 703 asmdba
/usr/sbin/groupadd -g 704 asmoper
/usr/sbin/groupadd -g 705 asmadmin

1.4. Create users that will own Oracle & Gird software.Enter these Command as "root" user.
/usr/sbin/useradd  -m -u 108 -g oinstall -G dba,oper,asmdba -d /home/oracle oracle
/usr/sbin/useradd  -m -u 109 -g oinstall -G dba,asmdba,asmoper,asmadmin -d /home/grid grid


Note: -m is used to create user's home directory[i.e. /home/oracle, /home/grid] . if you do not use 
-m in your useradd then you can get .profile by following:    
cd /etc/skel
cp .cshrc .exrc .login .profile /home/oracle/
chown oracle:oinstall /home/oracle

cd /etc/skel
cp .cshrc .exrc .login .profile /home/grid/
chown oracle:oinstall /home/grid

 >>read, write, execute permission to directory or profile
#chmod -R 777 /home/grid [On full directory]
or
>>>> Login as grid
$chmod -R 777 .profile [On grid profile]

#chmod -R 777 /home/oracle   [On full directory]
or
>>>> Login as oracle
$chmod -R 777 .profile [On oracle profile]

# id oracle
uid=108(oracle) gid=700(oinstall) groups=701(dba),702(oper),703(asmdba)
# id grid
uid=109(grid) gid=700(oinstall) groups=701(dba),703(asmdba),704(asmoper),705(asmadmin)

1.5. Set the password for the Oracle & Grid account.Enter these Command as "root" user.
# passwd grid
Changing password for grid
New password:
Re-enter new password:
Passwd successfully changed
# passwd oracle
Changing password for oracle
New password:
Re-enter new password:
Passwd successfully changed

1.6. Sample /etc/hosts file will be as below.

127.0.0.1       localhost       loopback
#################Public######################
172.31.xx.xx    dcpdb1.xxx.com      dcpdb1
172.31.xx.xx    dcpdb2.xxx.com      dcpdb2
#################Private#########################
10.x.x.1     dcpdb1-prv
10.x.x.2     dcpdb2-prv
################### VIP ##########################
172.31.xx.xx    dcpdb1-vip.xxx.com   dcpdb1-vip
172.31.xx.xx    dcpdb2-vip.xxx.com   dcpdb2-vip

Note: In SCAN IP is use in host file then only one IP will entry to host file. 

1.7. Setting Kernel Parameters.
On ALL RAC Nodes, verify that kernel parameters value greater or equal to recommended values. kernel parameters value can be set manually (or can also be fixed during installation using runfixup.sh).  
Note: On all RAC Nodes, if the current value of any parameter is higher than the value listed in this table, then do not change the value of that parameter.

>> Modifying Kernel Setting Using KCTUNE
Use /usr/sbin/kctune to view or change kernel parameters.

For example, to view kernel parameters, enter kctune with the -d flag. For example:
#/usr/sbin/kctune -d aio_listio_max

To modify a kernel parameter, enter kctune variable=value. For example:
#/usr/sbin/kctune aio_listio_max=512
Complete this procedure on ALL RAC nodes.

Soft & Hard Limit Setup:
/usr/sbin/kctune maxuprc=3686
/usr/sbin/kctune msgmni=4096
/usr/sbin/kctune msgtql=4096
/usr/sbin/kctune ncsize=35840 
/usr/sbin/kctune maxfiles_lim=63488 
/usr/sbin/kctune ninode=34816 
/usr/sbin/kctune semmni=4096 
/usr/sbin/kctune semmns=8192
/usr/sbin/kctune semmnu=4092
/usr/sbin/kctune shmseg=512
/usr/sbin/kctune -s maxfiles=16384
/usr/sbin/kctune -s maxfiles_lim=65536
/usr/sbin/kctune -s maxssiz=134217728
/usr/sbin/kctune -s shmmni=4096
/usr/sbin/kctune -s shmseg=512


1.8. Software(Package) requirement list for HPUX Itanium Platform[Install below package or upper versions].
Note: For Password less SSH Connectivity between nodes below Software packages must be installed on each & every node.  
gettext-0.19.5-ia64-11.31.depot
libiconv-1.14-ia64-11.31.depot
bash-4.3.033-ia64-11.31.depot
termcap-1.3.1-ia64-11.31.depot

>> To Install patch execute following command:
swinstall -s /tmp/oraclesoft/gettext-0.19.5-ia64-11.31.depot
swinstall -s /tmp/oraclesoft/libiconv-1.14-ia64-11.31.depot
swinstall -s /tmp/oraclesoft/bash-4.3.033-ia64-11.31.depot
swinstall -s /tmp/oraclesoft/termcap-1.3.1-ia64-11.31.depot

 >> To Check/View patchs execute following command:
swlist -l patch |grep PHCO*
swlist -l patch |grep PHSS*

>> Compiler Check
swlist -l product |grep -i compiler  

  
1.9. Create Required Symbolic/Soft Links
ln -s /usr/bin/ssh /usr/local/bin/ssh 
ln -s  /usr/bin/scp /usr/local/bin/scp

Below symbolic links are required only if Motif 2.1 development Environment package (X11MotifDevKit.MOTIF21-PRG) is not installed.

Check [X11MotifDevKit.MOTIF21-PRG] is exists ?  
#swlist -l product |grep -i X11Motif*  

 if not exists then-
#cd /usr/lib
ln -s libX11.3 libX11.sl
ln -s libXIE.2 libXIE.2.sl
ln -s libXext.3 libXext.sl
ln -s libXhp11.3 libXhp11.sl
ln -s libXi.3 libXi.sl
ln -s libXm.4 libXm.sl
ln -s libXp.2 libXp.sl
ln -s libXt.3 libXt.sl
ln -s libXtst.2 libXtst.sl

1.10. Create the Oracle Grid Infrastructure Home Directory [ALL Nodes]
# mkdir -p /u01/app/grid
# chmod -R 775 /u01/app/grid
# mkdir -p /u01/app/11.2.0.4/grid
# chmod -R 775 /u01/app/11.2.0.4/grid
# chown -R grid:oinstall /u01

$vi .profile [["grid" user for Node1]]
# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

##Set up Grid Environment:##
umask 022
export GRID_HOME=/u01/app/11.2.0.4/grid
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=$GRID_HOME
export ORACLE_SID=+ASM1
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

$ vi .profile    [["grid" user for Node2]]
# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

## Set up Grid Environment###
umask 022
export GRID_HOME=/u01/app/11.2.0.4/grid
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=$GRID_HOME
export ORACLE_SID=+ASM2
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

1.11. Create the Oracle Inventory Directory [ALL Nodes]
# mkdir -p /u01/app/oraInventory
# chown -R grid:oinstall /u01/app/oraInventory
# chmod -R 775 /u01/app/oraInventory


1.12. Create the Oracle BASE Directory [ALL Nodes]
# mkdir -p /u02/app/oracle/product/11.2.0.4/dbhome_1
# chown -R oracle:oinstall /u02
# chmod -R 775 /u02/app/oracle/product/11.2.0.4/dbhome_1

$vi .profile [["oracle" user for Node1]]
# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

## Set up Oracle Environment##
umask 022
export ORACLE_BASE=/u02/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.4/dbhome_1
export ORACLE_SID=dcpdb1
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

$vi .profile [["oracle" user for Node2]]
# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

##Set up Oracle Environment####
umask 022
export ORACLE_BASE=/u02/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.4/dbhome_1
export ORACLE_SID=dcpdb2
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

1.13. ASM Disk Group Sizing 
DATA: 600G >>> LUN:15,24,33
FRA: 800G  >>> LUN: 42,67,68,69
OCR: 6G >>> LUN: 134,135,136
VOTE: 6G >>> LUN: 137,138,139
REDO: 40G >>> LUN:140
CONTROL: 35G >>> 141

1.14. Configure Disk Devices for Oracle ASM
a. To make sure that the disks are available, enter the following command on ALL node:
# ioscan -fNnC disk
 Class     I  H/W Path  Driver S/W State   H/W Type     Description
===================================================================
disk      3  64000/0xfa00/0x1   esdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                      /dev/disk/disk3      /dev/disk/disk3_p2   /dev/rdisk/disk3     /dev/rdisk/disk3_p2
                      /dev/disk/disk3_p1   /dev/disk/disk3_p3   /dev/rdisk/disk3_p1  /dev/rdisk/disk3_p3
disk      4  64000/0xfa00/0x2   esdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                      /dev/disk/disk4   /dev/rdisk/disk4
disk      5  64000/0xfa00/0x4   esdisk   CLAIMED     DEVICE       hp      DVDRAM GTA0N
                      /dev/disk/disk5   /dev/rdisk/disk5
disk     15  64000/0xfa00/0x12  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk15   /dev/rdisk/disk15
disk     24  64000/0xfa00/0x13  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk24   /dev/rdisk/disk24
disk     33  64000/0xfa00/0x14  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk33   /dev/rdisk/disk33
disk     42  64000/0xfa00/0x15  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk42   /dev/rdisk/disk42
disk     67  64000/0xfa00/0x16  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk67   /dev/rdisk/disk67
disk     68  64000/0xfa00/0x17  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk68   /dev/rdisk/disk68
disk     69  64000/0xfa00/0x18  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk69   /dev/rdisk/disk69
disk    134  64000/0xfa00/0x19  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk134   /dev/rdisk/disk134
disk    135  64000/0xfa00/0x1a  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk135   /dev/rdisk/disk135
disk    136  64000/0xfa00/0x1b  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk136   /dev/rdisk/disk136
disk    137  64000/0xfa00/0x1c  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk137   /dev/rdisk/disk137
disk    138  64000/0xfa00/0x1d  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk138   /dev/rdisk/disk138
disk    139  64000/0xfa00/0x1e  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk139   /dev/rdisk/disk139
disk    140  64000/0xfa00/0x1f  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk140   /dev/rdisk/disk140
disk    141  64000/0xfa00/0x20  esdisk   CLAIMED     DEVICE       3PARdataVV
                      /dev/disk/disk141   /dev/rdisk/disk141

b. if the ioscan command does not display device name information for a device, enter the following command to install the special device files for any new devices:
#/usr/sbin/insf -e

c. For each disk to add to a disk group, enter the following command on any node to verify that it is not already part of an LVM volume group.
#/sbin/pvdisplay /dev/disk/disk140
if this command display volume group information, the disk is already part of a volume group. The disks that you choose must not be part of an LVM volume group.

d. The disks names for the same disk can be different on each node. A disk can be identified as the same one via the WWID. The WWID of a disk can be checked via the following command:
#scsimgr lun_map -D /dev/rdisk/disk15 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000800010670
scsimgr lun_map -D /dev/rdisk/disk24 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000900010670
scsimgr lun_map -D /dev/rdisk/disk33 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000a00010670
scsimgr lun_map -D /dev/rdisk/disk42 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000b00010670
scsimgr lun_map -D /dev/rdisk/disk67 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000c00010670

scsimgr lun_map -D /dev/rdisk/disk68 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000d00010670
scsimgr lun_map -D /dev/rdisk/disk69 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000e00010670

scsimgr lun_map -D /dev/rdisk/disk134 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000000f00010670

scsimgr lun_map -D /dev/rdisk/disk135 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000001000010670

scsimgr lun_map -D /dev/rdisk/disk136 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000001100010670
scsimgr lun_map -D /dev/rdisk/disk137 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000001200010670

scsimgr lun_map -D /dev/rdisk/disk138 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000001300010670
scsimgr lun_map -D /dev/rdisk/disk139 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000001400010670
scsimgr lun_map -D /dev/rdisk/disk140 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000001600010670
scsimgr lun_map -D /dev/rdisk/disk141 | grep WWID
World Wide Identifier(WWID)    = 0x60002ac0000000000000001700010670

e. Disk Size Check
#diskinfo /dev/rdisk/disk15
SCSI describe of /dev/rdisk/disk15:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 209715200 Kbytes 200 GB
   bytes per sector: 512

#diskinfo /dev/rdisk/disk24
SCSI describe of /dev/rdisk/disk24:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 209715200 Kbytes Size 200GB
   bytes per sector: 512

#diskinfo /dev/rdisk/disk33
SCSI describe of /dev/rdisk/disk33:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 209715200 Kbytes Size 200GB
   bytes per sector: 512

#diskinfo /dev/rdisk/disk42
SCSI describe of /dev/rdisk/disk42:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 209715200 Kbytes Size 200GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk67
SCSI describe of /dev/rdisk/disk67:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 209715200 Kbytes Size 200GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk68
SCSI describe of /dev/rdisk/disk68:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 209715200 Kbytes Size 200GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk69
SCSI describe of /dev/rdisk/disk69:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 209715200 Kbytes Size 200GB
   bytes per sector: 512

# diskinfo /dev/rdisk/disk134
SCSI describe of /dev/rdisk/disk134:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 2097152 Kbytes 2GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk135
SCSI describe of /dev/rdisk/disk135:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 2097152 Kbytes 2GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk136
SCSI describe of /dev/rdisk/disk136:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 2097152 Kbytes 2GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk137
SCSI describe of /dev/rdisk/disk137:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 2097152 Kbytes 2GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk138
SCSI describe of /dev/rdisk/disk138:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 2097152 Kbytes 2GB
   bytes per sector: 512

# diskinfo /dev/rdisk/disk139
SCSI describe of /dev/rdisk/disk139:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 2097152 Kbytes 2GB
   bytes per sector: 512

# diskinfo /dev/rdisk/disk140
SCSI describe of /dev/rdisk/disk140:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 41943040 Kbytes 40GB
   bytes per sector: 512
# diskinfo /dev/rdisk/disk141
SCSI describe of /dev/rdisk/disk141:
             vendor: 3PARdata
         product id: VV
               type: direct access
               size: 36700160 Kbytes 35GB
   bytes per sector: 512

F. Create a special oracle device directory and use mknod to create device paths in oracle folder.
#mkdir /dev/oracle
DATA
ll /dev/rdisk/disk15
crw-r-----   1 bin        sys         13 0x000012 Jul 22 22:40 /dev/rdisk/disk15
# mknod /dev/oracle/asm_data01 c 13 0x000012
# ll /dev/rdisk/disk15
crw-r-----   1 bin        sys         13 0x000012 Jul 22 22:40 /dev/rdisk/disk15
# ll /dev/oracle/asm_data01
crw-r--r--   1 root       sys         13 0x000012 Jul 31 00:34 /dev/oracle/asm_data01

# ll /dev/rdisk/disk24
crw-r-----   1 bin        sys         13 0x000013 Jul 22 22:44 /dev/rdisk/disk24
# mknod /dev/oracle/asm_data02 c 13 0x000013
# ll /dev/rdisk/disk24
crw-r-----   1 bin        sys         13 0x000013 Jul 22 22:44 /dev/rdisk/disk24
# ll /dev/oracle/asm_data02
crw-r--r--   1 root       sys         13 0x000013 Jul 31 00:37 /dev/oracle/asm_data02

# ll /dev/rdisk/disk33
crw-r-----   1 bin        sys         13 0x000014 Jul 22 22:45 /dev/rdisk/disk33
# mknod /dev/oracle/asm_data03 c 13 0x000014
# ll /dev/rdisk/disk33
crw-r-----   1 bin        sys         13 0x000014 Jul 22 22:45 /dev/rdisk/disk33
# ll /dev/oracle/asm_data03
crw-r--r--   1 root       sys         13 0x000014 Jul 31 00:38 /dev/oracle/asm_data03

FRA
# ll /dev/rdisk/disk42
crw-r-----   1 bin        sys         13 0x000015 Jul 22 23:03 /dev/rdisk/disk42
# mknod /dev/oracle/asm_fra01 c 13 0x000015
# ll /dev/rdisk/disk42
crw-r-----   1 bin        sys         13 0x000015 Jul 22 23:03 /dev/rdisk/disk42
# ll /dev/oracle/asm_fra01
crw-r--r--   1 root       sys         13 0x000015 Jul 31 00:40 /dev/oracle/asm_fra01

# ll /dev/rdisk/disk67
crw-r-----   1 bin        sys         13 0x000016 Jul 22 23:10 /dev/rdisk/disk67
# mknod /dev/oracle/asm_fra02 c 13 0x000016
# ll /dev/rdisk/disk67
crw-r-----   1 bin        sys         13 0x000016 Jul 22 23:10 /dev/rdisk/disk67
# ll /dev/oracle/asm_fra02
crw-r--r--   1 root       sys         13 0x000016 Jul 31 00:42 /dev/oracle/asm_fra02

# ll /dev/rdisk/disk68
crw-r-----   1 bin        sys         13 0x000017 Jul 22 23:10 /dev/rdisk/disk68
# mknod /dev/oracle/asm_fra03 c 13 0x000017
# ll /dev/rdisk/disk68
crw-r-----   1 bin        sys         13 0x000017 Jul 22 23:10 /dev/rdisk/disk68
# ll /dev/oracle/asm_fra03
crw-r--r--   1 root       sys         13 0x000017 Jul 31 00:43 /dev/oracle/asm_fra03

# ll /dev/rdisk/disk69
crw-r-----   1 bin        sys         13 0x000018 Jul 22 23:10 /dev/rdisk/disk69
# mknod /dev/oracle/asm_fra04 c 13 0x000018
# ll /dev/rdisk/disk69
crw-r-----   1 bin        sys         13 0x000018 Jul 22 23:10 /dev/rdisk/disk69
# ll /dev/oracle/asm_fra04
crw-r--r--   1 root       sys         13 0x000018 Jul 31 00:44 /dev/oracle/asm_fra04

OCR
# ll /dev/rdisk/disk134
crw-r-----   1 bin        sys         13 0x000019 Jul 22 23:13 /dev/rdisk/disk134
# mknod /dev/oracle/asm_ocr01 c 13 0x000019
# ll /dev/rdisk/disk134
crw-r-----   1 bin        sys         13 0x000019 Jul 22 23:13 /dev/rdisk/disk134
# ll /dev/oracle/asm_ocr01
crw-r--r--   1 root       sys         13 0x000019 Jul 31 00:48 /dev/oracle/asm_ocr01

# ll /dev/rdisk/disk135
crw-r-----   1 bin        sys         13 0x00001a Jul 22 23:13 /dev/rdisk/disk135
# mknod /dev/oracle/asm_ocr02 c 13 0x00001a
# ll /dev/rdisk/disk135
crw-r-----   1 bin        sys         13 0x00001a Jul 22 23:13 /dev/rdisk/disk135
# ll /dev/oracle/asm_ocr02
crw-r--r--   1 root       sys         13 0x00001a Jul 31 00:49 /dev/oracle/asm_ocr02

# ll /dev/rdisk/disk136
crw-r-----   1 bin        sys         13 0x00001b Jul 22 23:13 /dev/rdisk/disk136
# mknod /dev/oracle/asm_ocr03 c 13 0x00001b
# ll /dev/rdisk/disk136
crw-r-----   1 bin        sys         13 0x00001b Jul 22 23:13 /dev/rdisk/disk136
# ll /dev/oracle/asm_ocr03
crw-r--r--   1 root       sys         13 0x00001b Jul 31 00:51 /dev/oracle/asm_ocr03

VOTING:
# ll /dev/rdisk/disk137
crw-r-----   1 bin        sys         13 0x00001c Jul 22 23:13 /dev/rdisk/disk137
# mknod /dev/oracle/asm_voting01 c 13 0x00001c
# ll /dev/rdisk/disk137
crw-r-----   1 bin        sys         13 0x00001c Jul 22 23:13 /dev/rdisk/disk137
# ll /dev/oracle/asm_voting01
crw-r--r--   1 root       sys         13 0x00001c Jul 31 00:53 /dev/oracle/asm_voting01

# ll /dev/rdisk/disk138
crw-r-----   1 bin        sys         13 0x00001d Jul 22 23:13 /dev/rdisk/disk138
# mknod /dev/oracle/asm_voting02 c 13 0x00001d
# ll /dev/rdisk/disk138
crw-r-----   1 bin        sys         13 0x00001d Jul 22 23:13 /dev/rdisk/disk138
# ll /dev/oracle/asm_voting02
crw-r--r--   1 root       sys         13 0x00001d Jul 31 00:54 /dev/oracle/asm_voting02

# ll /dev/rdisk/disk139
crw-r-----   1 bin        sys         13 0x00001e Jul 22 23:13 /dev/rdisk/disk139
# mknod /dev/oracle/asm_voting03 c 13 0x00001e
# ll /dev/rdisk/disk139
crw-r-----   1 bin        sys         13 0x00001e Jul 22 23:13 /dev/rdisk/disk139
# ll /dev/oracle/asm_voting03
crw-r--r--   1 root       sys         13 0x00001e Jul 31 00:55 /dev/oracle/asm_voting03

REDO:
# ll /dev/rdisk/disk140
crw-r-----   1 bin        sys         13 0x00001f Jul 22 23:13 /dev/rdisk/disk140
# mknod /dev/oracle/asm_redo01 c 13 0x00001f
# ll /dev/rdisk/disk140
crw-r-----   1 bin        sys         13 0x00001f Jul 22 23:13 /dev/rdisk/disk140
# ll /dev/oracle/asm_redo01
crw-r--r--   1 root       sys         13 0x00001f Jul 31 00:58 /dev/oracle/asm_redo01

CONTROL:
# ll /dev/rdisk/disk141
crw-r-----   1 bin        sys         13 0x000020 Jul 22 23:13 /dev/rdisk/disk141
# mknod /dev/oracle/asm_control01 c 13 0x000020
# ll /dev/rdisk/disk141
crw-r-----   1 bin        sys         13 0x000020 Jul 22 23:13 /dev/rdisk/disk141
# ll /dev/oracle/asm_control01
crw-r--r--   1 root       sys         13 0x000020 Jul 31 00:59 /dev/oracle/asm_control01

g. set up disk permission
# chown -R grid:asmadmin /dev/oracle/*
# chmod -R 660 /dev/oracle/*

 dd if=/dev/zero of=/dev/oracle/asm_data01 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_data02 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_data03 bs=1024k count=1000

 dd if=/dev/zero of=/dev/oracle/asm_fra01 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_fra02 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_fra03 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_fra04 bs=1024k count=1000

 dd if=/dev/zero of=/dev/oracle/asm_ocr01 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_ocr02 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_ocr03 bs=1024k count=1000

 dd if=/dev/zero of=/dev/oracle/asm_voting01 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_voting02 bs=1024k count=1000
 dd if=/dev/zero of=/dev/oracle/asm_voting03 bs=1024k count=1000

 dd if=/dev/zero of=/dev/oracle/asm_redo01 bs=1024k count=1000

 dd if=/dev/zero of=/dev/oracle/asm_control01 bs=1024k count=1000

# ll /dev/oracle/
total 0
crw-rw----   1 grid       asmadmin    13 0x000020 Jul 31 00:59 asm_control01
crw-rw----   1 grid       asmadmin    13 0x000012 Jul 31 00:34 asm_data01
crw-rw----   1 grid       asmadmin    13 0x000013 Jul 31 00:37 asm_data02
crw-rw----   1 grid       asmadmin    13 0x000014 Jul 31 00:38 asm_data03
crw-rw----   1 grid       asmadmin    13 0x000015 Jul 31 00:40 asm_fra01
crw-rw----   1 grid       asmadmin    13 0x000016 Jul 31 00:42 asm_fra02
crw-rw----   1 grid       asmadmin    13 0x000017 Jul 31 00:43 asm_fra03
crw-rw----   1 grid       asmadmin    13 0x000018 Jul 31 00:44 asm_fra04
crw-rw----   1 grid       asmadmin    13 0x000019 Jul 31 00:48 asm_ocr01
crw-rw----   1 grid       asmadmin    13 0x00001a Jul 31 00:49 asm_ocr02
crw-rw----   1 grid       asmadmin    13 0x00001b Jul 31 00:51 asm_ocr03
crw-rw----   1 grid       asmadmin    13 0x00001f Jul 31 00:58 asm_redo01
crw-rw----   1 grid       asmadmin    13 0x00001c Jul 31 00:53 asm_voting01
crw-rw----   1 grid       asmadmin    13 0x00001d Jul 31 00:54 asm_voting02
crw-rw----   1 grid       asmadmin    13 0x00001e Jul 31 00:55 asm_voting03

1.15. Setting the Minor Number for Device Files
a. Log in as the "root" user.
b. Determine whether /dev/async exists. if the device does not exists, then use the following command to create it.
#/sbin/mknod /dev/async c 101 0x4

c. Alternatively, you can set the minor number value to 0x104 using the following command:
#/sbin/mknod /dev/async c 101 0x104

d. If /dev/async exists, then determine the current value of the minor number, as shown in the following example:
# ll  -l /dev/async

crw-rw-rw-   1 bin        bin        101 0x000000 Jun  8 02:25 /dev/async

e. If the existing minor number of the file is not 0x4 or 0x14, then change it to an expected value using one of the following commands:
# rm /dev/async
# mknod /dev/async c 101 0x104
# ll  -l /dev/async

crw-r--r--   1 root       sys        101 0x000104 Jul 31 02:01 /dev/async


<<<^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^>>>
1.16. Manual Password less SSH Connectivity to enable user equivalency between Nodes[Optional].
To Configure SSH user equivalency, you must create RSA and DSA keys on each  node and copy these keys from all the node members into an authorized key file on each node. Follow the below command to achive this task.

$ whoami
grid
$
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh

a. Generate the RSA and DSA keys on both Nodes[ dcpdb1 & dcpdb2].
$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Please be patient....   Key generation may take a few minutes

Enter file in which to save the key (/home/grid/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/grid/.ssh/id_rsa.

Your public key has been saved in /home/grid/.ssh/id_rsa.pub.

The key fingerprint is:

ee:ac:4d:f8:f0:9b:ae:62:19:42:c9:99:13:2e:7a:20 grid@dcpdb1

The key's randomart image is:

+--[ RSA 2048]----+

|                 |

|  .              |

| o =             |

|E O              |

|o+ .    S        |

|. o .  o         |

| . . oo o        |

|    +  O .       |

|   . .o+X.       |

+-----------------+


$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
57:4f:3d:10:9b:dc:d8:c7:41:2d:79:53:87:ae:bf:08 grid@dcpdb1
The key's randomart image is:
+--[ DSA 1024]----+
|             oo==|
|            . @+=|
|            .*.*=|
|           . o. o|
|        S .  ..  |
|         .  .    |
|          E  .   |
|           . ..  |
|            . .. |
+-----------------+
$



b. On dcpdb1:

Add keys to the authorized key file and then send the same file to every nodes.

$
$ touch ~/.ssh/authorized_keys
$ cd ~/.ssh
$
$
$ ssh dcpdb1 cat /home/grid/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host 'dcpdb1 (172.31.xx.xx)' can't be established.
ECDSA key fingerprint is 92:82:a2:45:aa:ea:db:06:1e:f2:9d:f9:d4:96:4a:db.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dcpdb1,172.31.xx.xx' (ECDSA) to the list of known hosts.
Password:
$ ssh dcpdb1 cat /home/grid/.ssh/id_dsa.pub >> authorized_keys
$  ssh dcpdb2  cat /home/grid/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host 'dcpdb2 (172.31.xx.xx)' can't be established.
ECDSA key fingerprint is 22:3a:21:1d:55:0c:1a:c2:b2:f0:0f:51:17:27:60:01.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dcpdb2,172.31.xx.xx' (ECDSA) to the list of known hosts.
Password:
$ ssh dcpdb2  cat /home/grid/.ssh/id_dsa.pub >> authorized_keys
Password:
$
$
$ scp authorized_keys dcpdb2:/home/grid/.ssh/
Password:
authorized_keys                                                                               100% 1988     1.9KB/s   1.9KB/s   00:00

c. On dcpdb2: 
$
$ ssh dcpdb1 cat /home/grid/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host 'dcpdb1 (172.31.xx.xx)' can't be established.
ECDSA key fingerprint is 92:82:a2:45:aa:ea:db:06:1e:f2:9d:f9:d4:96:4a:db.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dcpdb1,172.31.xx.xx' (ECDSA) to the list of known hosts.
$ ssh dcpdb1 cat /home/grid/.ssh/id_dsa.pub >> authorized_keys
$  ssh dcpdb2  cat /home/grid/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host 'dcpdb2 (172.31.xx.xx)' can't be established.
ECDSA key fingerprint is 22:3a:21:1d:55:0c:1a:c2:b2:f0:0f:51:17:27:60:01.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dcpdb2,172.31.xx.xx' (ECDSA) to the list of known hosts.
$ ssh dcpdb2  cat /home/grid/.ssh/id_dsa.pub >> authorized_keys
$
$
$ scp authorized_keys dcpdb1:/home/grid/.ssh/
Password:
authorized_keys                                                                               100% 1988     1.9KB/s   1.9KB/s   00:00

c. Change the permission of authorized_key file (on each node)
$chmod 600 ~/.ssh/authorized_keys

Now, try executing the date on remote node to make sure that grid is not asked for the password. Do the same on both nodes.
Node1:
$ ssh grid@dcpdb1 date
Mon Aug  3 05:28:45 GMT 2015
$ ssh grid@dcpdb2 date
Mon Aug  3 05:30:31 GMT 2015
Node2:
$ ssh grid@dcpdb1 date
Mon Aug  3 05:32:57 GMT 2015
$ ssh grid@dcpdb2 date
Mon Aug  3 05:33:42 GMT 2015
<<<~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>>>

1.17. Reboot ALL Nodes
After completing pre-installation tasks into ALL nodes then Reboot ALL nodes.


>>> Use full Command >>
# nslookup dcpdb-scan
# hostname
# cat /etc/hosts
# cat /etc/rc.config.d/netconf
# ifconfig lan0
# ioscan -kfNnC disk
# diskinfo /dev/rdisk/disk137
# machinfo |grep -i memory
# uname -a
# swlist -l bundle |grep QPK
# swlist -l patch |grep PH*
#swlist -l product |grep -i compiler
$ . ./.profile
$ echo $ORACLE_HOME
# /usr/local/bin/unzip p13390677_112040_HPUX-IA64_3of7.zip [Additionally need to install unzip s/w]
#swinstall -s /tmp/oraclesoft/gettext-0.19.5-ia64-11.31.depot  [To install patch]
#/usr/sbin/kctune aio_listio_max=512 [Modify kernel]


>> 2. Installation of oracle 11gR2 Grid Infrastructure (11.2.0.4.0)

>>>Run Installer Verify:
chown -R grid:oinstall /tmp/oraclesoft/grid/

echo > /tmp/oraclesoft/gird/test.txt
>>>Run as grid user:
./runcluvfy.sh stage -pre crsinst -n dcpdb1,dcpdb2 -r 11gr2 -verbose >> /u01/oraclesoft/grid/test.txt

a.  Login as "grid" user.
As the "grid" user (Grid Infrastructure software owner) start the installer by running "runInstaller".
b. Choose Skip Software Updates.

c. Select radio button 'Install and Configure Grid Infrastructure for a Cluster' and click Next> 
d. Select radio button 'Advanced Installation' and click Next >
e. Accept 'English' as language and click Next > 

f. Specify 'Cluster Name' & 'Scan Name', Unchecked Configure GNS and click Next >
g. Use the Edit and Add buttons to specify the node names and virtual IP address you configured previously in your /etc/hosts file.
h. Use the 'SSH Connectivity' button to configure/test the passwordless SSH Connectivity between nodes.
i. Type in the OS password for the user 'grid' and press 'setup' for SSH connectivity.

j. Passwordless SSH Connectivity Successful. 
***Problem face-1: we face problem to create passwordless SSH connectivity between nodes. after scrutinizing, we have found that below four packages were absent into system. you must make sure that below packages are present into system before proceeding to create passwordless SSH connectivity.      
gettext-0.19.5-ia64-11.31.depot
libiconv-1.14-ia64-11.31.depot
bash-4.3.033-ia64-11.31.depot
termcap-1.3.1-ia64-11.31.depot
Note: Manually you can also configure SSH connectivity between nodes.

k. Click on 'Interface type' next to the Interface you want to use for your cluster and select the correct values for 'Public', 'Private' and 'Do Not Use' when finished click 'Next>. 
l. Select radio button 'Automatic Storage Management (ASM)' click 'Next>. 
m. Select the 'DiskGroup Name' specify the 'Redundancy' and tick the disks you want to use, [if candidate disks are not shown then check the permission for /dev...folder] when done click ' Next >'
[if not candidate disk disks found, then change discovery path to /dev/oracle/*].
Note: Note the number of voting disk that will be created depend on the redundancy level you specify: EXTERNAL will create 1 voting disk, NORMAL will create 3 voting disks, HIGH will create 5 voting disks.
***Problem face-2: We face problem to get candidate disk for creating ASM disk groups. after scrutinizing we have found we had problem in mknode & permission on /dev/oracle. We have totally removed mknod & permission on dev/oracle path and freshly create mknod & permission. and then we got candidate disks.

n. Assign groups as below:
Oracle ASM Administrator(OSASM) Group: asmadmin
Oracle ASM DBA(OSDBA for ASM) Group: asmdba
Oracle ASM Operator(OSOPER for ASM) Group: asmoper
o. Specify the location for ORACLE_BASE and for the software location and click 'Next >
p. Specify the location for inventory directory and click 'Next >
q. Perform Prerequisite Check.

r. Prerequisite Check.
r-1.0. We have ignored this warning.
s. Summary
t. Install product
 >> Wait for the OUI to complete its task. At this point you have to run oraInstRoot.sh on all cluster nodes. Run on the first node alone then go for others nodes then press ok.
>> root.sh should be run on one node at a time. if root.sh fails, then resolve it and run again.

t. Install Completed
u. Cluster Start
>> 3. Installation of Oracle 11gR2 RDBMS Software (11.2.0.4.0)
a. Login as "oracle" user.
Changes into the directory where we staged the RDBMS software
b. Provide your email address, tick the check box and provide your oracle support password if you want to receive security updates from oracle  support and click ' Next >'

c. Skip software updates and click ' Next>'

d. Select the option 'Install Database Software Only' and click '  Next>'

e. Select the option 'Oracle Real Application Clusters database installation', and select all nodes. and click '  Next>'
Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between nodes. Type in the OS password for the oracle user and click setup.
f. To confirm English as selected language and click ' Next >'
g. Make sure radio button 'Enterprise Edition' is ticked &click ' Next>'

i. Specify path for oracle base(ORACLE_BASE) and location where we want to store the software (ORACLE_HOME). Click ' Next>'
j. Specify groups:
Database Administrator (OSDBA) Group: dba
Database Operator (OSOPER) Group: oper

k. Oracle Universal Installer performs prerequisite checks.
Check that the status of all check is 'Succeeded' and click ' Next>'

l. After Check Summary
m. Installation
Log in to a terminal window as root user and run the root.sh script on the first node. when finished do the same for all other nodes in cluster as well. when finished click 'Ok'. 
Note: root.sh should be run on one node at a time.

n. Click 'Close' to finish the installation of the RDBMS software. 

>> 4. ASMCA to Create Diskgroups
a.  Login as "grid" user

grid@dcpdb1 [/home/grid]$ cd $ORACLE_HOME/bin                           
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./asmca
b. Click 'Create' to create a new diskgroup. Then, type in a name for the diskgroup, select the redundancy we want to provide and mark the tick box for the disks you want to assign to the new diskgroup. we have used allocation unit size 4MB for DATA. Finally click 'Ok' button.

c. Click 'Create' to create diskgroup for the flash recovery area. then, type in a name for the diskgroup, select the redundancy we want to provide and mark the tick box for the disks you want to assign to the new diskgroup. we have used allocation unit size 4MB for FRA. Finally click 'Ok' button.


d. Finally, click on 'Exit'.

>> 5. Separate VOTING file from OCR diskgroup to VOTING diskgroup.
At the time of grid installation, voting disk was created on OCR diskgroup as I chosen OCR diskgroup on grid installation. Now, I will separate the voting from OCR diskgroup to VOTING diskgroup. below are the procedure to do this job:
>> Login as "grid" user.
grid@dcpdb1 [/home/grid]$ cd $ORACLE_HOME/bin
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3088
         Available space (kbytes) :     259032
         ID                       : 1258885786
         Device/File Name         :       +OCR
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   4610bd9667534f64bfe72c04a7f30bdd (/dev/oracle/asm_ocr01) [OCR]
 2. ONLINE   8c5ee368b6094f71bf4442d5f7a87dd0 (/dev/oracle/asm_ocr02) [OCR]
 3. ONLINE   5063be9ed5ef4fb1bfe16fbbdf9d4a2a (/dev/oracle/asm_ocr03) [OCR]
Located 3 voting disk(s).
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./crsctl replace votedisk +VOTING
Successful addition of voting disk 58507a1a88fc4ffcbf39fe7e58f15548.
Successful addition of voting disk 08272a459d914ff5bfc542ba1817c1c0.
Successful addition of voting disk 103e76316e414f64bfdb54b3d0fcdabc.
Successful deletion of voting disk 4610bd9667534f64bfe72c04a7f30bdd.
Successful deletion of voting disk 8c5ee368b6094f71bf4442d5f7a87dd0.
Successful deletion of voting disk 5063be9ed5ef4fb1bfe16fbbdf9d4a2a.
Successfully replaced voting disk group with +VOTING.
CRS-4266: Voting file(s) successfully replaced
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$  ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   58507a1a88fc4ffcbf39fe7e58f15548 (/dev/oracle/asm_voting01) [VOTING]
 2. ONLINE   08272a459d914ff5bfc542ba1817c1c0 (/dev/oracle/asm_voting02) [VOTING]
 3. ONLINE   103e76316e414f64bfdb54b3d0fcdabc (/dev/oracle/asm_voting03) [VOTING]
Located 3 voting disk(s).
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$

>> 6. Run NETCA to Create Listener for additional port (ALL RAC NODES)

a. Login as "grid" user
grid@dcpdb1 [/home/grid]$cd $ORACLE_HOME/bin 
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$netca
b. Net configuration Assistance: Listener Configuration, Listener
c. Net configuration Assistance: Listener Configuration, Listener Name
d. Net configuration Assistance: Listener Configuration, Select protocols 
e. Net configuration Assistance: Listener Configuration, Select port
f. Net configuration Assistance: Listener Configuration, More Listener?
g. Net configuration Assistance: Listener Configuration, done

>> 7. Run DBCA to create Database.
a. Login in as "oracle" user.
oracle@dcpdb1 [/home/oracle]$ cd $ORACLE_HOME/bin
oracle@dcpdb1 [/u02/app/oracle/product/11.2.0.4/dbhome_1/bin]$ dbca
> select 'Oracle Real Application Clusters database' and click 'Next >'
b. Choose option 'Create a Database' and click 'Next >'
c. Select the database template that you want to use for your database and click ' next >'
d. Type in the name you want to use for your database and select all nodes before you click ' Next>'

e. select the options you want use to manage your database and click 'Next >'
>> Enterprise Manager
>> Automatic Maintenance Tasks
f. Type the password you want to use and click 'Next >'

g. Select the diskgroup you created for the database files and Click on 'Multiplex Redo logs and control files.
>In the popup window define the diskgroup that should contain control files and redo log files and the diskgroup that should contain the mirrored files.
>Specify ASMSNMP password specific to ASM. When all are correct then click on Next > 
h. Specify the diskgroup that was created for the flash recovery area and define the size. if the size is smaller than recommended a warning will popup.
i. Select if you want to have sample schemas created in database and click 'Next>'
j. Review and change the settings for-
>>Memory Settings
>>Block Sizing & OS user process
>>Setting Character sets
>>Settings Connection Mode
>>  Setting All initialization Parameters

k. Database storage setting- 
>> Control files:
>> Data files:
>> Redo log groups:
Note: At first we have created 5 REDO LOG GROUPS# for Each THEAD#. Each REDO LOG GROUP consists two member and each member size 800MB. Later, we have created/increased another one REDO LOG GROUPS for each THREAD#. Finally, we had 6 REDO LOG GROUPS for each THREAD#, which consists two members and each member size 800MB.   

l. Make sure the tickbox 'Create Database' is ticked and click 'Finish'. 
n. Review the database configuration details again and click 'Ok'.

m. The database is now created, we can either change or unlock password or just click Exit to finish the database creation.

>> 8. Tablespace, Schema, Export & Import and Other Configuration.

8.1 Diskgroup Imbalance Check (Is your disk group balanced?)

Run the following query in your ASM instance to get the report on the disk group imbalance.
grid@dcpdb1 [/home/grid]$ sqlplus / as sysasm

SQL> 
column "Diskgroup" format A30 
column "Imbalance" format 99.9 Heading "Percent|Imbalance"
column "Variance" format 99.9 Heading "Percent|Disk Size|Variance"
column "MinFree" format 99.9 Heading "Minimum|Percent|Free"
column "DiskCnt" format 9999 Heading "Disk|Count"

column "Type" format A10 Heading "Diskgroup|Redundancy"

SQL> 

SELECT g.name "Diskgroup",100*(max((d.total_mb-d.free_mb)/d.total_mb)-min((d.total_mb-d.free_mb)/d.total_mb))/max((d.total_mb-d.free_mb)/d.total_mb) "Imbalance",100*(max(d.total_mb)-min(d.total_mb))/max(d.total_mb) "Variance",100*(min(d.free_mb/d.total_mb)) "MinFree",count(*) "DiskCnt",g.type "Type" FROM v$asm_disk d, v$asm_diskgroup g WHERE d.group_number = g.group_number and  d.group_number <> 0 and d.state = 'NORMAL' and d.mount_status = 'CACHED' GROUP BY g.name, g.type;


SQL>
select NAME, TOTAL_MB, FREE_MB from v$asm_disk;
select total_mb,free_mb from v$asm_diskgroup;
Select operation, state, est_work, est_minutes from v$asm_operation;
select group_number, name, total_mb, free_mb, type from v$asm_diskgroup where name='FRA'; 

SQL>
show parameter limit;
select * from v$asm_operation;
alter diskgroup FRA rebalance power 5;


8.2 Create Tablespace and add space to it under an OMF system: 

SQL> Show parameter DB_CREATE_FILE_DEST
SQL> show parameter db_create

Create Tablespace ASM.
SQL> CREATE TABLESPACE xxxDATA DATAFILE SIZE 30G;
SQL> CREATE TABLESPACE xxxxINDEX DATAFILE SIZE 30G;

SQL> select tablespace_name from dba_data_files;
Add Datafile to Tableapace:
SQL>ALTER TABLESPACE xxxxDATA ADD DATAFILE SIZE 30G;
SQL>ALTER TABLESPACE xxxxDATA ADD DATAFILE SIZE 30G;
SQL>ALTER TABLESPACE xxxxDATA ADD DATAFILE SIZE 30G;
SQL>ALTER TABLESPACE xxxxDATA ADD DATAFILE SIZE 30G;
SQL>ALTER TABLESPACE xxxxDATA ADD DATAFILE SIZE 30G;

8.3 Create Schema/User:
CREATE USER xxxxx IDENTIFIED BY *****
DEFAULT TABLESPACE xxxxDATA 
TEMPORARY TABLESPACE TEMP 
QUOTA UNLIMITED ON xxxxDATA 
QUOTA UNLIMITED ON xxxxINDEX;
GRANT CONNECT TO xxxxx;
GRANT DBA TO xxxxx;
GRANT RESOURCE TO xxxxx;
GRANT EXP_FULL_DATABASE TO xxxxx;
GRANT IMP_FULL_DATABASE TO xxxxx;

8.4 create datapump directory :

CREATE DIRECTORY dir as '/db_dump/soft/';

GRANT READ,WRITE ON DIRECTORY dir to xxxxx;




8.5 Export & Import:


Import
impdp xxxxx/xxxx directory=dir remap_tablespace=xxxxINDEX:xxxxxINDEX 


dumpfile=nblt24ac_010820152321%U.dmp parallel=4 cluster=N logfile=nblt24ac_010820152321_imp.log

Import Exclude a INDEX
impdp t24test/xxxx directory=dir1 dumpfile=nblt24ac_121120150241%U.dmp remap_schema=nblt24:t24test remap_tablespace=xxxxDATA:T24TESTDATA xxxxINDEX:T24TESTINDEX parallel=4 EXCLUDE=INDEX:\"IN \(\'IX_FBNK_STMT_ENTRY\'\)\" cluster=N logfile=t24test.log metrics=y

Export
expdp xxx/xxxxx DIRECTORY=dir dumpfile=nblt24ac_010820152321%U.dmp 
logfile=xxxx_010820152321.log schemas=xxx parallel=4 EXCLUDE=STATISTICS CLUSTER=N metrics=y


8.6 Add LOGFILE [Optional]

Node 1:

ALTER DATABASE ADD LOGFILE THREAD 1
     GROUP 6 ('+REDO','+CONTROL') SIZE 800M;
Node  2:
ALTER DATABASE ADD LOGFILE THREAD 2
     GROUP 12 ('+REDO','+CONTROL') SIZE 800M;

Node 1:

ALTER SYSTEM CHECKPOINT GLOBAL;
alter system switch logfile;
Node 2:
ALTER SYSTEM CHECKPOINT GLOBAL;
alter system switch logfile;



8.7 Listener [ALL RAC Nodes]  

>> Listener in "Grid" user
endpoints_listener.ora
LISTENERIT_DCPDB1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip)(PORT=1526))(ADDRESS=(PROTOCOL=TCP)(HOST=172.31.xx.xx)(PORT=1526)(IP=FIRST))))             # line added by Agent
LISTENERNBL_DCPDB1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip)(PORT=1525))(ADDRESS=(PROTOCOL=TCP)(HOST=172.31.xx.xx)(PORT=1525)(IP=FIRST))))            # line added by Agent

LISTENER_DCPDB1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=172.31.xx.xx)(PORT=1521)(IP=FIRST))))               # line added by Agent

listener.ora
LISTENERIT=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENERIT))))                # line added by Agent
LISTENERNBL=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENERNBL))))              # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))                # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))                # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENERNBL=ON           # line added by Agent

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENERIT=ON            # line added by Agent


Local Listener in Oracle [In oracle user]
alter system set local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1- vip.xxxx.com)(PORT=15xx))(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip.xxx.com)(PORT=19xx))))' scope=BOTH SID='dcpdb1';
alter system register;
alter system set local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb2- vip.xxx.com)(PORT=15xx))(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb2-vip.xxx.com)(PORT=19xx))))' scope=BOTH SID='dcpdb2';
alter system register;



8.8 Client tnsnames

For Scan
xxx =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb-scan.xxx.com)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = dcpdb.xxxx.com)
)
)

OR

For VIP
xxx =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDEASS = (PROTOCOL = TCP)(HOST = dcpdb1-vip.xxx.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb2-vip.xxx.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb1-vip.xxx.com)(PORT = 1525))
      (ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb2-vip.xxx.com)(PORT = 1525))
      (ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb1-vip.xxx.com)(PORT = 1526))
      (ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb2-vip.xxx.com)(PORT = 1526))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = dcpdb.xxx.com)
    )

  )


References:
http://docs.oracle.com/cd/E11882_01/install.112/e10815.pdf
http://www.oracledba.org/11gR2/Pre_Install_11gR2.htm
http://toddlerdba.blogspot.com/2013/01/oracle-11gr2-rac-on-hpux-itanium-1131.html
http://docs.oracle.com/cd/E11882_01/install.112/e49316/pre_install.htm
http://docs.oracle.com/cd/E11882_01/install.112/e48295/pre_hpux.htm
http://h18006.www1.hp.com/storage/pdfs/4AA2-7668ENW.pdf

No comments:

Post a Comment