Database Administrator's blog: Oracle 11g Release 2 (11.2.0.4.0) RAC Configuration on HP-UX Itanium (11.31) Server

This document show steps of installation and setting up 2-Node Oracle 11g Release 2 (11.2.0.4.0) RAC on HP-UX Itanium (11.31) server.

>> 2-Node RAC Initial
Operating system: HPUX Itanium 11.31
Oracle Version: 11.2.0.4.0
Server: ALL RAC Nodes + DNS Server

Grid Infrastructure software (Clusterware + ASM 11.2.0.4.0)
  GRID_HOME=/u01/app/11.2.0.4/grid
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=$GRID_HOME
  Owner: grid (Primary Group: oinstall, Secondary Group: dba,asmdba,asmadmin,asmoper)
Permission: 775
OCR/Voting Disk: ASM
Oracle Inventory Location: /u01/app/oraInventory

Oracle Database software (RAC 11.2.0.4.0)
  ORACLE_BASE=/u02/app/oracle
  ORACLE_HOME=/u02/app/oracle/product/11.2.0.4/dbhome_1
  Owner: oracle (Primary Group: oinstall, Secondary Group: dba,oper,asmadmin)
Permission: 775

>> Installation divided into Eight main parts:

1. Pre-Installation tasks.
2. Installation of oracle 11gR2 Grid Infrastructure (11.2.0.4.0).
3. Installation of Oracle 11gR2 RDBMS Software (11.2.0.4.0).
4. Run ASMCA to create diskgroups.
5. Separate VOTING from OCR diskgroups to VOTING diskgroup.
6. Run NETCA to create Listener [Additional Port].
7. Run DBCA to create database.
8. Tablespace, Schema, Export & Import and Others Configuration.

>> 1. Pre-Installation Tasks.
1.1. Configure NTP
[...Configure NTP...]
OR

/sbin/init.d/xntpd stop
mv /etc/ntp.conf /etc/ntp.conf.bak

1.2. Configure DNS & Verify DNS from RAC Nodes
>>/etc/resolv.conf (on RAC Nodes)
search nblitd.com
nameserver 172.31.xx.xx

>>/etc/resolv.conf (on RAC Nodes)
service named start
chkconfig --level 35 named on

1.3. Create OS Groups using the command below. Enter these Command as "root" user.
/usr/sbin/groupadd -g 700 oinstall
/usr/sbin/groupadd -g 701 dba
/usr/sbin/groupadd -g 702 oper
/usr/sbin/groupadd -g 703 asmdba
/usr/sbin/groupadd -g 704 asmoper
/usr/sbin/groupadd -g 705 asmadmin

1.4. Create users that will own Oracle & Gird software.Enter these Command as "root" user.
/usr/sbin/useradd -m -u 108 -g oinstall -G dba,oper,asmdba -d /home/oracle oracle
/usr/sbin/useradd -m -u 109 -g oinstall -G dba,asmdba,asmoper,asmadmin -d /home/grid grid

Note: -m is used to create user's home directory[i.e. /home/oracle, /home/grid] . if you do not use
-m in your useradd then you can get .profile by following:
cd /etc/skel
cp .cshrc .exrc .login .profile /home/oracle/
chown oracle:oinstall /home/oracle

cd /etc/skel
cp .cshrc .exrc .login .profile /home/grid/
chown oracle:oinstall /home/grid

>>read, write, execute permission to directory or profile

#chmod -R 777 /home/grid [On full directory]
or
>>>> Login as grid
$chmod -R 777 .profile [On grid profile]

#chmod -R 777 /home/oracle [On full directory]
or
>>>> Login as oracle
$chmod -R 777 .profile [On oracle profile]

# id oracle
uid=108(oracle) gid=700(oinstall) groups=701(dba),702(oper),703(asmdba)
# id grid
uid=109(grid) gid=700(oinstall) groups=701(dba),703(asmdba),704(asmoper),705(asmadmin)

1.5. Set the password for the Oracle & Grid account.Enter these Command as "root" user.

# passwd grid

Changing password for grid

New password:

Re-enter new password:

Passwd successfully changed

# passwd oracle

Changing password for oracle

New password:

Re-enter new password:

Passwd successfully changed

1.6. Sample /etc/hosts file will be as below.

127.0.0.1 localhost loopback

#################Public######################

172.31.xx.xx dcpdb1.xxx.com dcpdb1

172.31.xx.xx dcpdb2.xxx.com dcpdb2

#################Private#########################

10.x.x.1 dcpdb1-prv

10.x.x.2 dcpdb2-prv

################### VIP ##########################

172.31.xx.xx dcpdb1-vip.xxx.com dcpdb1-vip

172.31.xx.xx dcpdb2-vip.xxx.com dcpdb2-vip

Note: In SCAN IP is use in host file then only one IP will entry to host file.

1.7. Setting Kernel Parameters.

On ALL RAC Nodes, verify that kernel parameters value greater or equal to recommended values. kernel parameters value can be set manually (or can also be fixed during installation using runfixup.sh).

Note: On all RAC Nodes, if the current value of any parameter is higher than the value listed in this table, then do not change the value of that parameter.

>> Modifying Kernel Setting Using KCTUNE

Use /usr/sbin/kctune to view or change kernel parameters.

For example, to view kernel parameters, enter kctune with the -d flag. For example:

#/usr/sbin/kctune -d aio_listio_max

To modify a kernel parameter, enter kctune variable=value. For example:

#/usr/sbin/kctune aio_listio_max=512

Complete this procedure on ALL RAC nodes.

Soft & Hard Limit Setup:
/usr/sbin/kctune maxuprc=3686
/usr/sbin/kctune msgmni=4096
/usr/sbin/kctune msgtql=4096
/usr/sbin/kctune ncsize=35840
/usr/sbin/kctune maxfiles_lim=63488
/usr/sbin/kctune ninode=34816
/usr/sbin/kctune semmni=4096
/usr/sbin/kctune semmns=8192
/usr/sbin/kctune semmnu=4092
/usr/sbin/kctune shmseg=512
/usr/sbin/kctune -s maxfiles=16384
/usr/sbin/kctune -s maxfiles_lim=65536
/usr/sbin/kctune -s maxssiz=134217728
/usr/sbin/kctune -s shmmni=4096
/usr/sbin/kctune -s shmseg=512

1.8. Software(Package) requirement list for HPUX Itanium Platform[Install below package or upper versions].

Note: For Password less SSH Connectivity between nodes below Software packages must be installed on each & every node.

gettext-0.19.5-ia64-11.31.depot
libiconv-1.14-ia64-11.31.depot
bash-4.3.033-ia64-11.31.depot
termcap-1.3.1-ia64-11.31.depot

>> To Install patch execute following command:
swinstall -s /tmp/oraclesoft/gettext-0.19.5-ia64-11.31.depot
swinstall -s /tmp/oraclesoft/libiconv-1.14-ia64-11.31.depot
swinstall -s /tmp/oraclesoft/bash-4.3.033-ia64-11.31.depot
swinstall -s /tmp/oraclesoft/termcap-1.3.1-ia64-11.31.depot

>> To Check/View patchs execute following command:
swlist -l patch |grep PHCO*
swlist -l patch |grep PHSS*

>> Compiler Check
swlist -l product |grep -i compiler

1.9. Create Required Symbolic/Soft Links

ln -s /usr/bin/ssh /usr/local/bin/ssh

ln -s /usr/bin/scp /usr/local/bin/scp

Below symbolic links are required only if Motif 2.1 development Environment package (X11MotifDevKit.MOTIF21-PRG) is not installed.

Check [X11MotifDevKit.MOTIF21-PRG] is exists ?
#swlist -l product |grep -i X11Motif*

if not exists then-
#cd /usr/lib
ln -s libX11.3 libX11.sl
ln -s libXIE.2 libXIE.2.sl
ln -s libXext.3 libXext.sl
ln -s libXhp11.3 libXhp11.sl
ln -s libXi.3 libXi.sl
ln -s libXm.4 libXm.sl
ln -s libXp.2 libXp.sl
ln -s libXt.3 libXt.sl
ln -s libXtst.2 libXtst.sl

1.10. Create the Oracle Grid Infrastructure Home Directory [ALL Nodes]

# mkdir -p /u01/app/grid
# chmod -R 775 /u01/app/grid
# mkdir -p /u01/app/11.2.0.4/grid
# chmod -R 775 /u01/app/11.2.0.4/grid
# chown -R grid:oinstall /u01

$vi .profile [["grid" user for Node1]]
# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

##Set up Grid Environment:##
umask 022
export GRID_HOME=/u01/app/11.2.0.4/grid
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=$GRID_HOME
export ORACLE_SID=+ASM1
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

$ vi .profile [["grid" user for Node2]]

# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

## Set up Grid Environment###
umask 022
export GRID_HOME=/u01/app/11.2.0.4/grid
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=$GRID_HOME
export ORACLE_SID=+ASM2
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

1.11. Create the Oracle Inventory Directory [ALL Nodes]

# mkdir -p /u01/app/oraInventory
# chown -R grid:oinstall /u01/app/oraInventory
# chmod -R 775 /u01/app/oraInventory

1.12. Create the Oracle BASE Directory [ALL Nodes]
# mkdir -p /u02/app/oracle/product/11.2.0.4/dbhome_1
# chown -R oracle:oinstall /u02
# chmod -R 775 /u02/app/oracle/product/11.2.0.4/dbhome_1

$vi .profile [["oracle" user for Node1]]
# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

## Set up Oracle Environment##
umask 022
export ORACLE_BASE=/u02/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.4/dbhome_1
export ORACLE_SID=dcpdb1
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

$vi .profile [["oracle" user for Node2]]
# Set up the shell environment
set -u
trap "echo 'logout''' 0
stty erase ^?
set -o vi

# Set up the shell variables:
EDITOR=vi
export EDITOR
export HISTFILE=$HOME/.sh_history
export PS1="$(/usr/bin/whoami)@$(/usr/bin/hostname) [\$PWD]$"

##Set up Oracle Environment####
umask 022
export ORACLE_BASE=/u02/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.4/dbhome_1
export ORACLE_SID=dcpdb2
export PATH=$ORACLE_HOME/bin:$PATH
TZ=GMT6
export TZ

1.13. ASM Disk Group Sizing

DATA: 600G >>> LUN:15,24,33
FRA: 800G >>> LUN: 42,67,68,69
OCR: 6G >>> LUN: 134,135,136
VOTE: 6G >>> LUN: 137,138,139
REDO: 40G >>> LUN:140
CONTROL: 35G >>> 141

1.14. Configure Disk Devices for Oracle ASM
a. To make sure that the disks are available, enter the following command on ALL node:
# ioscan -fNnC disk
Class I H/W Path Driver S/W State H/W Type Description
===================================================================
disk 3 64000/0xfa00/0x1 esdisk CLAIMED DEVICE HP LOGICAL VOLUME
/dev/disk/disk3 /dev/disk/disk3_p2 /dev/rdisk/disk3 /dev/rdisk/disk3_p2
/dev/disk/disk3_p1 /dev/disk/disk3_p3 /dev/rdisk/disk3_p1 /dev/rdisk/disk3_p3
disk 4 64000/0xfa00/0x2 esdisk CLAIMED DEVICE HP LOGICAL VOLUME
/dev/disk/disk4 /dev/rdisk/disk4
disk 5 64000/0xfa00/0x4 esdisk CLAIMED DEVICE hp DVDRAM GTA0N
/dev/disk/disk5 /dev/rdisk/disk5
disk 15 64000/0xfa00/0x12 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk15 /dev/rdisk/disk15
disk 24 64000/0xfa00/0x13 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk24 /dev/rdisk/disk24
disk 33 64000/0xfa00/0x14 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk33 /dev/rdisk/disk33
disk 42 64000/0xfa00/0x15 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk42 /dev/rdisk/disk42
disk 67 64000/0xfa00/0x16 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk67 /dev/rdisk/disk67
disk 68 64000/0xfa00/0x17 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk68 /dev/rdisk/disk68
disk 69 64000/0xfa00/0x18 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk69 /dev/rdisk/disk69
disk 134 64000/0xfa00/0x19 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk134 /dev/rdisk/disk134
disk 135 64000/0xfa00/0x1a esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk135 /dev/rdisk/disk135
disk 136 64000/0xfa00/0x1b esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk136 /dev/rdisk/disk136
disk 137 64000/0xfa00/0x1c esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk137 /dev/rdisk/disk137
disk 138 64000/0xfa00/0x1d esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk138 /dev/rdisk/disk138
disk 139 64000/0xfa00/0x1e esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk139 /dev/rdisk/disk139
disk 140 64000/0xfa00/0x1f esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk140 /dev/rdisk/disk140
disk 141 64000/0xfa00/0x20 esdisk CLAIMED DEVICE 3PARdataVV
/dev/disk/disk141 /dev/rdisk/disk141

b. if the ioscan command does not display device name information for a device, enter the following command to install the special device files for any new devices:
#/usr/sbin/insf -e

c. For each disk to add to a disk group, enter the following command on any node to verify that it is not already part of an LVM volume group.
#/sbin/pvdisplay /dev/disk/disk140
if this command display volume group information, the disk is already part of a volume group. The disks that you choose must not be part of an LVM volume group.

d. The disks names for the same disk can be different on each node. A disk can be identified as the same one via the WWID. The WWID of a disk can be checked via the following command:
#scsimgr lun_map -D /dev/rdisk/disk15 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000800010670
scsimgr lun_map -D /dev/rdisk/disk24 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000900010670
scsimgr lun_map -D /dev/rdisk/disk33 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000a00010670
scsimgr lun_map -D /dev/rdisk/disk42 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000b00010670
scsimgr lun_map -D /dev/rdisk/disk67 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000c00010670

scsimgr lun_map -D /dev/rdisk/disk68 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000d00010670
scsimgr lun_map -D /dev/rdisk/disk69 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000e00010670

scsimgr lun_map -D /dev/rdisk/disk134 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000000f00010670

scsimgr lun_map -D /dev/rdisk/disk135 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000001000010670

scsimgr lun_map -D /dev/rdisk/disk136 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000001100010670
scsimgr lun_map -D /dev/rdisk/disk137 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000001200010670

scsimgr lun_map -D /dev/rdisk/disk138 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000001300010670
scsimgr lun_map -D /dev/rdisk/disk139 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000001400010670
scsimgr lun_map -D /dev/rdisk/disk140 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000001600010670
scsimgr lun_map -D /dev/rdisk/disk141 | grep WWID
World Wide Identifier(WWID) = 0x60002ac0000000000000001700010670

e. Disk Size Check
#diskinfo /dev/rdisk/disk15
SCSI describe of /dev/rdisk/disk15:
vendor: 3PARdata
product id: VV
type: direct access
size: 209715200 Kbytes 200 GB
bytes per sector: 512

#diskinfo /dev/rdisk/disk24
SCSI describe of /dev/rdisk/disk24:
vendor: 3PARdata
product id: VV
type: direct access
size: 209715200 Kbytes Size 200GB
bytes per sector: 512

#diskinfo /dev/rdisk/disk33
SCSI describe of /dev/rdisk/disk33:
vendor: 3PARdata
product id: VV
type: direct access
size: 209715200 Kbytes Size 200GB
bytes per sector: 512

#diskinfo /dev/rdisk/disk42
SCSI describe of /dev/rdisk/disk42:
vendor: 3PARdata
product id: VV
type: direct access
size: 209715200 Kbytes Size 200GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk67
SCSI describe of /dev/rdisk/disk67:
vendor: 3PARdata
product id: VV
type: direct access
size: 209715200 Kbytes Size 200GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk68
SCSI describe of /dev/rdisk/disk68:
vendor: 3PARdata
product id: VV
type: direct access
size: 209715200 Kbytes Size 200GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk69
SCSI describe of /dev/rdisk/disk69:
vendor: 3PARdata
product id: VV
type: direct access
size: 209715200 Kbytes Size 200GB
bytes per sector: 512

# diskinfo /dev/rdisk/disk134
SCSI describe of /dev/rdisk/disk134:
vendor: 3PARdata
product id: VV
type: direct access
size: 2097152 Kbytes 2GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk135
SCSI describe of /dev/rdisk/disk135:
vendor: 3PARdata
product id: VV
type: direct access
size: 2097152 Kbytes 2GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk136
SCSI describe of /dev/rdisk/disk136:
vendor: 3PARdata
product id: VV
type: direct access
size: 2097152 Kbytes 2GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk137
SCSI describe of /dev/rdisk/disk137:
vendor: 3PARdata
product id: VV
type: direct access
size: 2097152 Kbytes 2GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk138
SCSI describe of /dev/rdisk/disk138:
vendor: 3PARdata
product id: VV
type: direct access
size: 2097152 Kbytes 2GB
bytes per sector: 512

# diskinfo /dev/rdisk/disk139
SCSI describe of /dev/rdisk/disk139:
vendor: 3PARdata
product id: VV
type: direct access
size: 2097152 Kbytes 2GB
bytes per sector: 512

# diskinfo /dev/rdisk/disk140
SCSI describe of /dev/rdisk/disk140:
vendor: 3PARdata
product id: VV
type: direct access
size: 41943040 Kbytes 40GB
bytes per sector: 512
# diskinfo /dev/rdisk/disk141
SCSI describe of /dev/rdisk/disk141:
vendor: 3PARdata
product id: VV
type: direct access
size: 36700160 Kbytes 35GB
bytes per sector: 512

F. Create a special oracle device directory and use mknod to create device paths in oracle folder.
#mkdir /dev/oracle
DATA
ll /dev/rdisk/disk15
crw-r----- 1 bin sys 13 0x000012 Jul 22 22:40 /dev/rdisk/disk15
# mknod /dev/oracle/asm_data01 c 13 0x000012
# ll /dev/rdisk/disk15
crw-r----- 1 bin sys 13 0x000012 Jul 22 22:40 /dev/rdisk/disk15
# ll /dev/oracle/asm_data01
crw-r--r-- 1 root sys 13 0x000012 Jul 31 00:34 /dev/oracle/asm_data01

# ll /dev/rdisk/disk24
crw-r----- 1 bin sys 13 0x000013 Jul 22 22:44 /dev/rdisk/disk24
# mknod /dev/oracle/asm_data02 c 13 0x000013
# ll /dev/rdisk/disk24
crw-r----- 1 bin sys 13 0x000013 Jul 22 22:44 /dev/rdisk/disk24
# ll /dev/oracle/asm_data02
crw-r--r-- 1 root sys 13 0x000013 Jul 31 00:37 /dev/oracle/asm_data02

# ll /dev/rdisk/disk33
crw-r----- 1 bin sys 13 0x000014 Jul 22 22:45 /dev/rdisk/disk33
# mknod /dev/oracle/asm_data03 c 13 0x000014
# ll /dev/rdisk/disk33
crw-r----- 1 bin sys 13 0x000014 Jul 22 22:45 /dev/rdisk/disk33
# ll /dev/oracle/asm_data03
crw-r--r-- 1 root sys 13 0x000014 Jul 31 00:38 /dev/oracle/asm_data03

FRA
# ll /dev/rdisk/disk42
crw-r----- 1 bin sys 13 0x000015 Jul 22 23:03 /dev/rdisk/disk42
# mknod /dev/oracle/asm_fra01 c 13 0x000015
# ll /dev/rdisk/disk42
crw-r----- 1 bin sys 13 0x000015 Jul 22 23:03 /dev/rdisk/disk42
# ll /dev/oracle/asm_fra01
crw-r--r-- 1 root sys 13 0x000015 Jul 31 00:40 /dev/oracle/asm_fra01

# ll /dev/rdisk/disk67
crw-r----- 1 bin sys 13 0x000016 Jul 22 23:10 /dev/rdisk/disk67
# mknod /dev/oracle/asm_fra02 c 13 0x000016
# ll /dev/rdisk/disk67
crw-r----- 1 bin sys 13 0x000016 Jul 22 23:10 /dev/rdisk/disk67
# ll /dev/oracle/asm_fra02
crw-r--r-- 1 root sys 13 0x000016 Jul 31 00:42 /dev/oracle/asm_fra02

# ll /dev/rdisk/disk68
crw-r----- 1 bin sys 13 0x000017 Jul 22 23:10 /dev/rdisk/disk68
# mknod /dev/oracle/asm_fra03 c 13 0x000017
# ll /dev/rdisk/disk68
crw-r----- 1 bin sys 13 0x000017 Jul 22 23:10 /dev/rdisk/disk68
# ll /dev/oracle/asm_fra03
crw-r--r-- 1 root sys 13 0x000017 Jul 31 00:43 /dev/oracle/asm_fra03

# ll /dev/rdisk/disk69
crw-r----- 1 bin sys 13 0x000018 Jul 22 23:10 /dev/rdisk/disk69
# mknod /dev/oracle/asm_fra04 c 13 0x000018
# ll /dev/rdisk/disk69
crw-r----- 1 bin sys 13 0x000018 Jul 22 23:10 /dev/rdisk/disk69
# ll /dev/oracle/asm_fra04
crw-r--r-- 1 root sys 13 0x000018 Jul 31 00:44 /dev/oracle/asm_fra04

OCR
# ll /dev/rdisk/disk134
crw-r----- 1 bin sys 13 0x000019 Jul 22 23:13 /dev/rdisk/disk134
# mknod /dev/oracle/asm_ocr01 c 13 0x000019
# ll /dev/rdisk/disk134
crw-r----- 1 bin sys 13 0x000019 Jul 22 23:13 /dev/rdisk/disk134
# ll /dev/oracle/asm_ocr01
crw-r--r-- 1 root sys 13 0x000019 Jul 31 00:48 /dev/oracle/asm_ocr01

# ll /dev/rdisk/disk135
crw-r----- 1 bin sys 13 0x00001a Jul 22 23:13 /dev/rdisk/disk135
# mknod /dev/oracle/asm_ocr02 c 13 0x00001a
# ll /dev/rdisk/disk135
crw-r----- 1 bin sys 13 0x00001a Jul 22 23:13 /dev/rdisk/disk135
# ll /dev/oracle/asm_ocr02
crw-r--r-- 1 root sys 13 0x00001a Jul 31 00:49 /dev/oracle/asm_ocr02

# ll /dev/rdisk/disk136
crw-r----- 1 bin sys 13 0x00001b Jul 22 23:13 /dev/rdisk/disk136
# mknod /dev/oracle/asm_ocr03 c 13 0x00001b
# ll /dev/rdisk/disk136
crw-r----- 1 bin sys 13 0x00001b Jul 22 23:13 /dev/rdisk/disk136
# ll /dev/oracle/asm_ocr03
crw-r--r-- 1 root sys 13 0x00001b Jul 31 00:51 /dev/oracle/asm_ocr03

VOTING:
# ll /dev/rdisk/disk137
crw-r----- 1 bin sys 13 0x00001c Jul 22 23:13 /dev/rdisk/disk137
# mknod /dev/oracle/asm_voting01 c 13 0x00001c
# ll /dev/rdisk/disk137
crw-r----- 1 bin sys 13 0x00001c Jul 22 23:13 /dev/rdisk/disk137
# ll /dev/oracle/asm_voting01
crw-r--r-- 1 root sys 13 0x00001c Jul 31 00:53 /dev/oracle/asm_voting01

# ll /dev/rdisk/disk138
crw-r----- 1 bin sys 13 0x00001d Jul 22 23:13 /dev/rdisk/disk138
# mknod /dev/oracle/asm_voting02 c 13 0x00001d
# ll /dev/rdisk/disk138
crw-r----- 1 bin sys 13 0x00001d Jul 22 23:13 /dev/rdisk/disk138
# ll /dev/oracle/asm_voting02
crw-r--r-- 1 root sys 13 0x00001d Jul 31 00:54 /dev/oracle/asm_voting02

# ll /dev/rdisk/disk139
crw-r----- 1 bin sys 13 0x00001e Jul 22 23:13 /dev/rdisk/disk139
# mknod /dev/oracle/asm_voting03 c 13 0x00001e
# ll /dev/rdisk/disk139
crw-r----- 1 bin sys 13 0x00001e Jul 22 23:13 /dev/rdisk/disk139
# ll /dev/oracle/asm_voting03
crw-r--r-- 1 root sys 13 0x00001e Jul 31 00:55 /dev/oracle/asm_voting03

REDO:
# ll /dev/rdisk/disk140
crw-r----- 1 bin sys 13 0x00001f Jul 22 23:13 /dev/rdisk/disk140
# mknod /dev/oracle/asm_redo01 c 13 0x00001f
# ll /dev/rdisk/disk140
crw-r----- 1 bin sys 13 0x00001f Jul 22 23:13 /dev/rdisk/disk140
# ll /dev/oracle/asm_redo01
crw-r--r-- 1 root sys 13 0x00001f Jul 31 00:58 /dev/oracle/asm_redo01

CONTROL:
# ll /dev/rdisk/disk141
crw-r----- 1 bin sys 13 0x000020 Jul 22 23:13 /dev/rdisk/disk141
# mknod /dev/oracle/asm_control01 c 13 0x000020
# ll /dev/rdisk/disk141
crw-r----- 1 bin sys 13 0x000020 Jul 22 23:13 /dev/rdisk/disk141
# ll /dev/oracle/asm_control01
crw-r--r-- 1 root sys 13 0x000020 Jul 31 00:59 /dev/oracle/asm_control01

g. set up disk permission
# chown -R grid:asmadmin /dev/oracle/*
# chmod -R 660 /dev/oracle/*

dd if=/dev/zero of=/dev/oracle/asm_data01 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_data02 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_data03 bs=1024k count=1000

dd if=/dev/zero of=/dev/oracle/asm_fra01 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_fra02 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_fra03 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_fra04 bs=1024k count=1000

dd if=/dev/zero of=/dev/oracle/asm_ocr01 bs=1024k count=1000

dd if=/dev/zero of=/dev/oracle/asm_ocr02 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_ocr03 bs=1024k count=1000

dd if=/dev/zero of=/dev/oracle/asm_voting01 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_voting02 bs=1024k count=1000
dd if=/dev/zero of=/dev/oracle/asm_voting03 bs=1024k count=1000

dd if=/dev/zero of=/dev/oracle/asm_redo01 bs=1024k count=1000

dd if=/dev/zero of=/dev/oracle/asm_control01 bs=1024k count=1000

# ll /dev/oracle/

total 0
crw-rw---- 1 grid asmadmin 13 0x000020 Jul 31 00:59 asm_control01

crw-rw---- 1 grid asmadmin 13 0x000012 Jul 31 00:34 asm_data01

crw-rw---- 1 grid asmadmin 13 0x000013 Jul 31 00:37 asm_data02

crw-rw---- 1 grid asmadmin 13 0x000014 Jul 31 00:38 asm_data03

crw-rw---- 1 grid asmadmin 13 0x000015 Jul 31 00:40 asm_fra01

crw-rw---- 1 grid asmadmin 13 0x000016 Jul 31 00:42 asm_fra02

crw-rw---- 1 grid asmadmin 13 0x000017 Jul 31 00:43 asm_fra03

crw-rw---- 1 grid asmadmin 13 0x000018 Jul 31 00:44 asm_fra04

crw-rw---- 1 grid asmadmin 13 0x000019 Jul 31 00:48 asm_ocr01

crw-rw---- 1 grid asmadmin 13 0x00001a Jul 31 00:49 asm_ocr02

crw-rw---- 1 grid asmadmin 13 0x00001b Jul 31 00:51 asm_ocr03

crw-rw---- 1 grid asmadmin 13 0x00001f Jul 31 00:58 asm_redo01

crw-rw---- 1 grid asmadmin 13 0x00001c Jul 31 00:53 asm_voting01

crw-rw---- 1 grid asmadmin 13 0x00001d Jul 31 00:54 asm_voting02

crw-rw---- 1 grid asmadmin 13 0x00001e Jul 31 00:55 asm_voting03

1.15. Setting the Minor Number for Device Files
a. Log in as the "root" user.
b. Determine whether /dev/async exists. if the device does not exists, then use the following command to create it.
#/sbin/mknod /dev/async c 101 0x4

c. Alternatively, you can set the minor number value to 0x104 using the following command:
#/sbin/mknod /dev/async c 101 0x104

d. If /dev/async exists, then determine the current value of the minor number, as shown in the following example:
# ll -l /dev/async

crw-rw-rw- 1 bin bin 101 0x000000 Jun 8 02:25 /dev/async

e. If the existing minor number of the file is not 0x4 or 0x14, then change it to an expected value using one of the following commands:
# rm /dev/async
# mknod /dev/async c 101 0x104
# ll -l /dev/async

crw-r--r-- 1 root sys 101 0x000104 Jul 31 02:01 /dev/async

<<<^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^>>>
1.16. Manual Password less SSH Connectivity to enable user equivalency between Nodes[Optional].

To Configure SSH user equivalency, you must create RSA and DSA keys on each node and copy these keys from all the node members into an authorized key file on each node. Follow the below command to achive this task.

$ whoami

grid

$ mkdir ~/.ssh

$ chmod 700 ~/.ssh

a. Generate the RSA and DSA keys on both Nodes[ dcpdb1 & dcpdb2].

$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Please be patient.... Key generation may take a few minutes

Enter file in which to save the key (/home/grid/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/grid/.ssh/id_rsa.

Your public key has been saved in /home/grid/.ssh/id_rsa.pub.

The key fingerprint is:

ee:ac:4d:f8:f0:9b:ae:62:19:42:c9:99:13:2e:7a:20 grid@dcpdb1

The key's randomart image is:

+--[ RSA 2048]----+

| |

| . |

| o = |

|E O |

|o+ . S |

|. o . o |

| . . oo o |

| + O . |

| . .o+X. |

+-----------------+

$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/grid/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/grid/.ssh/id_dsa.

Your public key has been saved in /home/grid/.ssh/id_dsa.pub.

The key fingerprint is:

57:4f:3d:10:9b:dc:d8:c7:41:2d:79:53:87:ae:bf:08 grid@dcpdb1

The key's randomart image is:

+--[ DSA 1024]----+

| oo==|

| . @+=|

| .*.*=|

| . o. o|

| S . .. |

| . . |

| E . |

| . .. |

+-----------------+

b. On dcpdb1:

Add keys to the authorized key file and then send the same file to every nodes.

$ touch ~/.ssh/authorized_keys

$ cd ~/.ssh

$ ssh dcpdb1 cat /home/grid/.ssh/id_rsa.pub >> authorized_keys

The authenticity of host 'dcpdb1 (172.31.xx.xx)' can't be established.

ECDSA key fingerprint is 92:82:a2:45:aa:ea:db:06:1e:f2:9d:f9:d4:96:4a:db.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'dcpdb1,172.31.xx.xx' (ECDSA) to the list of known hosts.

Password:

$ ssh dcpdb1 cat /home/grid/.ssh/id_dsa.pub >> authorized_keys

$ ssh dcpdb2 cat /home/grid/.ssh/id_rsa.pub >> authorized_keys

The authenticity of host 'dcpdb2 (172.31.xx.xx)' can't be established.

ECDSA key fingerprint is 22:3a:21:1d:55:0c:1a:c2:b2:f0:0f:51:17:27:60:01.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'dcpdb2,172.31.xx.xx' (ECDSA) to the list of known hosts.

Password:

$ ssh dcpdb2 cat /home/grid/.ssh/id_dsa.pub >> authorized_keys

Password:

$ scp authorized_keys dcpdb2:/home/grid/.ssh/

Password:

authorized_keys 100% 1988 1.9KB/s 1.9KB/s 00:00

c. On dcpdb2:

$ ssh dcpdb1 cat /home/grid/.ssh/id_rsa.pub >> authorized_keys

The authenticity of host 'dcpdb1 (172.31.xx.xx)' can't be established.

ECDSA key fingerprint is 92:82:a2:45:aa:ea:db:06:1e:f2:9d:f9:d4:96:4a:db.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'dcpdb1,172.31.xx.xx' (ECDSA) to the list of known hosts.

$ ssh dcpdb1 cat /home/grid/.ssh/id_dsa.pub >> authorized_keys

$ ssh dcpdb2 cat /home/grid/.ssh/id_rsa.pub >> authorized_keys

The authenticity of host 'dcpdb2 (172.31.xx.xx)' can't be established.

ECDSA key fingerprint is 22:3a:21:1d:55:0c:1a:c2:b2:f0:0f:51:17:27:60:01.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'dcpdb2,172.31.xx.xx' (ECDSA) to the list of known hosts.

$ ssh dcpdb2 cat /home/grid/.ssh/id_dsa.pub >> authorized_keys

$ scp authorized_keys dcpdb1:/home/grid/.ssh/

Password:

authorized_keys 100% 1988 1.9KB/s 1.9KB/s 00:00

c. Change the permission of authorized_key file (on each node)

$chmod 600 ~/.ssh/authorized_keys

Now, try executing the date on remote node to make sure that grid is not asked for the password. Do the same on both nodes.

Node1:

$ ssh grid@dcpdb1 date

Mon Aug 3 05:28:45 GMT 2015

$ ssh grid@dcpdb2 date

Mon Aug 3 05:30:31 GMT 2015

Node2:

$ ssh grid@dcpdb1 date

Mon Aug 3 05:32:57 GMT 2015

$ ssh grid@dcpdb2 date

Mon Aug 3 05:33:42 GMT 2015

<<<~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>>>

1.17. Reboot ALL Nodes

After completing pre-installation tasks into ALL nodes then Reboot ALL nodes.

>>> Use full Command >>

# nslookup dcpdb-scan

# hostname

# cat /etc/hosts

# cat /etc/rc.config.d/netconf

# ifconfig lan0

# ioscan -kfNnC disk

# diskinfo /dev/rdisk/disk137

# machinfo |grep -i memory

# uname -a

# swlist -l bundle |grep QPK

# swlist -l patch |grep PH*
#swlist -l product |grep -i compiler

$ . ./.profile

$ echo $ORACLE_HOME

# /usr/local/bin/unzip p13390677_112040_HPUX-IA64_3of7.zip [Additionally need to install unzip s/w]
#swinstall -s /tmp/oraclesoft/gettext-0.19.5-ia64-11.31.depot [To install patch]
#/usr/sbin/kctune aio_listio_max=512 [Modify kernel]

>> 2. Installation of oracle 11gR2 Grid Infrastructure (11.2.0.4.0)

>>>Run Installer Verify:
chown -R grid:oinstall /tmp/oraclesoft/grid/

echo > /tmp/oraclesoft/gird/test.txt
>>>Run as grid user:
./runcluvfy.sh stage -pre crsinst -n dcpdb1,dcpdb2 -r 11gr2 -verbose >> /u01/oraclesoft/grid/test.txt

a. Login as "grid" user.
As the "grid" user (Grid Infrastructure software owner) start the installer by running "runInstaller".

b. Choose Skip Software Updates.

c. Select radio button 'Install and Configure Grid Infrastructure for a Cluster' and click Next>

d. Select radio button 'Advanced Installation' and click Next >

e. Accept 'English' as language and click Next >

f. Specify 'Cluster Name' & 'Scan Name', Unchecked Configure GNS and click Next >

g. Use the Edit and Add buttons to specify the node names and virtual IP address you configured previously in your /etc/hosts file.

h. Use the 'SSH Connectivity' button to configure/test the passwordless SSH Connectivity between nodes.

i. Type in the OS password for the user 'grid' and press 'setup' for SSH connectivity.

j. Passwordless SSH Connectivity Successful.

***Problem face-1: we face problem to create passwordless SSH connectivity between nodes. after scrutinizing, we have found that below four packages were absent into system. you must make sure that below packages are present into system before proceeding to create passwordless SSH connectivity.

gettext-0.19.5-ia64-11.31.depot

libiconv-1.14-ia64-11.31.depot

bash-4.3.033-ia64-11.31.depot

termcap-1.3.1-ia64-11.31.depot

Note: Manually you can also configure SSH connectivity between nodes.

k. Click on 'Interface type' next to the Interface you want to use for your cluster and select the correct values for 'Public', 'Private' and 'Do Not Use' when finished click 'Next>.

l. Select radio button 'Automatic Storage Management (ASM)' click 'Next>.

m. Select the 'DiskGroup Name' specify the 'Redundancy' and tick the disks you want to use, [if candidate disks are not shown then check the permission for /dev...folder] when done click ' Next >'

[if not candidate disk disks found, then change discovery path to /dev/oracle/*].

Note: Note the number of voting disk that will be created depend on the redundancy level you specify: EXTERNAL will create 1 voting disk, NORMAL will create 3 voting disks, HIGH will create 5 voting disks.

***Problem face-2: We face problem to get candidate disk for creating ASM disk groups. after scrutinizing we have found we had problem in mknode & permission on /dev/oracle. We have totally removed mknod & permission on dev/oracle path and freshly create mknod & permission. and then we got candidate disks.

n. Assign groups as below:

Oracle ASM Administrator(OSASM) Group: asmadmin

Oracle ASM DBA(OSDBA for ASM) Group: asmdba

Oracle ASM Operator(OSOPER for ASM) Group: asmoper

o. Specify the location for ORACLE_BASE and for the software location and click 'Next >

p. Specify the location for inventory directory and click 'Next >

q. Perform Prerequisite Check.

r. Prerequisite Check.

r-1.0. We have ignored this warning.

s. Summary

t. Install product

>> Wait for the OUI to complete its task. At this point you have to run oraInstRoot.sh on all cluster nodes. Run on the first node alone then go for others nodes then press ok.

>> root.sh should be run on one node at a time. if root.sh fails, then resolve it and run again.

t. Install Completed

u. Cluster Start

>> 3. Installation of Oracle 11gR2 RDBMS Software (11.2.0.4.0)

a. Login as "oracle" user.
Changes into the directory where we staged the RDBMS software

b. Provide your email address, tick the check box and provide your oracle support password if you want to receive security updates from oracle support and click ' Next >'

c. Skip software updates and click ' Next>'

d. Select the option 'Install Database Software Only' and click ' Next>'

e. Select the option 'Oracle Real Application Clusters database installation', and select all nodes. and click ' Next>'

Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between nodes. Type in the OS password for the oracle user and click setup.

f. To confirm English as selected language and click ' Next >'

g. Make sure radio button 'Enterprise Edition' is ticked &click ' Next>'

i. Specify path for oracle base(ORACLE_BASE) and location where we want to store the software (ORACLE_HOME). Click ' Next>'

j. Specify groups:
Database Administrator (OSDBA) Group: dba
Database Operator (OSOPER) Group: oper

k. Oracle Universal Installer performs prerequisite checks.

Check that the status of all check is 'Succeeded' and click ' Next>'

l. After Check Summary

m. Installation

Log in to a terminal window as root user and run the root.sh script on the first node. when finished do the same for all other nodes in cluster as well. when finished click 'Ok'.
Note: root.sh should be run on one node at a time.

n. Click 'Close' to finish the installation of the RDBMS software.

>> 4. ASMCA to Create Diskgroups

a. Login as "grid" user

grid@dcpdb1 [/home/grid]$ cd $ORACLE_HOME/bin

grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./asmca

b. Click 'Create' to create a new diskgroup. Then, type in a name for the diskgroup, select the redundancy we want to provide and mark the tick box for the disks you want to assign to the new diskgroup. we have used allocation unit size 4MB for DATA. Finally click 'Ok' button.

c. Click 'Create' to create diskgroup for the flash recovery area. then, type in a name for the diskgroup, select the redundancy we want to provide and mark the tick box for the disks you want to assign to the new diskgroup. we have used allocation unit size 4MB for FRA. Finally click 'Ok' button.

d. Finally, click on 'Exit'.

>> 5. Separate VOTING file from OCR diskgroup to VOTING diskgroup.
At the time of grid installation, voting disk was created on OCR diskgroup as I chosen OCR diskgroup on grid installation. Now, I will separate the voting from OCR diskgroup to VOTING diskgroup. below are the procedure to do this job:
>> Login as "grid" user.
grid@dcpdb1 [/home/grid]$ cd $ORACLE_HOME/bin
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3088
Available space (kbytes) : 259032
ID : 1258885786
Device/File Name : +OCR
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 4610bd9667534f64bfe72c04a7f30bdd (/dev/oracle/asm_ocr01) [OCR]
2. ONLINE 8c5ee368b6094f71bf4442d5f7a87dd0 (/dev/oracle/asm_ocr02) [OCR]
3. ONLINE 5063be9ed5ef4fb1bfe16fbbdf9d4a2a (/dev/oracle/asm_ocr03) [OCR]
Located 3 voting disk(s).
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./crsctl replace votedisk +VOTING
Successful addition of voting disk 58507a1a88fc4ffcbf39fe7e58f15548.
Successful addition of voting disk 08272a459d914ff5bfc542ba1817c1c0.
Successful addition of voting disk 103e76316e414f64bfdb54b3d0fcdabc.
Successful deletion of voting disk 4610bd9667534f64bfe72c04a7f30bdd.
Successful deletion of voting disk 8c5ee368b6094f71bf4442d5f7a87dd0.
Successful deletion of voting disk 5063be9ed5ef4fb1bfe16fbbdf9d4a2a.
Successfully replaced voting disk group with +VOTING.
CRS-4266: Voting file(s) successfully replaced
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 58507a1a88fc4ffcbf39fe7e58f15548 (/dev/oracle/asm_voting01) [VOTING]
2. ONLINE 08272a459d914ff5bfc542ba1817c1c0 (/dev/oracle/asm_voting02) [VOTING]
3. ONLINE 103e76316e414f64bfdb54b3d0fcdabc (/dev/oracle/asm_voting03) [VOTING]
Located 3 voting disk(s).
grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$

>> 6. Run NETCA to Create Listener for additional port (ALL RAC NODES)

a. Login as "grid" user

grid@dcpdb1 [/home/grid]$cd $ORACLE_HOME/bin

grid@dcpdb1 [/u01/app/11.2.0.4/grid/bin]$netca

b. Net configuration Assistance: Listener Configuration, Listener

c. Net configuration Assistance: Listener Configuration, Listener Name

d. Net configuration Assistance: Listener Configuration, Select protocols

e. Net configuration Assistance: Listener Configuration, Select port

f. Net configuration Assistance: Listener Configuration, More Listener?

g. Net configuration Assistance: Listener Configuration, done

>> 7. Run DBCA to create Database.

a. Login in as "oracle" user.

oracle@dcpdb1 [/home/oracle]$ cd $ORACLE_HOME/bin

oracle@dcpdb1 [/u02/app/oracle/product/11.2.0.4/dbhome_1/bin]$ dbca

> select 'Oracle Real Application Clusters database' and click 'Next >'

b. Choose option 'Create a Database' and click 'Next >'

c. Select the database template that you want to use for your database and click ' next >'

d. Type in the name you want to use for your database and select all nodes before you click ' Next>'

e. select the options you want use to manage your database and click 'Next >'

>> Enterprise Manager

>> Automatic Maintenance Tasks

f. Type the password you want to use and click 'Next >'

g. Select the diskgroup you created for the database files and Click on 'Multiplex Redo logs and control files.

>In the popup window define the diskgroup that should contain control files and redo log files and the diskgroup that should contain the mirrored files.

>Specify ASMSNMP password specific to ASM. When all are correct then click on Next >

h. Specify the diskgroup that was created for the flash recovery area and define the size. if the size is smaller than recommended a warning will popup.

i. Select if you want to have sample schemas created in database and click 'Next>'

j. Review and change the settings for-
>>Memory Settings

>>Block Sizing & OS user process

>>Setting Character sets

>>Settings Connection Mode

>> Setting All initialization Parameters

k. Database storage setting-
>> Control files:

>> Data files:

>> Redo log groups:

Note: At first we have created 5 REDO LOG GROUPS# for Each THEAD#. Each REDO LOG GROUP consists two member and each member size 800MB. Later, we have created/increased another one REDO LOG GROUPS for each THREAD#. Finally, we had 6 REDO LOG GROUPS for each THREAD#, which consists two members and each member size 800MB.

l. Make sure the tickbox 'Create Database' is ticked and click 'Finish'.

n. Review the database configuration details again and click 'Ok'.

m. The database is now created, we can either change or unlock password or just click Exit to finish the database creation.

>> 8. Tablespace, Schema, Export & Import and Other Configuration.

8.1 Diskgroup Imbalance Check (Is your disk group balanced?)

Run the following query in your ASM instance to get the report on the disk group imbalance.
grid@dcpdb1 [/home/grid]$ sqlplus / as sysasm

SQL>
column "Diskgroup" format A30
column "Imbalance" format 99.9 Heading "Percent|Imbalance"
column "Variance" format 99.9 Heading "Percent|Disk Size|Variance"
column "MinFree" format 99.9 Heading "Minimum|Percent|Free"
column "DiskCnt" format 9999 Heading "Disk|Count"

column "Type" format A10 Heading "Diskgroup|Redundancy"

SQL>

SELECT g.name "Diskgroup",100*(max((d.total_mb-d.free_mb)/d.total_mb)-min((d.total_mb-d.free_mb)/d.total_mb))/max((d.total_mb-d.free_mb)/d.total_mb) "Imbalance",100*(max(d.total_mb)-min(d.total_mb))/max(d.total_mb) "Variance",100*(min(d.free_mb/d.total_mb)) "MinFree",count(*) "DiskCnt",g.type "Type" FROM v$asm_disk d, v$asm_diskgroup g WHERE d.group_number = g.group_number and d.group_number <> 0 and d.state = 'NORMAL' and d.mount_status = 'CACHED' GROUP BY g.name, g.type;

SQL>
select NAME, TOTAL_MB, FREE_MB from v$asm_disk;
select total_mb,free_mb from v$asm_diskgroup;
Select operation, state, est_work, est_minutes from v$asm_operation;
select group_number, name, total_mb, free_mb, type from v$asm_diskgroup where name='FRA';

SQL>

show parameter limit;

select * from v$asm_operation;

alter diskgroup FRA rebalance power 5;

8.2 Create Tablespace and add space to it under an OMF system:

SQL> Show parameter DB_CREATE_FILE_DEST

SQL> show parameter db_create

Create Tablespace ASM.
SQL> CREATE TABLESPACE xxxDATA DATAFILE SIZE 30G;

SQL> CREATE TABLESPACE xxxxINDEX DATAFILE SIZE 30G;

SQL> select tablespace_name from dba_data_files;

Add Datafile to Tableapace:

SQL>ALTER TABLESPACE xxxxDATA ADD DATAFILE SIZE 30G;

8.3 Create Schema/User:

CREATE USER xxxxx IDENTIFIED BY *****

DEFAULT TABLESPACE xxxxDATA

TEMPORARY TABLESPACE TEMP

QUOTA UNLIMITED ON xxxxDATA

QUOTA UNLIMITED ON xxxxINDEX;

GRANT CONNECT TO xxxxx;

GRANT DBA TO xxxxx;

GRANT RESOURCE TO xxxxx;

GRANT EXP_FULL_DATABASE TO xxxxx;

GRANT IMP_FULL_DATABASE TO xxxxx;

8.4 create datapump directory :

CREATE DIRECTORY dir as '/db_dump/soft/';

GRANT READ,WRITE ON DIRECTORY dir to xxxxx;

8.5 Export & Import:

Import
impdp xxxxx/xxxx directory=dir remap_tablespace=xxxxINDEX:xxxxxINDEX

dumpfile=nblt24ac_010820152321%U.dmp parallel=4 cluster=N logfile=nblt24ac_010820152321_imp.log

Import Exclude a INDEX
impdp t24test/xxxx directory=dir1 dumpfile=nblt24ac_121120150241%U.dmp remap_schema=nblt24:t24test remap_tablespace=xxxxDATA:T24TESTDATA xxxxINDEX:T24TESTINDEX parallel=4 EXCLUDE=INDEX:\"IN $\'IX_FBNK_STMT_ENTRY\'$\" cluster=N logfile=t24test.log metrics=y

Export

expdp xxx/xxxxx DIRECTORY=dir dumpfile=nblt24ac_010820152321%U.dmp

logfile=xxxx_010820152321.log schemas=xxx parallel=4 EXCLUDE=STATISTICS CLUSTER=N metrics=y

8.6 Add LOGFILE [Optional]

Node 1:
ALTER DATABASE ADD LOGFILE THREAD 1
GROUP 6 ('+REDO','+CONTROL') SIZE 800M;
Node 2:
ALTER DATABASE ADD LOGFILE THREAD 2
GROUP 12 ('+REDO','+CONTROL') SIZE 800M;

Node 1:
ALTER SYSTEM CHECKPOINT GLOBAL;
alter system switch logfile;
Node 2:
ALTER SYSTEM CHECKPOINT GLOBAL;
alter system switch logfile;

8.7 Listener [ALL RAC Nodes]

>> Listener in "Grid" user
endpoints_listener.ora
LISTENERIT_DCPDB1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip)(PORT=1526))(ADDRESS=(PROTOCOL=TCP)(HOST=172.31.xx.xx)(PORT=1526)(IP=FIRST)))) # line added by Agent
LISTENERNBL_DCPDB1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip)(PORT=1525))(ADDRESS=(PROTOCOL=TCP)(HOST=172.31.xx.xx)(PORT=1525)(IP=FIRST)))) # line added by Agent

LISTENER_DCPDB1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=172.31.xx.xx)(PORT=1521)(IP=FIRST)))) # line added by Agent

listener.ora
LISTENERIT=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENERIT)))) # line added by Agent
LISTENERNBL=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENERNBL)))) # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENERNBL=ON # line added by Agent

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENERIT=ON # line added by Agent

Local Listener in Oracle [In oracle user]
alter system set local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1- vip.xxxx.com)(PORT=15xx))(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb1-vip.xxx.com)(PORT=19xx))))' scope=BOTH SID='dcpdb1';
alter system register;
alter system set local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb2- vip.xxx.com)(PORT=15xx))(ADDRESS=(PROTOCOL=TCP)(HOST=dcpdb2-vip.xxx.com)(PORT=19xx))))' scope=BOTH SID='dcpdb2';
alter system register;

8.8 Client tnsnames

For Scan
xxx =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb-scan.xxx.com)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = dcpdb.xxxx.com)
)
)

OR

For VIP
xxx =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDEASS = (PROTOCOL = TCP)(HOST = dcpdb1-vip.xxx.com)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb2-vip.xxx.com)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb1-vip.xxx.com)(PORT = 1525))
(ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb2-vip.xxx.com)(PORT = 1525))
(ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb1-vip.xxx.com)(PORT = 1526))
(ADDRESS = (PROTOCOL = TCP)(HOST = dcpdb2-vip.xxx.com)(PORT = 1526))
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = dcpdb.xxx.com)
)

)

References:

http://docs.oracle.com/cd/E11882_01/install.112/e10815.pdf
http://www.oracledba.org/11gR2/Pre_Install_11gR2.htm
http://toddlerdba.blogspot.com/2013/01/oracle-11gr2-rac-on-hpux-itanium-1131.html
http://docs.oracle.com/cd/E11882_01/install.112/e49316/pre_install.htm
http://docs.oracle.com/cd/E11882_01/install.112/e48295/pre_hpux.htm
http://h18006.www1.hp.com/storage/pdfs/4AA2-7668ENW.pdf

Database Administrator's blog

Pages

Tuesday, July 28, 2015

Oracle 11g Release 2 (11.2.0.4.0) RAC Configuration on HP-UX Itanium (11.31) Server

No comments:

Post a Comment