System and network installations.
TABLE OF CONTENT:
System and network installations.
1. Prerequisites and hardware.
4.3) Copy installation scripts on NFS server.
5. iSCSI and raw devices configuration preparing for RAC and ASM installation.
5.3) Running iscsi and creating volumes on iSCSI.
5.6) Installing rawnames script, turning on iSCSI, and testing how it works thru system reboot.
5.7) Now reboot to verify that everything will come well.
Table of scripts.
File 002.create-lvmgroups.1sh:
1. Prerequisites and hardware.
You must have (these all are commercial products, available for evaluation and/or under special development licenses):
- SuSe Linux Enterprise Server 9 (better with Service Pack 1);
- Oracle 10.1.0.3 for AMD64/ET64 (2 CD or 2 cpio files) with Cluster Ready Services (1 CD or 1 cpio file). I believe that 10.1.0.4 will not make any difference.
- 2 AMD64/ET64 servers (I used DELL 2850 servers with 2 CPU each);
- Network Appliance (NetApp) NAS system with iSCSI license;
- 1Gbit Ethernet switch (or VLAN on big enterprise switch).
Logical connection in my example:
Filer is connected to storage VLAN (and is connected, in my case, to access VLAN for redundancy). Filer IP are:
- fas-1a-1 (controller 1)
- fas-1b-1 (controller 2)
2. System installation.
2.1) Allocate names and IP addresses. You will need 4 IP in access network and 2 IP in storage network.
Virtual IP are used by Oracle cluster as service IP and can float from failed server to server which takeover its functions. Add records for primary IP (not virtual) into hosts file (IMPORTANT! I saw problems if IP resolved into fully qualified names, not into single world names).
We used in these example:
|
Server |
eth0-ip |
eth0 virtual ip |
eth1-ip |
|
testrac11 |
testrac11 = 10.25.32.111 |
testrac11-vip = 10.25.32.113 |
testrac11-1 = 10.253.23.111 |
|
testrac12 |
testrac12 = 10.25.32.112 |
testrac12-vip = 10.25.32.114 |
testrac12-1 = 10.253.23.112 |
Configure these names in DNS:
|
Host |
Domain |
Type |
Pref |
Data |
|
testrac11 |
A |
|
10.25.32.111 |
|
|
testrac11-1 |
A |
|
10.253.32.111 |
|
|
testrac11-vip |
A |
|
10.25.32.113 |
|
|
testrac12 |
A |
|
10.25.32.112 |
|
|
testrac12-1 |
A |
|
10.253.32.112 |
|
|
testrac12-vip |
A |
|
10.25.32.114 |
IMPORTANT – ASSIGN ALL IP BEFORE INSTALLATION. DOUBLE CHECK THESE NAMES, or DOUBLE CHECK YOUR DNS. 90% of cluster installation problems have been caused by improper host names. I recommend using ‘/etc/hosts’ file for your configuration, see example here:
|
10.25.32.111 testrac11 testrac11.mydomain.com 10.25.32.112 testrac12 testrac12.mydomain.com 10.253.32.111 testrac11-1 testrac11-1.mydomain.com 10.253.32.112 testrac12-1 testrac12-1.mydomain.com 10.25.32.113 testrac11-vip 10.25.32.114 testrac12-vip |
2.2) Install SuSe Linux Enterprise server 9, with service pack1. Here are my selections:
I recommend to install, at least:
- Basic runtime system;
- Graphical base system;
- Linux tools;
- LSB Runtime environment;
- help and support documentation;
- C/C++ development;
- Kernel sources (from Various Linux tools).
If you are doing it all very first time, better follow setting above without variations.
2.3) Configure network. [1]
Do not forget to EDIT existing interface first (which will have static IP), then add second interface and set it up for back network. See here:
Set up jumbo frames for second interface:
2.4) Upgrade system to ServicePack 1.[2]
There are many methods to upgrade. You can login as root from console (using GUI, not text mode), insert SP1 disk, and follow prompt. Or you can, instead, select ‘YaST2 -> System -> Patch CD update, and use your copy of SP1 disks as a source.
3. Images and X11.[3]
3.1) System images.
I recommend setting up external NFS image server, and creating installation sources by copying installation disks:
- SLES9 CD1 as SLES9-CORE (copy CD1 into these directory);
- SLES9 CD2-CD6 as SLES9-Image (copy CD2-CD6 into this directory accepting all rewrites);
- SLES9 SP1 CD1-3 as SLES9-SP1 (copy SP1 disks into this directory accepting all rewrites);
For example:
After installing basic system, I open YaST2, mount these disks in NFS, and then set up new installation sources:
Now you can install any component, without inserting CDs.
3.2) Prepare Oracle installation disks.
You will need Oracle/DB and Oracle/CRS directories (first with all CD's from Database, second with CD for Cluster Ready Services) from Oracle 10g for Linux x86-64. You can use CD and copy files from them, install from CD directly, or (if you have account and can download) you can download files from Oracle:[4]
ship.crs.lnxx86-64.cpio ship.db_Disk2.lnxx86-64.cpio.gz ship.db_Disk1.lnxx86-64.cpio.gz |
Now expand them (on local disks on onto image server), for example:
|
mkdir
/image/UNIX/Oracle10 |
(You must agree with development license if you download these files).
Important. If you use Windows NFS server, do not copy disks on Windows - these CD's contain symlinks, so you must use Unix for unarchiving and copy operations.
For example:
3.3) Copy installation scripts on NFS server.
Now, take installation scripts attached to this document, and unpack them on this NFS server. You must have these scripts available from all nodes, for simple installation.
3.4) X11 access.
Oracle installation requires X11 screen (X11 is Unix windows system). There are many X11 systems available (and you always have one when are working from Linux console in graphical mode). I usually use ‘CYGWIN’ (http://www.cygwin.com/) package for Windows, when I must use Microsoft windows screen, installing:
- Basic cygwin set;
- Basic X11 set;
- Open SSH;
After it, you can login into SuSe server using such commands:
- Start cygwin terminal (click on icon);
- startx & - opens xterm window;
- slogin –X root@hostname
Or, better, allow XDM on SUSE Linux (open YaST2 -> system -> /etc/sysconfig editor, search for ‘DISPLAYMANAGER_REMOTE_ACCESS’,
Set it to ‘yes’, Apply, then run from console: ‘init 3’, then ‘init 5’), then run X in query mode:
- X –query host-name
Important notices:
- To change user when you have X11 access, use ‘sux’ instead of ‘su’. For example: ‘sux – oracle’;
- To login into remote host with X11 forwarding, run ‘slogin –X ….’.
You can use any other method to get X11 access (xhosts, etc…). Just be prepared – you will need X11 access. The best method is use Linux console in graphical mode.
3.5) Starting installation.
Login as root on both servers (open 2 xterm windows, 1 from first server, 1 from second). For example. I run ‘X –query testrac11’, login as myself, then:
- Click on terminal icon, and have terminal for the first server, run ‘sux –‘ to became root;
- Click on terminal icon, then run ‘slogin –X root@second-server’ and login onto second node (with X11 forwarding).
Now it’s time to begin using scripts. Copy them onto image server (you can use on of RAC nodes as such server), and mount it.
testrac12:~ # mkdir -p /image/UNIX
testrac12:~ # mkdir -p /image/REP
testrac12:~ # YaST2 nfs &
testrac12:~ # df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 20971896 2481108 18490788 12% /
tmpfs 2026756 16 2026740 1% /dev/shm
/dev/sda3 10181952 32828 9631904 1% /u01
image:/Repository 195358400 174076348 21282052 90% /image/REP
image:/UNIX 195358400 174076344 21282056 90% /image/UNIX
Now, chdir to installation directory and edit CONFIG.sh file.
testrac12:/image/UNIX/SuSe9 # cd /image/UNIX/SuSe9/SLES9-ORA10RAC-iSCSI-ASM
testrac12:/image/UNIX/SuSe9/SLES9-ORA10RAC-iSCSI-ASM # ls ..
. SLES9-ORA10RAC-iSCSI-ASM SLES9-x64-ORA10.tgz docs tests
.. SLES9-i386-ORA10 SLES9-x64-ORA10RAC index.html
RPMs SLES9-x64-ORA10 SLES9-x64-ORA10RAC.tgz iscsi-files
testrac12:/image/UNIX/SuSe9/SLES9-ORA10RAC-iSCSI-ASM # ln -s `pwd` /INST
testrac12:/image/UNIX/SuSe9/SLES9-ORA10RAC-iSCSI-ASM # kwrite CONFIG.sh
4. Filer and server iSCSI configuration.[5]
4.1) Filer configuration.
Now we must open NetApp and configure iSCSI lun’s on it. I configured (in these example) 4 iSCSI luns – OCRFile (100Mb), CSSFile (20Mb), DB1 for ASM disk group for database, and LOGS2 for ASM group for logs.
FAS-1a:
|
LUN |
Description |
Size |
Status |
Maps |
|
OCR file for testrac cluster |
20 MB |
online |
||
|
OCR file for testrac cluster |
8 GB |
online |
||
|
LOGS and REDO |
8 GB |
online |
||
|
OCR file for testrac cluster |
100 MB |
online |
In cluster configuration (optional) I configured 2 more LUN’s on second filer in the cluster:
|
LUN |
Description |
Size |
Status |
Maps |
|
DB2 ASM disk |
8 GB |
online |
||
|
LOGS and REDO for testrac |
8 GB |
online |
(As you can see, I used sequential LUN numbers, just for simplicity).
Initiator group configured as:
|
Group Name: |
|
|||
|
|
||||
|
Type: |
|
|||
|
|
||||
|
Operating System: |
|
|||
|
|
||||
|
Initiators: |
iqn.1987-05.com.exigengroup:testrac11.sjclab iqn.1987-05.com.exigengroup:testrac12.sjclab |
|
||
Names must be the same as you configured in ‘/etc/initiatorname.iscsi’ above.
4.2) Create initiator ID.
You must create iSCSI ID on each of the servers. You can do it by ANY of 2 methods:
-
start iscsi very first
time and allow it to create iscsi id for you:
vi /etc/iscsi.conf
(add
DiscoveryAddress=Filer-IP-1
DiscoveryAddress=Filer-IP-2
and run
rciscsi start
rciscsi stop
)
-
Create ID manually,
editing file:
vi /etc/initiatorname.iscsi
file should contain only comments and 1 line with the name:
InitiatorName=iqn.1987-05.com.reverse-domain:server-name
I used such names in this installation:
testrac11: InitiatorName=iqn.1987-05.com.exigengroup:testrac11.sjclab
testrac12: InitiatorName=iqn.1987-05.com.exigengroup:testrac12.sjclab
This approach provides more readable names. You will need these names on the next step (so do it here, not in iSCSI configuration section).
5. iSCSI and raw devices configuration preparing for RAC and ASM installation.
Now we can start iSCSI and create all necessary volumes. We will use Automated Storage Management for Oracle data (including archived logs) and raw devices for Cluster Ready Services. To simplify these tasks, I used introduced modified ‘raw’ script (rawnames) which allow me to use symbolic names in Oracle and CRS instead of magical ‘raw1’, ‘raw2’ etc (which is extremely tricky when it’s all about dynamic iSCSI disks).
5.1) List of raw devices.
We must configure few raw devices for RAC CLUSTER:
1) OCRFile – 100 Mb – for CRS server;
2) CSSFile – 20 Mb – for CSS server (cache synchronization);
3) DB1 (and DB2 in my example) – 8 Gb – ASM storage for database;
4) LOGS1 (and LOGS2 in my example) – for REDO and archive logs (in my example).,
(Only first 3 are required for minimal installation).
5.2) Patching iscsi script.[6]
You can skip this step, if you do not experience long Ethernet card negotiation OR do not use LVM (I used it before).
Script 001-patch-iscsi.sh apply patch to iscsi script and install rawnames script (script and patch are in FILES.d directory).
Script 001-patch-iscsi.sh:
|
#!/bin/sh # # 001 script - patch iscsi # . CONFIG.sh echo "Patching system iscsi script" patch -d /etc/init.d < FILES.d/iscsi.patch
|
Patch iscsi.patch:
|
*** /etc/init.d/iscsi~ Wed Dec 15 14:01:46 2004 --- /etc/init.d/iscsi Mon Mar 21 22:42:52 2005 *************** *** 26,31 **** --- 26,32 ----
test -d $BASEDIR || exit 0
+ rc_reset for configfile in /etc/iscsi.conf /etc/initiatorname.iscsi; do if [ ! -f ${configfile} ]; then *************** *** 167,172 **** --- 168,177 ---- echo "$min $default $TCP_WINDOW_SIZE" > /proc/sys/net/ipv4/tcp_wmem fi fi + # Take some time to adapt interfaces + echo -n "Sleeping 30 seconds to allow port negotiations... " + sleep 30 + # ping -c 2 10.254.32.106 || sleep 30
# start echo -n "Starting iSCSI: iscsi" *************** *** 219,224 **** --- 224,233 ---- touch /var/lock/subsys/iscsi fi
+ # + # Now, sleep 20 seconds to allow iSCSI discover targents, and then run LVM again to red new volumes + echo -n " sleeping 20 seconds... " && sleep 20 + /etc/init.d/boot.lvm $* # if we have an iSCSI fstab, process it if [ -f /etc/fstab.iscsi ] ; then echo -n " fsck/mount " |
Output:
testrac12:/INST # sh 001*sh
Patching system iscsi script
patching file iscsi
testrac12:/INST # ls /etc/init.d/rc5.d
. K10sshd S01irq_balancer S12ldap
.. K12nfs S01isdn S12rawnames
K07splash_late K12nfsboot S01random S12running-kernel
K07xdm K14portmap S02coldplug S12sshd
K08cron K14resmgr S05network S13kbd
K08hwscan K14slpd S06syslog S13postfix
K08nscd K14smbfs S08portmap S13powersaved
K09postfix K14splash_early S08resmgr S13splash
K09powersaved K16syslog S08slpd S14cron
K09splash K17network S08smbfs S14hwscan
K10alsasound K20coldplug S08splash_early S14nscd
K10cups K21hotplug S10nfs S15splash_late
K10fbset K21irq_balancer S10nfsboot S15xdm
K10ldap K21isdn S12alsasound
K10rawnames K21random S12cups
K10running-kernel S01hotplug S12fbset
Run it on BOTH nodes.
5.3) Running iscsi and creating volumes on iSCSI.
Now, you must configure and start iscsi on the servers. First, edit /etc/iscsi.conf file, adding target IP (and possible some other parameters, for redundancy for example). Start iscsi on node-1 first (do not start it until you created logical volumes). Here is my configuration:[7]
# iSCSI configuration file - see iscsi.conf(5)
PortalFailover=yes
PreferredSubnet=10.254.0.0/16
PreferredPortal=10.254.32.105
PreferredPortal=10.254.32.106
#
DiscoveryAddress=10.254.32.105
DiscoveryAddress=10.254.32.106
rciscsi start
…..
testrac12:/INST # netstat -an | grep 3260
testrac12:~ # netstat -an | grep 3260
tcp
0 0 10.254.32.112:32811
10.254.32.106:3260 ESTABLISHED
tcp 0 0
10.254.32.112:32778
10.254.32.106:3260 ESTABLISHED
tcp 0 0
10.254.32.112:32812
10.254.32.105:3260 ESTABLISHED
tcp 0 0
10.254.32.112:32794
10.254.32.105:3260 ESTABLISHED
5.5) Creating RAW devices.[8]
After you started iscsi, you must found your lun’s (created on filer) on server.
testrac11:/INST # ls /dev/disk/by-id/iscsi*
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-0
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-1
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-2
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-5
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1b-3
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1b-4
testrac11:/INST # od
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1b-4
0000000 000000 000000 000000 000000 000000 000000 000000 000000
*
^C
… (check all devices).
You can see this LUN’s in YAST:
Now, edit ‘rawnames.conf’ file (in installation scripts) to create raw devices for all these names. Use content of /dev/disk/by-id, and lun numbers from filer (see table above):
# ls /dev/disk/by-id/iscsi*
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.eqx-fas-1a-10
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.eqx-fas-1b-11
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-0
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-1
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-2
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-5
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1b-3
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1b-4
testrac11:/INST # ls /dev/disk/by-id/iscsi* >> FILES.d/rawnames.conf
testrac12# vi FILES.d/rawnames.conf
# /etc/rawnames.conf
#
# The format of this file is:
# raw<N>,<blockdev>,symbolic-name[,owner]
#
# example:
# ---------
# raw1,hdb1,localdisk
#
# this means: bind /dev/raw/raw1 to /dev/hdb1
# and link it to /dev/rawnames/localdisk
# Delimiter ',' used to allow ':' in device names (such as iSCSI devices).
# ...
# This group if for iSCSI ASM installation
raw1,disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-0,OCRFile
raw2,disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-1,CSSFile
raw3,disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-2,DB1
raw4,disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1a-5,LOGS2
raw5,disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1b-3,DB2
raw6,disk/by-id/iscsi-iqn.1992-08.com.netapp:exigengroup.fas-1b-4,LOGS1
5.6) Installing rawnames script, turning on iSCSI, and testing how it works thru system reboot.
File 005-copy-rawnames.sh:
|
#!/bin/sh # cp FILES.d/rawnames /etc/init.d/rawnames insserv rawnames # echo 'Now verify rawnames.conf shared file' if [ -f /etc/rawnames.conf ] then echo You have already /etc/rawnames.conf installed echo Here is difference with shared file in scripts diff /etc/rawnames.conf FILES.d/rawnames.conf echo -n 'You can ^C to abort or press Enter to continue_' read x fi echo vi FILES.d/rawnames.conf vi FILES.d/rawnames.conf # echo We are about to copy rawnames.conf into system and activate iscsi echo -n Press ENTER to continue_ read x cp FILES.d/rawnames.conf /etc/rawnames.conf insserv iscsi rciscsi start # # repeat these 2 servcies here, because we added something between # /etc/init.d/boot.lvm start /etc/init.d/rawnames start ls -l /dev/rawnames |
Run on both nodes:
sh 005*sh
5.7) Now reboot to verify that everything will come well.
reboot
and when systems come back, verify that it started iSCSI, iSCSI disks and created ‘rawnames’ devices.
testrac11:/INST # ls
/dev/rawnames
. .. CSSFile DB1 DB2 LOGS1 LOGS2
OCRFile
I zero new lun’s (do not forget to configure reallocate’ on filer, to prevent performance degradation in time) and verify their size by running script (I have not OTHER devices in rawnames):
testrac11:/INST # for i
in /dev/rawnames/*; do dd if=/dev/zero of=$i bs=1M;
echo $i;done
dd: writing `/dev/rawnames/CSSFile': No space left on device
21+0 records in
20+0 records out
/dev/rawnames/CSSFile
dd: writing `/dev/rawnames/DB1': No space left on device
8193+0 records in
8192+0 records out
/dev/rawnames/DB1
…
CONGRATULATIONS. You can proceed with real ORACLE installation.
[1] I found important network issue in case of
DELL 2850 servers connected to Cisco 2970 catalist switch – port negotiation
took about 20 seconds, after system already treated interfaces as connected.
So, I added 30 second delay into /etc/init.d/iscsi script (which I do modify by
patch) because these script runs just after network script. You can add ‘sleep
30’ into ‘/etc/init.d/network’ script instead. Verify system reboot after
installation, and if you see that system cannot mount NFS disks in boot time,
but works fine later, add such delay.