![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
Hardware: xSeries server Model: x345 [8670-IQS], QL2300 based HBA cards, IBM FAStT600, SAN Switch [2005H16]
Operating System: Red Hat Advanced Server v2.1
Software: IBM UDB DB2 v8.2, TSA for Multiplatform v1.2.0.0 and patch v1.2.0.2, SAM policies v1.2.1.0-0, IBM Storage Manager v9.1
I. Install and configure the HBA Cards
Install the HBA adapters in the nodes, if they are not already installed. Disconnect the fiber optic cable for now. We will connect the cable later on. Switch on the nodes and press CTRL-Q to get the Controller Card Numbers. It will be long 16-digit number
Example:
node1 210000E08B130ECB node2 210000E08B1309C0 |
TIP: Record these numbers from both the nodes. These will be required while configuring storage partitioning in Storage Manager
Do the following on both the nodes for ALL the available HBA cards:
Loop reset delay : Change this setting to 8
Adapter Port Name : Note this number as it will be required later for storage partitioning
Select the Advanced adapter settings:
LUNs per target: This should be 0 (zero)
Enable target reset : This should be yes
Port down retry count: Change this to 12
There are two different packages available for the IBM FAStT HBA family:
a. A fail-over (or multipath) device driver for FAStT attachment
b. A single path driver for ESS attachment
NOTE: We will be using the multipath driver in our setup.
Copy the tar file from the supplied CD to any temporary directory. i.e. /dump
# cp /mnt/cdrom/i2x00-fo*.tgz /dump # tar -zxvf /dump/i2x00-fo*.tgz # cd /dump/i2x00-v6.0 # sh drvsetup # ./libinstall [OPTIONAL] |
For Red Hat, the drivers require that the source headers be prepared before compilation.
Change to the kernel source directory:
# cd /usr/src/linux2.4 # vi Makefile [Search for EXTRAVERS] Before EXTRAVERS -e.24custom After EXTRAVERS -e.24 # make menuconfig (to load the Linux Kernel Configurator tool). |
Check the processor setting is correct (SMP, cyclone support for x440 (not required for x360, not necessary but suggested for x440 with one CEC) Exit and save the configuration.
# make dep The fiber HBA driver can now be compiled. # cd /dump/ix200-v6.0/ # make all [For an SMP machine ; # make all SMP=1] The above command will generate qla2200.o and qla2300.o modules |
II. Loading HBA driver under Red hat
# /etc/modules.conf alias eth0 bcm5700 alias scsi_hostadapter ips alias scsi_hostadapater2 qla2300 options scsi_mod max_scsi_luns=128 :wq! # depmod -a [Updates the modules.dep files] |
Now, you have two options, either you can load qla2300 manually or automatically on system reboot
Manually :
# rmmod qla2300 or # modprobe -r qla2300 [Remove if its already loaded] # modprobe qla2300 |
Automatically :
Shutdown all the nodes.
Connect all the FAStT devices
Switch on FAStT devices first, then all the connected nodes.
Read More : Linux on xSeries and BladeCenter
III. How to configure and reset FAStT600?
Connectivity with the node
- Connectivity with the node (i.e. node1) using Cross Cable
- Default IP information
Default IPs configured : 192.168.128.101 (Controller 1) and 192.168.128.102 (Controller 2) - How to Reset?
Sometimes you need to reset the FAStT600 as the device was used for any prior setup. On resetting, the device gets reset to factory defaults. You can thereafter configure it as per your requirements.
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
Read More : Implementing Linux with IBM Disk Storage
IV. Installation of IBM Storage Manager
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
Our Test Setup
In our testing, we configured first three hard disks under RAID5 (/dev/sdb) and one hard disk under RAID0 (/dev/sd.
The partitions are created on /dev/sdb as follows :
Read More : Implementing Linux with IBM Disk Storage (Chapter-9)
V. Installation of IBM UDB DB2 v8.2
DB2 must be installed on ALL the nodes that will potentially host DB2 instance or the DAS. After the installation is complete, check for the following folder "/opt/IBM/db2/V8.1/ha/salinux" contains important scripts required later.
Default Folder: /opt/IBM/db2
DB2 : un-installation
# cd /dump/009_ESE_LNX_32_NLV # ./db2_deinstall # rm -rf /var/db2 # userdel dasusr1 # rm -rf /home/dasusr1 # vi /etc/group remove dasadm1 |
VI. Installation of TSA for Multiplatform
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
SAM should be installed locally on ALL the nodes participating in this setup.
TSA : un-installation # cd /dump/SAM12/i386/ # ./uninstallSAM |
Read more: IBM System Implementation for Multiplatforms, Guide and Reference
VII. Installation of SAM policies
Install "sam.policies" rpm on all the nodes participating in NFS sharing
# rpm -ivh /mnt/cdrom/sam.policies/sam.policies-1.2.1.0-0.i386.rpm Default Folder: /usr/sbin/rsct/sapolicies/ SAM Policies : un-installation # rpm -e sam.policies-1.2.1.0-0 |
VIII. Configure SAM to manage NFS resources
On each node i.e. node1 and node2
# /usr/sbin/rsct/bin/preprpnode node1 node2 On only one node i.e. node1 |
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
In our test, we blocked aside one hard disk on FAStT600 just for this purpose. i.e. /dev/sdc
Check for the details :
An HA NFS server relies on two types of data located on one or more disks. The first kind of data is required for the management of the NFS server. These are normally stored locally under "/var/lib/nfs" on each NFS server node. To make an NFS server HA requires these data to be stored on a small filesystem, that is also located of shared disk accessible by all the nodes potentially hosting the NFS server. Following steps must be performed to put the data from "/var/lib/nfs" on a filesystem on a shared disk.
Perform the following steps on both the nodes.
# mkdir -p /varlibnfs # vi /etc/fstab (append) /dev/sdb1 /varlibnfs ext3 noauto 0 0 |
Perform the following steps on only one node.
# mount /varlibnfs # cp / var/lib/nfs/* /varlibnfs # umount /varlibnfs |
Follow the following steps on all the nodes
# chkconfig --level 35 nfs off [Stop default NFS services] # cd /var/lib # mv nfs nfs.orig # ln -s /varlibnfs nfs |
The configuration of the exported directories is stored in the file "/etc/exports". An alternative to manually keep the files in sync on the nodes is putting the "/etc/exports" file also in the filesystem "/varlibnfs" on the shared disk, and making a link to this file from the "/etc" directory to the file on the "/varlibnfs" directory similar as for the "/var/lib/nfs" directory. But note this is not a must, its an option.
The second type of data are those, which are exported to NFS clients by the NFS server. These are stored on one
or more large filesystems on shared disks accessible by each of the nodes. make sure that there are following files
available in "/usr/sbin/rsct/sapolicies/nfsserver"
cfgnfsserver
sa-nfsserver.conf.sample
nfsserverctrl-data
nfsserverctrl-server
nfsserverctrl-server.DB2
There are two versions of the NFS server control script contained in the policy-package. Normally you use nfsserverctrl-server for the NFS Server HA policy. However if you want to make the home directory for the DB2 DPF instance highly available, then you must use the nfsserverctrl-server.DB2 script. This will not use the scripts in "/etc/init.d" directory to start the NFS server processes, but will instead start/stop the NFS server processes with the appropriate commands. To use this script, you have to rename the script nfsserverctrl-server.DB2 to nfsserverctrl-server. Save the original nfsserverctrl-server first under a different name.
To do so execute the following commands.
# cd /usr/sbin/rsct/sapolicies/nfsserver # mv nfsserverctrl-server nfsserverctrl-server.orig # mv nfsserverctrl-server.DB2 nfsserverctrl-server |
Important : Modify "nfsserverctrl-server" script to include compatible commands. After that, the policy will pick up the modified version of that script:
# vi /etc/exports [Update on both the nodes] /db2sharedhome *(rw,no_root_squash,sync) # vi /etc/fstab [Update on both the nodes] /dev/sdb2 /db2sharedhome ext3 noauto 0 0 10.0.10.108:/db2sharedhome /home nfs rw,bg,vers=2,timeo=600, / retrans=2,hard,nointr,noac,suid,nolock 0 0 # /usr/sbin/rsct/sapolicies/nfsserver/cfgnfsserver -p Run the command given below to see if the resources were actually created. # /usr/sbin/rsct/sapolicies/nfsserver/cfgnfsserver -l We must first change the Nominal State of SA-nfsserver-rg to Online, before we can expect the ResourceGroup and its members to become Online. To do that, use: # lsrg [Lists all the resources available] # chrg -o Online SA-nfsserver-rg Check the status of all the resources # /usr/sbin/rsct/sapolicies/bin/getstatus -- Resource Groups and Resources -- Group Name Resources ----------------- --------------- SA-nfsserver-rg SA-nfsserver-server SA-nfsserver-rg SA-nfsserver-ip-1 SA-nfsserver-rg SA-nfsserver-data-varlibnfs SA-nfsserver-rg SA-nfsserver-data-work Resource Name Node Name State ----------------------- ---------------- --------- SA-nfsserver-server node1 Online SA-nfsserver-server node2 Offline - - - SA-nfsserver-ip-1 node1 Online SA-nfsserver-ip-1 node2 Offline - - - SA-nfsserver-data-varlibnfs node1 Online SA-nfsserver-data-varlibnfs node2 Offline - - - SA-nfsserver-data-work node1 Online SA-nfsserver-data-work node2 Offline |
Testing the NFS server functionality
Before trying any fail-over we must ensure that NFS server works.
Update both the nodes for the following entries :
# cat /etc/exports /db2sharedhome *(rw,no_root_squash,sync) #showmount -e 10.0.10.108 Export list for 10.0.10.108 /db2sharedhome * # exportfs /db2sharedhome * |
If there are errors reported, solve them before going ahead.
Access automated NFS server from an NFS client
Go to any other node in the network. i.e. spc_client (10.0.10.3)
# mount 10.0.10.108:/db2sharedhome /db2_data # mount |
Testing the NFS fail-over
# samctrl -u a node1 [Add node1 in excluded list since this is presently Online]
# /usr/sbin/rsct/bin/sapolicies/nfsserver/getstatus
-- Resource Groups and Resources --
Group Name Resources
----------------- ---------------
SA-nfsserver-rg SA-nfsserver-server
SA-nfsserver-rg SA-nfsserver-ip-1
SA-nfsserver-rg SA-nfsserver-data-varlibnfs
SA-nfsserver-rg SA-nfsserver-data-work
Resource Name Node Name State
----------------------- ---------------- ---------
SA-nfsserver-server node1 Offline
SA-nfsserver-server node2 Online
- - -
SA-nfsserver-ip-1 node1 Offline
SA-nfsserver-ip-1 node2 Online
- - -
SA-nfsserver-data-varlibnfs node1 Offline
SA-nfsserver-data-varlibnfs node2 Online
- - -
SA-nfsserver-data-work node1 Offline
SA-nfsserver-data-work node2 Online
#lssamctrl
ExcludedNodes = {node1}
# samctrl -u d node1 [Add node1 back into automation]
|
Read More : IBM Tivoli System Automation for Multiplatforms, Version 1.1 (August 26, 2004)
Execute the following commands on ALL the nodes
# ntsysv (Select rsh, rlogin and rexec) # vi /etc/securetty rsh (Append rsh in the content) # vi /etc/hosts.equiv node1 db2tsa node2 db2tsa # init 6 |
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
The output should look similar to the line given below:
DBI1070 Program db2icrt completed successfully.
Note: Update /etc/services for the DB2 related services on all the nodes
# vi /etc/services DB2_db2tsa 60000/tcp DB2_db2tsa_1 60001/tcp DB2_db2tsa_2 60002/tcp DB2_db2tsa_END 60003/tcp xdb2tsa 60004/tcp To verify the creation of an instance, login as db2tsa #su - db2tsa $ db2start You should see the following output 0 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful. After verification, configure the database partitions. For this modify "db2nodes.cfg" file $ db2stop Now enable the Database Partitioning Feature (DPF) $ vi ~/sqllib/db2nodes.cfg Append the line shown in bold in "db2nodes.cfg": 0 node1 0 1 node2 0 :wq Start the db2 service $ db2start You should see the following output : 0 0 SQL1063N DB2START processing was successful. 1 0 SQL1063N DB2START processing was successful. $ exit |
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
XII. Configure SAM to manage DB2 resources
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
After creating the instance and configuring, we need to execute the following command to protect each of the DB2 partitions and well-known IP address i.e. 10.0.10.111
# cd /opt/IBM/db2/V8.2/ha/salinux # ./regdb2salin -a db2tsa -i 10.0.10.111 This may take a couple of minutes. On successful completion, you get a highly available instance. Example : About to register db2inst1 with TSA Checking cluster validity... Checking configuration ... Checking db2rb01 ... Checking db2rb02 ... Making resource group db2_db2inst1_0-rg ... Making IP resource db2_db2inst1_0-rs_ip ... Making DB2 resource db2_db2inst1_0-rs ... Making resource group db2_db2inst1_1-rg ... Making DB2 resource db2_db2inst1_1-rs ... Online resource group db2_db2inst1_0-rg ... Online resource group db2_db2inst1_1-rg ... db2_db2inst1_0-rg is now online db2_db2inst1_1-rg is now online Done, instance is now HA To Verify : # /usr/sbin/rsct/saplocies/bin/getstatus -- Resource Groups and Resources -- Group Name Resources ----------------- --------------- SA-nfsserver-rg SA-nfsserver-server SA-nfsserver-rg SA-nfsserver-ip-1 SA-nfsserver-rg A-nfsserver-data-varlibnfs SA-nfsserver-rg SA-nfsserver-data-work db2_db2tsa_0-rg db2_db2tsa_0-rs db2_db2tsa_0-rg db2_db2tsa_0-rs_mount db2_db2tsa_0-rg db2_db2tsa_0-rs_ip db2_db2tsa_1-rg db2_db2tsa_1-rs db2_db2tsa_1-rg db2_db2tsa_1-rs_mount -- Resources -- Resource Name Node Name State ----------------------- ---------------- --------- SA-nfsserver-server node1 Online SA-nfsserver-server node2 Offline - - - SA-nfsserver-ip-1 node1 Online SA-nfsserver-ip-1 node2 Offline - - - SA-nfsserver-data-varlibnfs node1 Online SA-nfsserver-data-varlibnfs node2 Offline - - - SA-nfsserver-data-work node1 Online SA-nfsserver-data-work node2 Offline - - - db2_db2tsa_0-rs node1 Offline db2_db2tsa_0-rs node2 Online - - - db2_db2tsa_0-rs_mount node1 Offline db2_db2tsa_0-rs_mount node2 Online - - - db2_db2tsa_0-rs_ip node1 Offline db2_db2tsa_0-rs_ip node2 Online - - - db2_db2tsa_1-rs node1 Online db2_db2tsa_1-rs node2 Offline - - - db2_db2tsa_1-rs_mount node1 Online db2_db2tsa_1-rs_mount node2 Offline All DB2 resources are now created. Next its time to test the HA db2 instance. Create a sample database by executing "db2sampl" # su - db2tsa $ db2sampl A SAMPLE database will be created $ db2 connect to sample $db2 Command to test the DPF feature of DB2 db2> select distinct dbpartitionnum (empno) from employee The above command should display the result as shown below : 0 1 ___ 2 records selected |
| XML error: The image is not displayed because the width is greater than the maximum of 580 pixels. Please decrease the image width. |
- High Availability for partitioned databases using Tivoli System Automation - July 2004
- Linux on xSeries and BladeCenter
- IBM System Automation for Multiplatforms, Guide and Reference
- IBM Tivoli System Automation for Multiplatforms, Version 1.1 (August 26, 2004)
- Implementing Linux with IBM Disk Storage
Harish has been with IBM since 1998 and has 13 years of experience. During his last seven years with IBM, he has spent five years at the India Research Lab and over a year and a half at the IBM T.J.Watson Research Center. Harish currently leads the Linux Center of Competency in Bangalore, India. You can contact him at hchauhan@in.ibm.com.
Table of contents
Next steps from IBM
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
