 | Warning
This page still needs heavy tweaking! |
In this section, we describe IBM Tivoli System Automation (TSA) for Multiplatforms and apply it to an example scenario of creating a high availability DB2 instance using TSA and shared storage.
Introduction to Tivoli System Automation
Tivoli System Automation for Multiplatforms (TSA) is a policy-based high availability solution for Linux and AIX that automates the control of resources such as processes, file systems, and IP addresses and other resources in a Linux-based cluster.
For more information please refer to: http://www.ibm.com/software/tivoli/products/sys-auto-linux/
IBM Tivoli System Automation for Multiplatforms policy is based on resource information, groups of resources, and relationships. Contrary to other automation products, scripts and procedures are not required. Its design is derived from zSeries System automation and technically, it is based on Reliable Scalable Cluster Technology (RSCT) from AIX. IBM TSA is available for Linux on xSeries, zSeries, pSeries, and iSeries and can be used to achieve high availability in a cross-platform cluster environment. Clusters are homogenous in their platform, but with the End-to-End Component a cross cluster automation is possible. Currently, with Version 2.1.1 the supported Linux distributions are RHEL 3.0, RHEL 4.0, SLES 8 and SLES 9.
For the most recent compatibility information please refer to: http://www.ibm.com/software/tivoli/products/sys-auto-linux/platforms.html
Our test scenario is based on a white paper "Highly Available DB2 Universal Database using Tivoli System Automation for Linux" written by Enrico Joedecke, Steve Raspudic, and Andy Beaton [1].
1 ftp://ftp.software.ibm.com/software/data/pubs/papers/halinux.pdf
More documentation and a user guide can be found at:
http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms2.1.html
We created our example scenario using the following steps.
Step 1: Storage attachment
In our test environment, we use two LUNs on an IBM FAStT700. One LUN is used as a target volume for DB2, and the other is a small LUN without any data on it; it is required only to enable tiebreaker functionality in a two-node cluster.
If one node loses all communication to the other and a cluster split happens, the decision about which side is going to stay available needs to be made. This decision is made based on the ability of the node to set a SCSI reservation bit on the storage LUN.
We have two nodes in our test cluster (p630-lpar2 and p630-lpar3), as shown in Figure 8-1. Both can access and mount the same LUN from the FAStT, and on both nodes this storage is represented as /dev/sdd.
We will not mount the file system concurrently because we are not using a cluster file system like IBM General Parallel File System (GPFS), and this can result in a broken file system. We can see both devices from both LPARs in Example 8-1.
- Example: Creating a file system
Now, we create a mount point on both LPARs (/home_db2), and mount our file system. If the mount is successful, we can add the following line to /etc/fstab:
Important: Do not mount this file system automatically at reboot.
Step 2: Installing DB2 on both nodes
We need to install DB2 on both nodes, although we do not need /home_db2 to be mounted at this point. We needed only to start the graphical DB2 installer, and follow all the default steps except that we choose not to create a database instance. The instance will be created later on manually.
DB2 will be installed in /opt/IBM/db2/V8.1/. In your case, if you have a recent version already installed, you will find a directory /opt/IBM/db2/V8.1/ha/salinux and all the Tivoli System Automation scripts in it.
If you do not have this directory, you can download the scripts from the following site, and then select Sample Policies and scripts to make DB2 high available.
http://www.ibm.com/software/tivoli/products/sys-auto-linux/downloads.html
Step 3: Creating users and a DB2 instance
First, we need to mount /dev/sdd under the mountpoint /home_db2 and then we create users and groups needed for DB2 operation on both nodes, as shown in Example:
- Example 3 Creating users and groups on both nodes
After issuing all the commands on the first node, we umount /home and mount it on the second node. Then we repeat the groupadd and useradd commands on the second node, as shown in Example 8-3.
After the users and the groups are created on both nodes, we can create a DB2 instance as shown in Example 8-4. We only need to do this task once on one of the nodes. In your case, remember to check that /home_db2 is mounted on the node.
- Example 4 Creating the DB2 instance
Step 4: Exchanging SSH keys between the nodes
For security reasons, we decided to use ssh and not rsh. In order to enable password less logins to both nodes from each other, we need to exchange ssh keys between them. We use ssh-keygen to generate the keys, and place them into the /root/.ssh/authorized_keys2 file on both nodes.
First let's create the keys without passphrase in this example.
p630-lpar2:~ #/usr/bin/ssh-keygen -f /root/.ssh/id_dsa -q -t dsa -N ""
p630-lpar2:~ #/usr/bin/ssh-keygen -f /root/.ssh/id_rsa -q -t rsa -N ""
Now add those keys to /root/.ssh/authorized_keys2.
p630-lpar2:~ # cat /root/.ssh/id_dsa.pub >> /root/.ssh/authorized_keys2
p630-lpar2:~ # cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys2
Repeat the above steps on the second node of the cluster!
Finally we are exchanging or merging the keys of both nodes.
On Node1:
p630-lpar2:~ # ssh p630-lpar3 cat /root/.ssh/authorized_keys2 >> /root/.ssh/authorized_keys2
{code:none}
On Node2:
{code:none}
p630-lpar3:~ # ssh p630-lpar2 cat /root/.ssh/authorized_keys2 >> /root/.ssh/authorized_keys2
{code:none}
Now you should be able to invoke commands from one node on the other node without beeing prompted for a password.
{code:none}
p630-lpar2:~ # ssh p630-lpar3 uname -a
Linux p630-lpar3 2.4.21-111-pseries64 #1 SMP Thu Oct 30 23:57:04 UTC 2003 ppc64 unknown
 | Please note...
On the first connection to the remote note ssh will prompt you to accept the key because it is not yet know. This can sometimes be a problem in scripts when they are not using a non-interactive approach. It is usefull to invoke ssh for all interfaces and hostnames (short and fully qualified) to get all the required keys and to avoid to be prompted for a password. You should do this on both nodes.
p630-lpar2:~ # ssh -o StrictHostKeyChecking=no p630-lpar3 date
p630-lpar2:~ # ssh -o StrictHostKeyChecking=no p630-lpar3.munich.de.ibm.com date |
Most of the TSA scripts assume we are working with rsh and not with ssh.
Thanks to the compatible syntax of ssh and rsh commands, we can use a trick instead of editing the scripts: we just move /usr/bin/rsh and create a symbolic link from /usr/bin/ssh to /usr/bin/rsh, as shown in Example 8-5.
- Example5 Replacing rsh with ssh
We need to repeat this command on both nodes.
| Please note...
With TSA 2.1.1.0 you'll have the option to use ssh for the setup of TSA and DB2 - see below |
Step 5: Installing TSA
After unpacking the tar archive which includes TSA (e.g. sam-1.2.0.0-Linux.tar C899KML.tar) change to the subdirectory created during the extraction of the packet (e.g. SAM2110Base).
| Please note...
TSA Version 2.1 comes with a prerequisite checker called prereqSAM. You can run this checker before you start the installation. |
In the subdirectory (e.g. SAM2110Base) you'll find the installation script called installSAM. Run this script and it will install all necessary filesets required for TSA.
| Please note...
Problems could occur if you're running RHEL4 and you've installed the rsct and src filesets which comes with the service and productivity tools (from http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/home.html).
You must uninstall these packages first and the run the {installSAM}} script again!
You may need to run ''rpm -ev --allmatches --nodeps'' options in order to clean it up.
On two SLES9-SP3 LPARs running the latest available version of lopdiags (as of 20.Sept. 2006) and using TSA 2.1.1 there were no problems detected concerning the rsct and src filesets. |
 | Tip!
You can use installSAM with the option --slient - this enables an installation of TSA during an automated installation script (and you don't have to "read" the whole license stuff). |
The installation must be done on both nodes.
Step 6: Setting up the RSCT cluster (RPD domain)
As mentioned before, Tivoli System Automation relies on RSCT technology for internode communication, heartbeat, and messaging. So we need to set up the underlying rpdomain. Before setting up a domain, we define a CT Management
scope using the following command:
| Please note...
The CT_MANAGEMENT_SCOPE variable determines the management scope that is used for the session with the RMC daemon to monitor and control the resources and resource classes.The management scope determines the set of possible target nodes where the resources and resource classes can be monitored and controlled. The valid values are:
0 Specifies local scope.
1 Specifies local scope.
2 Specifies peer domain scope.
If this environment variable is not set, local scope is used. |
Tip: Add the line: ''export CT_MANAGEMENT_SCOPE=2'' to /etc/profile.local (for SLES9) or /etc/profile or $HOME/.bashrc on both nodes.
First we prepare the nodes to join the domain:
Repeat the above step on the second node!
Then we create a domain (only on one node):
Finally, we start the domain:
We can check if the domain is established with the lsrpdomain and lsrpnode commands after a minute or so:
If you receive a message with the state: ''pending online'', simply wait a few more minutes.
Step 7: Defining a tiebreaker
Each two-node cluster needs a tiebreaker definition. Tiebreaker ensures that TSA can decide which node still owns the cluster if communication between the nodes is lost. In our case, it prevents the database from being started on both nodes simultaneously.
We use a LUN on the FAStT as a tiebreaker disk, and this LUN is seen as /dev/sde from both nodes. For tiebreaker definition, we need to know the four-integer definition of our disk, not just its logical device name.
On our LPARs, we have three internal discs and two LUNs on the FAStT visible, as shown in Example 8-6.
- Example6 Listing scsi devices
In the output of cat /proc/scsi/scsi, we can see all devices attached but we do not see their symbolic names. We use the ''dmesg | grep scsi'' command, as shown in Example 8-7.
- Example 7 Using dmesg to determine the tiebreaker disk
Our designated tiebreaker-disk is /dev/sde at host scsi1, channel 0, id 0 lun 1.
This disk can be minimum in size, and it does not need to be formatted. Tivoli will use a SCSI reservation bit in order to make sure this disk is only owned by one LPAR.
Now, we create a TSA resource with a tiebreaker definition, as shown in Example 8-8. After creation you must tell TSA to use it - the default TieBreaker is "Operator" which means that there'll be no automatic failover and the cluster will wait for you (the operator) to break the tie!
- Example 8 Defining a tiebreaker
| Please note...
Please note that it is absolutely mandatory to use the correct spelling expected by TSA - otherwise the command will fail.
Here's an example of the error message from the mkrsrc command used with the option Heart*B*eatPeriod instead of Heart*b*eatPeriod:
mkrsrc: Do not specify attribute HeartBeatPeriod when you define the resource IBM.TieBreaker row: 0 |
We can check our settings with the command shown in Example 8-9.
- Example 9 Listing tiebreaker settings
Step 8: Creating a high availability DB2 instance and resources
Assuming all the previous steps have been completed without errors, we are ready to run the TSA script provided for DB2. We change to the directory
/opt/IBM/db2/V8.1/ha/salinux and run the regdb2salin script:
| Tip!
For DB2 V9.1 the directory is /opt/ibm/db2/V9.1/ha/tsa.
And the really good thing is that the freely available DB2 Express-C version includes those scripts, too! |
| Tip!
To use ssh directly without tweaking use the option -s:
regdb2salin -s -a db2inst1 -m /home_db2 -i 9.154.2.123 |
We only need to run this script on one of the nodes. The script creates a highly available DB2 instance, protects its home directory (/home_db2), and assigns a cluster IP address (9.156.175.30). We do need to provide the names of the nodes because they are known through the RPD domain definition.
If the script produces no errors the cluster status can be checked, as shown in Example 8-10.
- Example 10 Checking TSA cluster status
From the getstatus command output we can see that the regdb2salin script created a resource group named db2_db2inst1_0-rg. This resource group contains the DB2 instance resource db2_db2inst1_0-rs, the mount resource
db2_db2inst1_0-rs_mount and the cluster IP address resource db2_db2inst1_0-rs_ip.
The DB2 instance resource depends on the mount and IP address. This can be seen by displaying the resource relationships using the lsres command, as shown in Example 11.
- Example 11 Displaying resource relations
Step 9: Testing the cluster
The command used to check the cluster state is the getstatus script introduced in Example 8-10 on page 370.
Before restarting the active node, we use a soft method to initiate takeover by excluding the active node from the automation. For example, in order to add node p630-lpar3 to the list of excluded nodes, we use the following command:
To include the node again (delete it from the list of excluded nodes) we use:
Refer to the ''man samctrl'' command for more information.
After we exclude the active node, we use the command
''/opt/IBM/db2/V8.1/ha/salinux/getstatus'' to see the resources starting on the
other node.
Step 10: Installing and starting the graphical user interface
| Info
The chapter about the grafical user interface is out dated. It os more advisable to use the WebSphere based "Operations Console" which comes as part of the product. |
Tivoli System Automation does not requires a GUI for installation and administration. Nevertheless, using a GUI is very useful for a quick visualization of the cluster setup and its state. For example, basic actions such as excluding and including nodes from the cluster can be initiated through the GUI. The GUI contains two parts: the server and the client. The server needs to be installed on the cluster nodes. The client can be installed on any remote machine running Linux on pSeries, Linux on xSeries, or Microsoft Windows.
The pSeries SAM GUI server and client are both provided in one RPM package.
The package can be installed by running the following command:
After the RPM installation the GUI server must be started, as shown in Example 8-12.
- Example 12 Starting the GUI server
Now we can start the client on the same machine with the command:
Another alternative is to install the GUI on any other machine and start it there.
We need to provide the IP address of the node we want to connect to, as well as the user name and the password. The default user name is rmcuser, and the password is rmc4all. For security reasons, the user name and password should
be changed as soon as possible. Example 8-13 shows how to change the user name.
- Example 13 Creating rmcuser
Example 14 shows how to change the password.
- Example 14 Changing rmcuser password
The GUI offers a great overview of our cluster setup, as shown in Figure 8-2. It also allows you to exclude one of the nodes from the automation in order to migrate the application to other nodes. This can be done for maintenance tasks.
TSA SAM GUI PICTURE MISSING !!!
