Red Hat Cluster on IBM Power System (PowerLinux)
mhhaque 2700012HF4 Comments (11) Visits (3370)
We have configured two node red hat HA cluster on IBM Power System. But there is no official support on Red Hat High Availability Add-On for IBM Power from Red Hat, Inc.
Collect All Information for Implementation:
The following information required before configure and implement your Red Hat HA Cluster.
Note: Here we use only one network for our two node red hat enterprise Linux 6.4 demo HA cluster installation
We have created two LPAR on IBM p740 System and Install Red Hat Enterprise Linux 6.4 on LPAR system. After that we perform below pre configuration on
Operating System for all target system.
Host Name & IP Address Configuration:
We configure Hostname, /etc/hosts file and IP configuration on both system as per the given information.
Stop Unnecessary Startup Services:
We stop all unnecessary services. Below script will help to do it in a single shot.
# for i in auditd bluetooth cups ip6tables iscsi iscsid \
mdmonitor postfix NetworkManager rpcbind rpcgssd rpcidmapd
chkconfig $i off
service $i stop
Enabling IP Ports:
Before deploying the Red Hat High Availability Add-On, you must enable certain IP ports on the both cluster nodes and on computers that run luci (the Conga user interface server).
We run the below iptable command on both node to identify the IP ports to be enabled:
# iptables -I INPUT -m state --state NEW -m multiport -p udp -s 192.168.70.0/24 -d 192.168.70.0/24 --dports 5404,5405 -j ACCEPT
# iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 192.168.70.0/24 --dports 5404,5405 -j ACCEPT
# iptables -I INPUT -m state --state NEW -p tcp -s 192.168.70.0/24 -d 192.168.70.0/24 --dport 21064 -j ACCEPT
# iptables -I INPUT -m state --state NEW -p tcp -s 192.168.70.0/24 -d 192.168.70.0/24 --dport 11111 -j ACCEPT
# iptables -I INPUT -m state --state NEW -p tcp -s 192.168.70.0/24 -d 192.168.70.0/24 --dport 16851 -j ACCEPT
# iptables -I INPUT -m state --state NEW -p tcp -s 192.168.70.0/24 -d 192.168.70.0/24 --dport 8084 -j ACCEPT
# service iptables save ; service iptables restart
Run below command to check current selinux status & change the selinux setting as required.
# setenforce 0
Here we use rpm command to install all Red Hat High Availability Add-On software packages, that we have compiled from source rpm. Run below command to verify the installed packages as required.
# rpm -q corosync corosynclib openais openaislib clusterlib\
modcluster ricci fence-agents fence-agents-lpar fence-agents-common cman\
rgmanager gfs2-utils luci
Red Hat High Availability Cluster Configuration:
Configuring Red Hat High Availability Add-On software consists of using configuration tools to specify the relationship among the cluster components.
The following cluster configuration tools are available with Red Hat High Availability Add-On:
Conga — this is a comprehensive user interface for installing, configuring, and managing Red Hat High Availability Add-On.
ccs — this command configures and manages Red Hat High Availability Add-On.
Command-line tools — this is a set of command-line tools for configuring and managing Red Hat High Availability Add-On.
Considerations for ricci:
Red Hat Enterprise Linux 6, ricci replaces ccsd. Therefore, it is necessary that ricci is running in each cluster node to be able to propagate updated cluster configuration whether it is via the cm an_tool version -r command, the ccs command, or the luci user interface server.
You can start ricci by using service ricci start or by enabling it to start at boot time via chkconfig.
For the Red Hat Enterprise Linux 6.1 release and later, using ricci requires a password the first time you propagate updated cluster configuration from any particular node. You set the ricci password as root after you install ricci on your system with the passwd ricci command, for user ricci.
Configure Red Hat High Availability Cluster Using Conga:
Creating a cluster with luci consists of naming a cluster, adding cluster nodes to the cluster, entering the ricci passwords for each node, and submitting the request to create a cluster. If the node information and passwords are correct, Conga automatically installs software into the cluster nodes (if the appropriate software packages are not currently installed) and starts the cluster.
To administer Red Hat High Availability Add-On with Conga, install and run luci as follows:
1. Select a computer to host luci and install the luci software on that computer. For example:
2. Start luci using service luci start. For example:
[root@node1 ~]# chkconfig luci on
[root@node1 ~]# service luci start
Adding following auto-detected host IDs (IP addresses/domain names), corresponding to `node1.example.com' address, to the configuration of self-managed certificate `/va
(none suitable found, you can still do it manually as mentioned above)
Generating a 2048 bit RSA private key
writing new private key to '/va
Point your web browser to http
3. At a Web browser, place the URL of the luci server into the URL address box and click Go (or the equivalent).
Step 1: Define a Cluster:
Define the cluster name & input all the information for the two node cluster.
Step 2: Define the fence device:
Define the fence device & input all the information for the fence device like: login, password, IP address etc.
Step 3: Assign hosts to fence device ports:
Define the power port for each server
Step 4: Define Failover Domains
Define failover domain and set node prioritized as well as failback policy as per your requirement.
Step 5: Define Resources for Clustered Web Service
Define Shared Storage (if not in fstab)
Define service IP address
Define application control script (stop/start/status) for Ftp Server Resource
Step 6: Define Clustered Web Service
Add storage resource (if not in fstab)
Add ip address resource
Add Ftp Server script resource
The Final Red Hat High Availability Cluster Configuration File:
The sample final red hat HA cluster configuration file (/ec
We can check the cluster status using below command from an of the node:
Red Hat High Availability Cluster Administration:
How we can start cluster services:
cman & rgmanager are the primary service for red hat HA cluster in RHEL 5 & RHEL 6.
To ensure cman & rgmanager services will automatically restart during system bootup:
How we relocate or move cluster services from primary node to standby (Secondary) node:
To disable cluster resources group:
Note: Disable activity will stop the service and place into the disabled state. This is the only permissible operation when a service is in the failed state.
To enable cluster resources group:
Note: This is the only permissible operation when a service is in the Disable state.
To freeze & unfreeze cluster resources group:
Note: Freeze services will allow users to upgrade rgmanager, CMAN, or any other software on the system while minimizing down-time of rgmanager-managed services.
Red Hat High Availability Cluster Troubleshooting:
How to check or verify fencing communication working fine:
# fence_node node2.example.com
If we get success output of that command as like above screenshot , then fencing working properly. we can also check with the log file ( /var/log/messages)
We can also check or verify the fence agent program manually, as follows:
To check cluster multicast communication:
multicast packet loss or multicast communication problem is one of the primary reason for the abnormal behavior of Red hat HA Cluster.
Use tcpdump command:
# tcpdump -n multicast -i eth0
# tcpdump -n dst 126.96.36.199 and udp src port 5404 -i eth0
Use netstat command:
To check cluster communication UDP & TCP port & status:
Modify cluster configuration using ccs command line tools:
add ftp application script resource (ftpsrv) into the existing cluster configuration file:
add ftp application script resource in the service group (SRV):
Propagate the current configuration changes among the all node in the cluster.
Verify the update cluster configuration version.
Note: When you going to use text editor to modify the existing cluster configuration file, you have to change the version of the configuration file at the top of that file.
Please feel free to contact with me, If there any query on that.