Configuring HMC or PowerVM NovaLink to work with Resource Optimized High Availability

If you are running PowerHA® SystemMirror® Version 7.2.2 for AIX® or later, and if you want to use the Resource Optimized High Availability (ROHA) function, you must configure each HMC or PowerVM® NovaLink LPAR link to use Secure Shell (SSH). However, if you are running PowerHA SystemMirror Version 7.2.2 for AIX or later, you can also use Representational State Transfer (REST) application programming interface (API) to connect to the HMC instead of SSH. You must also configure a backup HMC. The ROHA function can be used only with HMC version 8.40, or later.

SSH communication with HMC or PowerVM NovaLink

For systems that are running PowerHA SystemMirror Version 7.2.2 for AIX or later, LPARs must use SSH to communicate with the Hardware Management Console (HMC) or PowerVM NovaLink.

You must configure SSH to not require a password when PowerHA SystemMirror is communicating with the HMC or PowerVM NovaLink. To configure SSH communication, you can run the ssh-keygen command on each LPAR node to generate a public and private key pair. The public key must be copied to the HMCs or PowerVM NovaLink public key file that is authorized. The following example displays how to set up SSH from the LPAR:


# /usr/bin/ssh-keygen –t rsa
Generating public/private rsa key pair.
Enter file in which to save key (//.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in //.ssh/id_rsa.
Your public key has been saved in //.ssh/id_rsa.pub.
The key fingerprint is:
9c:00:9f:61:d9:40:60:0c:1d:6b:89:ac:f9:8e:fc:f5 root@4ndc1
# mykey=`cat ~/.ssh/id_rsa.pub`
# ssh hmcuser@cuodhmc mkauthkeys –a \”$mykey\”

Verify that the SSH communication is configured correctly by running the ssh <HMCuser>@hmcname ls /tmp command on LPAR, where HMCuser is the login ID configured in the HMC and hmcname is the name of the HMC or PowerVM NovaLink.

Note: PowerHA SystemMirror Version 7.2.5 for AIX or later supports nonroot user to connect with the HMC by using SSH communications. PowerHA SystemMirror Version 7.2.4 for AIX, or earlier, only supports hscroot as HMC username.

To add nonroot user support, run the clmgr command.

To add a username, while adding the HMC definition to PowerHA SystemMirror configuration, enter the following command:

clmgr add hmc <HMC> USER_NAME=<hmcuser>

To add a username, if HMC is already added to the PowerHA SystemMirror configuration, enter the following command:

clmgr modify hmc <HMC> USER_NAME=<hmcuser>

REST API communication with HMC

In PowerHA SystemMirror Version 7.2.2 for AIX, or later, you can use SSH or a REST API to communicate securely with an HMC. The REST APIs are provided in HMC Version 8.40, or later, and are based on HTTP protocols. An HMC user should have privileges to run the query and to acquire, and release resource operations when a nonroot user is used.

You can configure the REST API by using one of the following methods:

SMIT

From the command line, enter smit sysmirror.
Select Cluster Applications and Resources > Resources > Configure User Applications (Scripts and Monitors) > Resource Optimized High Availability > HMC Configuration > Change/Show Default HMC Tunables, and press Enter.
Change Connection Type as REST API.

clmgr command

To specify the connection type, run the clmgr manage cluster hmc command.

Example: clmgr manage cluster hmc CONNECTION_TYPE=rest

To add the user name and password, enter the following command:

clmgr add hmc

To modify the user name and password, enter the following command:

clmgr modify hmc

To view help information about modifying the HMC parameters, enter the following command:

clmgr manage cluster hmc –h

HMC REST API support

You must use the HMC Version 8.4.3, HMC Version 8.5.3, HMC Version 8.6.2, or later.
You can download the curl package from the IBM® AIX Toolbox for Linux® Applications website. You also need to install ca-certificates during curl package installation. The ssl and crypto library versions that are shipped with the AIX® operation system must be compatible with the curl package. The following versions of AIX operating system are tested:
- AIX 7.2 with Technology Level 0 with SP 2
- AIX 7.2 Technology Level 1 with SP 1
- AIX 7.1 with Technology Level 4 with SP 4
Note: The port number that is used for HMC REST API is 12443
The latest versions of HMC supports both 12443 and 443 ports. Starting with PowerHA SystemMirror Version 7.2.8, or later, you can modify the HMC port by making following entry in the /etc/environment file.
```
HMC_PORT=<port number>
```
By default, the port 12443 is used if the port entry is not made in the /etc/environment file.

REST API communication with PowerVM NovaLink

In PowerHA SystemMirror Version 7.2.2, you cannot use REST API to communicate with the PowerVM NovaLink. Therefore, the PowerHA SystemMirror cannot verify the REST API communication.

Timeout and retry mechanism

If the HMC cannot get the CEC lock, the LPAR requests that to the HMC might fail. If a failure occurs between the LPAR and the HMC, PowerHA SystemMirror tries to re-establish a connection. You can configure the frequency of times PowerHA SystemMirror attempts to establish a connection by using one of the following methods:

SMIT

From the command line, enter smit sysmirror.
Select Cluster Applications and Resources > Resources > Configure User Applications (Scripts and Monitors) > Resource Optimized High Availability > HMC Configuration > Change/Show Default HMC Tunables, and press Enter.

clmgr command

To display the HMC parameters, run the clmgr query cluster roha command.

To view help information about modifying the HMC parameters, run the

clmgr manage
cluster hmc –h

command.

Note: HMC 8.2, or later, queues all HMC commands if the HMC cannot get the CEC lock. This function suppresses the requirement to retry and establish a connection. Therefore, the retry mechanism parameters are only used if the HMC is 8.2 or earlier.

If the PowerVM NovaLink is not reachable, the LPAR requests that to the PowerVM NovaLink mightPowerVM fail. If a failure occurs between the LPAR and the PowerVM NovaLink, PowerHA SystemMirror retries to establish a connection. You can configure the frequency of times PowerHA SystemMirror attempts to establish a connection by using one of the following methods:

SMIT

From the command line, enter smit sysmirror.
Select Cluster Applications and Resources > Resources > Configure User Applications (Scripts and Monitors) > Resource Optimized High Availability > NovaLink Configuration > Change/Show Default NovaLink Tunables, and press Enter.
Change Connection Type as REST API.x

clmgr command

To display the PowerVM NovaLink parameters, run the clmgr query cluster nova command.

To view help information about modifying the PowerVM NovaLink parameters, run the clmgr manage cluster nova –h command.

Configuring a backup HMC

You can configure more than one HMC. If one HMC fails to respond, the Resource Optimized High Availability (ROHA) function can switch to another HMC. The HMCs are used in the order they are listed in the HMC list. For example, in the HMC list, if you have three systems that are listed in the following order: HMC1, HMC2, and HMC3. If HMC1 fails, ROHA stops trying to communicate with HMC1 and starts to communicate with the next HMC in the list (HMC2). The currently used HMC is saved in the cluster configuration so that the ROHA function skips any failing HMCs and communicates with a working HMC. At the end of the session, the falling HMC information is cleared and HMC1 is listed as the first HMC system in the list again.

To change the order of HMCs in a list, use one of the following methods:

SMIT

From the command line, enter smit sysmirror.
Select Cluster Applications and Resources > Resources > Configure User Applications (Scripts and Monitors) > Resource Optimized High Availability > HMC Configuration > Change/Show Default HMC Tunables, and press Enter.

clmgr command

To display the HMC parameters, run the clmgr query cluster hmc command.

To view help information about modifying the HMC parameters run, the

clmgr manage
cluster hmc –h

command.

Primary HMC and backup HMC

In HMC Version 8.40, or earlier, PowerHA SystemMirror can perform an Enterprise Pool CoD (EPCoD) operation only on a primary HMC. An EPCoD can work with a pair of HMCs. One HMC is the primary and performs changes on the EPCoD, and the other HMC is the backup that can only send query requests to the EPCoD. If the primary HMC fails (does not respond or is not functioning), PowerHA SystemMirror uses another HMC. The new HMC can be considered as the new primary HMC only if this new HMC was previously designated to the EPCoD as a backup HMC. If the HMC was designated to the EPCoD as the backup HMC, PowerHA SystemMirror can transform this backup HMC to the primary HMC. If the HMC was not designated to the EPCoD as the backup HMC, PowerHA SystemMirror cannot transform this backup HMC to the primary HMC unless you provide an XML file that contains the EPCoD definition that was used to create the EPCoD pool. Therefore, it is important to save this XML file in a directory that can be accessed easily.

Note: The concept of a primary and backup HMC is applicable only for EPCoD pools and operations.

The HMC that you use to create the EPCoD pool is the primary HMC. When you create the primary HMC, you can create a second HMC and configure it as the backup HMC for the primary HMC. The second HMC can be a backup HMC only if it manages all of the servers that belong to the EPCoD pool.

If the primary HMC is online, you must use the primary HMC to set the backup HMC. You can set a backup HMC by running the chcodpool -p epcodpoolname -o update -a "backup_master_mc_name=backup_hmc" command, where backup_hmc is the name of the backup HMC.

In HMC Version 8.50, or later, all servers in a EPCoD pool are not required to be managed by the same HMC pair. When the HMC primary is defined, all EPCoD pool requests are routed through the primary HMC. All managing HMCs must have a network connection with the primary HMC. Requests that are initiated on the managing HMC are first sent to the primary HMC. If the primary HMC is not managing the EPCoD pool or if it is not connected to the target server, all requests are sent to an HMC that is managing that server. Servers that are managed by HMCs send events to the primary HMC. The primary HMC sends the EPCoD pool data to all other HMCs every time the EPCoD pool resources are updated.

Note: The backup primary HMCs are deprecated in HMC Version 8.50, or later.

In HMC Version 8.50, or later, if partitions exist at the HMC group level and if two primary HMC are active at the same time, the resources might move to an uncompliance state when resource acquisition occurs. After communication is restored between first primary HMC and the new primary HMC, the new primary HMC takes over as the HMC primary for the entire EPCoD pool. In this scenario, the first primary HMC is removed and all HMCs that are managing the EPCoD pool are notified. The resources that are assigned to the servers in the EPCoD pool might not be same as the resources identified by the new primary HMC. Therefore, you must resync all resources with the new primary HMC.

Note: Verify and synchronize the cluster after all changes have been made, use any of the following methods.

SMIT:

From the command line enter smit sysmirror.
Select Cluster Applications and Resources- > Verify and Synchronize Cluster configuration and press Enter.

clmgr command:

To verify and synchronize the cluster, enter the following command:

clmgr sync cluster