Configuring an active-active IBM Storage Scale deployment

You can deploy a IBM Storage Scale Primary instance and attach a deployed IBM Storage Scale Mirror instance and a IBM Storage Scale Tiebreaker instance to create a highly available active /active IBM Storage Scale configuration.

About this task

Active-active is a high availability (HA) configuration. In this configuration, if one side goes down, the other one takes over right away, with no data loss for the clients. It is possible to have a timeout for the clients that attempt to connect to the file system during this time, depending on the IBM Storage Scale server cluster failureDetectionTime and leaseRecoveryWait parameters, and if one of the primary, secondary server, cluster manager, or the file system manager are on the side that goes down. The timeout can be up to failureDetectionTime + leaseRecoveryWait. To find out the values for these parameters, run this command from the command line for any of the IBM Storage Scale server nodes:
su - gpfsprod -c '/usr/lpp/mmfs/bin/mmlsconfig leaseRecoveryWait'
su - gpfsprod -c '/usr/lpp/mmfs/bin/mmlsconfig failureDetectionTime'
If you are deploying a IBM Storage Scale server instance that uses IBM Storage Scale Pattern 1.2.15.0 or later, run the Get Cluster Status operation to list the values for these properties.
Warning: The minimum allowed value for leaseRecoveryWait and failureDetectionTime is 10 seconds, and the default is 35 seconds. Update these properties with caution as lower values could result in data corruption when nodes in the cluster are down. Contact the IBM Storage Scale service team and read the IBM Storage Scale documentation to understand the implications before you update these values.

To avoid this timeout, when one side goes down in a planned failover, before you bring it down, move the primary, secondary server, cluster manager, and the file system manager to the side that remains active. For information about the operations to make these cluster changes, see Failover active-active operations. If you have a IBM Storage Scale server instance that was deployed with IBM Storage Scale Pattern 1.2.7.0 or older, run this command from the command line for any of the IBM Storage Scale server nodes to update the cluster and file system manager:

su - gpfsprod -c 'sudo /usr/lpp/mmfs/bin/mmchmgr -c nodeIP' to make the nodeIP the cluster manager.

su - gpfsprod -c 'sudo /usr/lpp/mmfs/bin/mmchmgr fileSystemName nodeIP' to make the nodeIP manager for fileSystemName.

In an active-active configuration, the replication is synchronous and is managed by IBM Spectrum Scale by using an active Mirror replica. This configuration is different from the active-passive configuration where replication is not managed by IBM Storage Scale, it is done by using volume replication. For this reason, a Primary instance cannot be part of both an active-active and an active-passive configuration at the same time. The Primary instance can be either:
  • Part of an active-active configuration and use IBM Spectrum Scale synchronous replication, with an active Mirror replica, or
  • Part of an active-passive replication, and use volume replication, with a IBM Storage Scale Passive side ready for takeover.
The following procedure shows the general steps you can take to configure an active-active IBM Storage Scale deployment for high availability. In this configuration, a IBM Storage Scale Primary instance is deployed to one configuration (which is referred to as the Primary configuration), while a IBM Storage Scale Mirror instance and IBM Storage Scale Tiebreaker instance are each deployed on separate configurations (referred to as the Mirror configuration and Tiebreaker configuration), preferably at separate locations. Generally this type of configuration is supported only for geographic distances less than 300 km apart, to avoid latency problems with data transfer.
Important: As a best practice toward ensuring high availability, your Primary, Mirror, and Tiebreaker configurations should be located on separate systems in your environment, such that if any one IBM Storage Scale configuration becomes unavailable, the other two are unaffected and can continue to operate.

If you have three Cloud Pak System Software for Power® instances in different data centers, or in different zones of a single data center, you should deploy these three IBM Storage Scale Server configurations each in a separate system.

If you have two Cloud Pak System Software for Power instances, instead of deploying a IBM Storage Scale Tiebreaker configuration, you can install an external Tiebreaker node on a separate system and attach that separate node to your IBM Storage Scale cluster. For more information about setting up this external Tiebreaker node instead, see the Related links.

If you choose to deploy a IBM Storage Scale Tiebreaker configuration, and if you are limited to two Cloud Pak System Software for Power instances, you can locate the IBM Storage Scale Tiebreaker configuration on the same system with either the Primary or Mirror configuration. For highest availability, assign each configuration to separate cloud groups which do not share compute nodes. With this setup, the Tiebreaker node does not reside on the same compute node as the Primary or Mirror node, and failure of any one compute node cannot bring down two IBM Storage Scale instances at once.

As a further consideration, when locating the IBM Storage Scale Tiebreaker configuration on the same system with either the Primary or Mirror configuration, if one Cloud Pak System Software for Power instance is larger than the other, you might choose to locate the IBM Storage Scale Primary and Tiebreaker configurations on your largest system, and the IBM Storage Scale Mirror configuration on the smaller system.

Procedure

  1. On the Primary system, create appropriate volumes as needed.
  2. On the Mirror system, create identical (in number and size) volumes as on the Primary rack.
  3. On the tiebreaker system, attach a volume, which is used by the tiebreaker configuration to store IBM Storage Scale metadata.
    You need to attach one volume per file system. Since the tiebreaker volume holds IBM Spectrum Scale metadata only, 1 GB is sufficient for the size of the volume used by the tiebreaker.
    Note: You can reuse volumes when you deploy the primary, mirror, or tiebreaker configurations. However, the existing volumes must not be partitioned and must have been used by IBM Storage Scale pattern instances only. Note that any data from the reused volumes is lost because IBM Storage Scale reformats the volumes when they are attached to the IBM Storage Scale nodes.
  4. Deploy a IBM Storage Scale Mirror instance on the Mirror rack. On the Placement dialog box, attach the volumes from the mirror rack.
    Ensure that the following requirements are met:
    • This Mirror instance has the same number of IBM Storage Scale nodes as the Primary configuration to which it will be attached.
    • The same number of volumes, with the same sizes, are used in both the Mirror and Primary deployments for the specified file system.
    • The file system name on the Mirror deployment must be the same as the file system name on the Primary deployment.

    Wait for the virtual machines to be allocated and note the IP address of its IBM Storage Scale_manager node.

  5. Deploy a IBM Storage Scale Tiebreaker instance on the Tiebreaker rack. On the Placement dialog box, attach the volume.
    Ensure that the file system name on the Tiebreaker deployment is the same as the file system name on the Primary deployment. Wait for the virtual machines to be allocated and note the IP address of its IBM Storage Scale_manager node.

    Instead of using the IBM Storage Scale Tiebreaker pattern to deploy a Tiebreaker instance on Cloud Pak System Software, you can choose to deploy an External Tiebreaker on an external virtual machine. For more information about setting up the external Tiebreaker node and verifying installation of the tie node and disk, see Related concepts.

  6. Wait for these two virtual applications to finish launching. When the processing completes, a status of Running is displayed for each.
  7. Deploy a IBM Storage Scale Primary instance on the Primary configuration.
    As you deploy your Primary instance, you can configure and attach the Mirror and Tiebreaker instances during deployment, or you can deploy the Primary instance by itself now, and attach the Mirror and Tiebreaker instances later.
    • If you choose to configure and attach the deployed Mirror and Tiebreaker instances during this Primary deployment, expand the Active Configuration section and supply the IP addresses for the IBM Storage Scale_manager nodes for both the Mirror and Tiebreaker instances.
    • If you use an external tiebreaker node, you cannot attach the tiebreaker instances during the primary deployment. You first deploy the primary instance then install the external tiebreaker node by using the binary files that are retrieved from the primary deployment. After that, you can attach the tiebreaker node to the primary configuration.
  8. Proceed to the Placement dialog box and attach appropriate volumes.
  9. Start the deployment process.
    When the processing completes, a status of Running is displayed.
  10. If you deployed the Primary instance by itself, then from the Primary system run the Add New Member operation and specify the IP address of the IBM Storage Scale_manager node for the Mirror instance to attach it to the Primary instance.
    Then, repeat this operation, specifying the IP address of the IBM Storage Scale_manager node for the Tiebreaker instance to attach it to the Primary instance.

Results

This procedure attaches the Mirror and Tiebreaker configurations to the Primary configuration, creating the active-active environment.

What to do next

If you have not already done so, you can deploy an instance of the IBM® shared service for IBM Storage Scale and configure it to use your new cluster.As an alternative to using the IBM Storage Scale shared service, starting with IBM Storage Scale pattern V1.2.5.0, the clients can directly provide the IBM Storage Scale server connection information at deployment time.

Deploy other workload patterns that contain the IBM Storage Scale Client Policy (or alternatively, the IBM Storage Scale Client script packages). These workloads can then access the volumes made available by your IBM Storage Scale cluster.

When failover situations occur, depending on the nature of the problem you might be able to recover in several ways. Generally, if either the Primary or Mirror configuration fails, you might be able to recover the IBM Storage Scale cluster. If the Primary or Mirror configuration fails and the Tiebreak configuration fails, the IBM Storage Scale cluster ceases to function.

If you lose the primary configuration, or the mirror and tiebreak configurations, and the failure is recoverable:
  • You can run the Become Single Rack operation on the surviving Primary or Mirror configuration.
  • After fixing the problem on the failing configuration, you can run the Become Replicated Cluster operation.
If a single primary, mirror, or tiebreak configuration goes down and the failure is not recoverable, you need to remove the failed instance from the cluster and re-create a new configuration:
  • You can run the Remove Member operation on the surviving instance and point to the failing instance to remove it from the cluster. For example, if the failed instance is a mirror instance, run the Remove Member operation, and then select the Mirror option. If you need to remove the primary configuration, run the Remove Member operation, and then select the Primary option. Wait for the operation to finish successfully. Run the Get Cluster Status operation to ensure that the nodes and disks for the failed instance are no longer part of the cluster.
  • If the instance removed in the previous step is still running, delete it so you can reuse the attached volumes.
  • Deploy a new instance, with the same type as the one removed in the previous step. If possible, reattach the same volumes that are used by the deleted instance. If these volumes are not available, that is, the volumes are not healthy or your new instance is deployed in a cloud group or rack different than the deleted instance, you can use new volumes.
  • Add the new instance to the surviving primary or mirror instance by using the Add Member operation.
  • If more than one file systems are available on the surviving instance, add volumes to all these file systems on the new deployment.
    Note: When you expand the cluster with new nodes, data on the primary volumes is preserved when you attach a mirror or tiebreak instance to the primary instance. Data is also preserved when you remove a member from an existing cluster, if at least one of the mirror or primary instances are running. There is no outage for existing clients when the cluster is expanded. There might be a temporary outage when a member is removed from the cluster until the operation is finished.

If the cluster member's servers and disks are stopped (perhaps due to maintenance or manual intervention), you can restart the disks and servers in that member by running the Recover Lost Connection operation.