Configuring an active-active IBM Spectrum Scale deployment

You can deploy a IBM Spectrum Scale Primary instance and attach a deployed IBM Spectrum Scale Mirror instance and a IBM Spectrum Scale Tiebreaker instance to create a highly available active-active IBM Spectrum Scale configuration.

About this task

Active-active is a high availability (HA) configuration. In this configuration, if one side goes down, the other one takes over right away, with no data loss for the clients. It is possible to have a timeout for the clients that attempt to connect to the file system during this time, depending on the IBM Spectrum Scale server cluster failureDetectionTime and leaseRecoveryWait parameters, and if one of the primary, secondary server, cluster manager, or the file system manager are on the side that goes down. The timeout can be up to failureDetectionTime + leaseRecoveryWait. To find out the values for these parameters, run this command from the command line for any of the IBM Spectrum Scale server nodes:
su - gpfsprod -c '/usr/lpp/mmfs/bin/mmlsconfig leaseRecoveryWait'
su - gpfsprod -c '/usr/lpp/mmfs/bin/mmlsconfig failureDetectionTime'
Run the Get Cluster Status operation to list the values for these properties.
Warning: The minimum allowed value for leaseRecoveryWait and failureDetectionTime is 10 seconds, and the default is 35 seconds. Update these properties with caution as lower values could result in data corruption when nodes in the cluster are down. Contact the IBM Spectrum Scale service team and read the IBM Spectrum Scale documentation to understand the implications before you update these values.
To avoid this timeout, when one side goes down in a planned failover, before you bring it down, move the primary, secondary server, cluster manager, and the file system manager to the side that remains active. For information about the operations to make these cluster changes, see Failover active-active operations. If you have a IBM Spectrum Scale server instance that was deployed with IBM Spectrum Scale Pattern 1.2.15.0 or older, run this command from the command line for any of the IBM Spectrum Scale server nodes to update the cluster and file system manager:
Note: With V1.2.16.0, the command is managed automatically. You need not exclusively run the command. When one node goes down, the other node manager is automatically updated to become the cluster manager and the file system manager.

su - gpfsprod -c 'sudo /usr/lpp/mmfs/bin/mmchmgr -c nodeIP' to make the nodeIP the cluster manager.

su - gpfsprod -c 'sudo /usr/lpp/mmfs/bin/mmchmgr fileSystemName nodeIP' to make the nodeIP manager for fileSystemName.

In an active-active configuration, the replication is synchronous and is managed by IBM Spectrum Scale by using an active Mirror replica. This configuration is different from the active-passive configuration where replication is not managed by IBM Spectrum Scale, it is done by using volume replication. For this reason, a Primary instance cannot be part of both an active-active and an active-passive configuration at the same time. The Primary instance can be either:
  • Part of an active-active configuration and use IBM Spectrum Scale synchronous replication, with an active Mirror replica, or
  • Part of an active-passive replication, and use volume replication, with a IBM Spectrum Scale Passive side ready for takeover.

The following procedure shows the general steps you can take to configure an active-active IBM Spectrum Scale deployment for high availability. In this configuration, a IBM Spectrum Scale Primary instance is deployed to one rack (which is referred to as the Primary rack), while a IBM Spectrum Scale Mirror instance and IBM Spectrum Scale Tiebreaker instance are each deployed on separate racks (referred to as the Mirror rack and Tiebreaker rack), preferably at separate locations. Generally, this type of configuration is supported only for geographic distances less than 300 km apart to avoid latency problems with data transfer.

Important: As a best practice toward ensuring high availability, your Primary, Mirror, and Tiebreaker configurations should be on separate systems in your environment, such that if any one IBM Spectrum Scale configuration becomes unavailable, the other two are unaffected and can continue to operate.

If you have three systems with IBM® Cloud Pak products installed (either Cloud Pak Software or Cloud Pak System Software) in different data centers, or in different zones of a single data center, you should deploy these three IBM Spectrum Scale Server configurations each in a separate system.

If you have two systems with IBM Cloud Pak products installed, instead of deploying a IBM Spectrum Scale Tiebreaker configuration, you can install an external Tiebreaker node on a separate system and attach that separate node to your IBM Spectrum Scale cluster. For more information about setting up this external Tiebreaker node instead, see the Related links.

If you choose to deploy a IBM Spectrum Scale Tiebreaker configuration, and if you are limited to two systems with IBM Cloud Pak products installed, you can locate the IBM Spectrum Scale Tiebreaker configuration on the same system with either the Primary or Mirror configuration. For highest availability, assign each configuration to separate cloud groups which do not share compute nodes. With this setup, the Tiebreaker node does not reside on the same compute node as the Primary or Mirror node, and failure of any one compute node cannot bring down two IBM Spectrum Scale instances at once.

As a further consideration, when locating the IBM Spectrum Scale Tiebreaker configuration on the same system with either the Primary or Mirror configuration, if one system with IBM Cloud Pak products installed is larger than the other, you might choose to locate the IBM Spectrum Scale Primary and Tiebreaker configurations on your largest system, and the IBM Spectrum Scale Mirror configuration on the smaller system.

Procedure

  1. On the Primary system, create appropriate volumes as needed.
  2. On the Mirror system, create identical (in number and size) volumes as on the Primary rack.
  3. On the tiebreaker system, attach a volume, which is used by the tiebreaker configuration to store IBM Spectrum Scale metadata.
    You need to attach one volume per file system. Since the tiebreaker volume holds IBM Spectrum Scale metadata only, 1 GB is sufficient for the size of the volume used by the tiebreaker.
    Note: You can reuse volumes when you deploy the primary, mirror, or tiebreaker configurations. However, the existing volumes must not be partitioned and must have been used by IBM Spectrum Scale pattern instances only. Note that any data from the reused volumes is lost because IBM Spectrum Scale reformats the volumes when they are attached to the IBM Spectrum Scale nodes.
  4. Deploy a IBM Spectrum Scale Mirror instance on the Mirror rack. On the Placement dialog box, attach the volumes from the mirror rack.
    Ensure that the following requirements are met:
    • This Mirror instance has the same number of IBM Spectrum Scale nodes as the Primary configuration to which is attached.
    • The same number of volumes, with the same sizes, are used in both the Mirror and Primary deployments for the specified file system.
    • The file system name on the Mirror deployment must be the same as the file system name on the Primary deployment.

    Wait for the virtual machines to be allocated and note the IP address of its IBM Spectrum® Scale_manager node.

  5. Deploy a IBM Spectrum Scale Tiebreaker instance on the Tiebreaker rack. On the Placement dialog box, attach the volume.
    Ensure that the file system name on the Tiebreaker deployment is the same as the file system name on the Primary deployment. Wait for the virtual machines to be allocated and note the IP address of its IBM Spectrum Scale_manager node.

    Instead of using the IBM Spectrum Scale Tiebreaker pattern to deploy a Tiebreaker instance on Cloud Pak System, you can choose to deploy an External Tiebreaker on an external virtual machine. For more information about setting up the external Tiebreaker node and verifying installation of the tie node and disk, see Related concepts.

  6. Wait for these two virtual applications to finish starting. When the processing completes, a status of Running is displayed for each.
  7. Deploy a IBM Spectrum Scale Primary instance on the Primary rack.
    As you deploy your Primary instance, you can configure and attach the Mirror and Tiebreaker instances during deployment, or you can deploy the Primary instance by itself now, and attach the Mirror and Tiebreaker instances later.
    • If you choose to configure and attach the deployed Mirror and Tiebreaker instances during this Primary deployment, expand the Active Configuration section and supply the IP addresses for the IBM Spectrum Scale_manager nodes for both the Mirror and Tiebreaker instances.
    • If you use an external tiebreaker node, you cannot attach the tiebreaker instances during the primary deployment. You first deploy the primary instance then install the external tiebreaker node by using the binary files that are retrieved from the primary deployment. After that, you can attach the tiebreaker node to the primary configuration.
  8. Proceed to the Placement dialog box and attach appropriate volumes.
  9. Start the deployment process.
    When the processing completes, a status of Running is displayed.
  10. If you deployed the Primary instance by itself, then from the Primary system, run the Add New Member operation and specify the IP address of the IBM Spectrum Scale_manager node for the Mirror instance to attach it to the Primary instance.
    Then, repeat this operation, specifying the IP address of the IBM Spectrum Scale_manager node for the Tiebreaker instance to attach it to the Primary instance.

Results

This procedure attaches the Mirror and Tiebreaker configurations to the Primary configuration, creating the active-active environment.

What to do next

If you have not already done so, you can deploy an instance of the IBM shared service for IBM Spectrum Scale and configure it to use your new cluster. As an alternative to using the IBM Spectrum Scale shared service, starting with IBM Spectrum Scale pattern V1.2.5.0, the clients can directly provide the IBM Spectrum Scale server connection information at deployment time.

Deploy other workload patterns that contain the IBM Spectrum Scale Client Policy (or alternatively, the IBM Spectrum Scale Client script packages). These workloads can then access the volumes made available by your IBM Spectrum Scale cluster.

When failover situations occur, depending on the nature of the problem you might be able to recover in several ways. Generally, if either the Primary or Mirror configuration fails, you might be able to recover the IBM Spectrum Scale cluster. If the Primary or Mirror configuration fails and the tiebreak configuration fails, the IBM Spectrum Scale cluster ceases to function.

If you lose the primary configuration, or the mirror and tiebreak configurations, and the failure is recoverable:
  • You can run the Become Single Rack operation on the surviving Primary or Mirror configuration.
  • After you fix the problem on the failing configuration, you can run the Become Replicated Cluster operation.
If a single primary, mirror, or tiebreak configuration goes down and the failure is not recoverable, you need to remove the failed instance from the cluster and re-create a new configuration:
  • You can run the Remove Member operation on the surviving instance and point to the failing instance to remove it from the cluster. For example, if the failed instance is a mirror instance, run the Remove Member operation, and then select the Mirror option. If you need to remove the primary configuration, run the Remove Member operation, and then select the Primary option. Wait for the operation to finish successfully. Run the Get Cluster Status operation to ensure that the nodes and disks for the failed instance are no longer part of the cluster.
  • If the instance removed in the previous step is still running, delete it so you can reuse the attached volumes.
  • Deploy a new instance, with the same type as the one removed in the previous step. If possible, reattach the same volumes that are used by the deleted instance. If these volumes are not available, that is, the volumes are not healthy or your new instance is deployed in a cloud group or rack different than the deleted instance, you can use new volumes.
  • Add the new instance to the surviving primary or mirror instance by using the Add Member operation.
  • If more than one file systems are available on the surviving instance, add volumes to all these file systems on the new deployment.
    Note: When you expand the cluster with new nodes, data on the primary volumes is preserved when you attach a mirror or tiebreak instance to the primary instance. Data is also preserved when you remove a member from an existing cluster, if at least one of the mirror or primary instances are running. There is no outage for existing clients when the cluster is expanded. There might be a temporary outage when a member is removed from the cluster until the operation is finished.

If the cluster member's servers and disks are stopped (due to maintenance or manual intervention), you can restart the disks and servers in that member by running the Recover Lost Connection operation.