Configuring a ML for IBM z/OS cluster for high availability

If your machine learning workload is large, mission critical, or both, you can configure your MLz core services for high availability. To achieve high availability of core services, consider configuring a MLz cluster. Each cluster can consist of two or more MLz instances that run either on a single LPAR or across different LPARs.

Before you begin

A MLz instance contains a set of core services for model training, deployment, batch scoring, ingestion, repository, and data connection management, and the core services are supported by an active runtime environment. The runtime environment can be provided by Spark, Python, or both. To make core services in a MLz cluster highly available means to keep one runtime environment active at all time.

  • Decide the type of MLz cluster you want to configure. As shown in the following diagram, you can configure a cluster with multiple MLz instances running on the same LPAR (Cluster type 1) or across different LPARs (Cluster type 2).
    Figure 1. MLz cluster

    Begin figure description. MLz cluster. End figure description
  • Set up the TCP SHAREPORT port or the sysplex distributor port to be used by the cluster. If your MLz cluster is type 1, enable the SHAREPORT. If your cluster is type 2, enable the sysplex distributor port.
  • Provision and install additional system capacity to support your MLz cluster.

    It is recommended that you plan and start each MLz instance in a cluster, regardless of the cluster type, with the basic system capacity as described in Planning system capacity for ML for IBM z/OS. You can adjust the basic capacity in terms of CPU, memory, or DASD over time based on your machine learning workload.

    If your cluster is type 1 and if your initial workload is small, you might be able to share the basic capacity of 1 GCP, 4 zIIPs, and 100 GB memory across multiple MLz instances on the same LPAR because only one runtime environment is active at any given time. You can increase the CPU and memory allocation as your workload increases. However, you must allocate and start with 100 GB DASD for each instance.

Procedure

  1. Complete the installation and configuration of the first MLz instance by completing all required tasks as described in Installation roadmap.
  2. Retrieve the following configuration information of your first MLz instance from the System Configuration page of the administration dashboard:
    • Keystore. All MLz instances in a cluster must use the same keystore for secure connections and user authentication. Make a note of the keyring name, the keyring owner, and the certificate label.
    • Metadata schema. All MLz instances in a cluster must use the same metadata schema. Write down the schema name as well as the metadata database, storage group, and buffer pool information.
    • Core services port. All MLz instances in a cluster must use the same cluster host IP and MLz core services port number. If the cluster is type 1 where all instances run on the same LPAR, write down the LPAR IP address and the SHAREPORT number. If the cluster is type 2 where the instances run across different LPARs, write down the sysplex IP address and the SD port number.
  3. Install and configure the second MLz instance into the cluster.

    Follow instructions in Configuring MLz to configure the new instance. When prompted, make sure that you specify the keystore, metadata schema, and core services port information that you collected in Step 2.

    • On the Authentication page, specify the keystore type and related information used in the first MLz instance. This ensures that all MLz instances in the cluster use the same keystore to secure connections and to authenticate users.
    • On the Metadata repository page, specify the same metadata schema name, database, storage group, and buffer pool used in the first MLz instance. This ensures that all instances in the cluster use the same metadata objects.
    • On the UI and core services page, specify the cluster host IP address and MLz core services port number. If the cluster is type 1 where all instances run on the same LPAR, specify the LPAR IP as the cluster host IP and the SHAREPORT number as the MLz core services port. If the cluster is type 2 where the instances run across different LPARs, specify the sysplex IP address as the cluster host IP and the SD port number as the core services port.
  4. Repeat Step 3 to install and configure any additional instance into the cluster.
  5. Complete the cluster setup and start the cluster.
    1. Repeat Steps 3 - 4 to complete the cluster setup.
    2. Verify that all MLz instances in the cluster are started and running.
    3. Verify that the runtime environment of one MLz instance is active.
  6. Configure the REST API of your machine learning application to call the host IP and core services port of your MLz cluster.

    For cluster type 1, the cluster host IP is the LPAR IP address and the core services port is the SHAREPORT number. For cluster type 2, the cluster host IP is the sysplex distributor IP address and the core services port is the sysplex distributor (SD) port number.

  7. In the event that the active MLz instance in your cluster is down, activate the runtime environment of another instance by using the administration dashboard.
    1. Sign into the administration dashboard.
    2. From the sidebar, navigate to the System management - Runtime Environments page.
    3. Select the runtime environment you want to activate and from the ACTIONS menu, click the Connection icon to connect and activate.
    4. Verify that the new runtime environment is active and the cluster is up and running.