Deploying and testing a model

After you create or import, train, and save a model, you can deploy the model, test the deployment, and run the scoring service for predictions.

Before you begin

If you run your ML for IBM z/OS® on z/OS V2.5 or higher and plan to use the micro-batching inference and on-chip AI acceleration capabilities for online scoring of your ONNX models or snapML models, apply the PTFs that install the IBM® Z Deep Neural Network Library (zDNN), as described in Installing prerequisite hardware and software for ML for IBM z/OS.

Procedure

  1. Sign in to the ML for IBM z/OS user interface at https://<yourMLzUI-URL> with your username and password, where <yourWMLzUI-URL> is the host IP address and port number for your MLz UI.

    See Configuring MLz for more information about the host IP address and port number of your MLz.

  2. Go to the Models tab and select the model (for example, tempModel) that you want to deploy.
  3. From the Action menu for tempModel, select Deploy.
  4. On the Create deployment page, enter a deployment name, such as tempModelDeploy, and select a deployment type.
  5. On the Create deployment page, enter a deployment name, such as tempModelDeploy, and select a deployment type. By default, the deployment will result in an auto generated Deployment ID. This Deployment ID will be used for inference requests and any interactions with the deployment itself, for example, update deployment and delete deployment.
    Optionally, you can specify a Serving ID during deployment. All inference requests associated with this deployment will use the Serving ID instead of the Deployment ID. This approach allows multiple models to be deployed across various scoring servers—whether clustered or standalone—across multiple LPARs, and for each deployment to specify the same Serving ID. This eliminates the need for look up of the auto generated Deployment ID and alterations of applications as they progress from a test to the production environment.

    A Serving ID:

    • Must start with a letter
    • Must be 3–36 characters long
    • Can contain only alphanumeric characters and underscores (_)
    • Spaces and special characters are not allowed
    Note:

    When a Serving ID is provided, the Deployment ID will continue to be auto generated. The Deployment ID will be required for any deployment related information retrieval, update, or deletion activities.

  6. Select a version of tempModel that you want to deploy.
  7. Select a standalone or cluster scoring service on which you want to deploy the version.
  8. Optionally, decide if you want to use the on-chip AI accelerator for your ONNX models.

    Check the Use on-chip AI accelerator box if you decide to use this option. On-chip AI acceleration may not be available to all scoring servers or model versions that you select. AI acceleration requires the zDNN library on your z/OS system and the Telum processor on your IBM Z system. To use the AI accelerator, you must select a scoring server that runs on z16™ or z17 and a version of a model that is imported with zDNN available on your system.

  9. Optionally, decide if you want to enable micro-batching inference for your ONNX models.

    Check the Enable micro-batching inference box if you want to make the online scoring of your ONNX models more efficient. Instead of processing one record at a time, the MLz ONNX scoring engine can queue up scoring requests so that more records can be inferenced together. This capability, together with the on-chip AI accelerator of z16 or z17™ significantly improves the throughput of your ONNX model scoring.

    The micro-batching inference option is available only when you select a model that has at least one dimension for dynamic input. If you enable the option, specify the maximum batch size, which is the maximum number of scoring requests that constitutes a scoring batch. The valid range is 1-2147483647, and the default is 8. Also, specify the maximum latency in milliseconds for the processing of each scoring batch. The valid range is 1-3600000, and the default is 5ms.

  10. Optionally, decide if you want to specify a scoring timeout value for your deployment.

    Machine Learning for IBM z/OS delivers exceptional throughput and performance for inferencing requests. However, in rare scenarios, an inferencing request may take longer than expected. To handle this, use the scoring timeout option to cancel the long-running inferencing requests automatically.

    This option is available only for PMML and SnapML models with online scoring. This option defines the maximum time a scoring request can run against the deployment before it is canceled.
    • Valid range: 1–60,000 milliseconds. For example, "timeout": 60000
    • Default value: not specified
      Note: By default, if the scoring timeout duration is not specified, scoring requests will run until completion. The timeout is not applicable to online batch scoring (JES).
    Once the timeout value is set, the specified timeout value is uniformly applied to all incoming online scoring requests for the deployment.
    • If a scoring request completes within the defined timeout duration, it returns a response as expected.
    • If a scoring request exceeds the defined timeout duration, it is canceled and ALNB435E: Scoring request timed out error is returned.
    Important:

    Set a reasonable timeout value. A low timeout might reject most incoming inferencing requests, causing real-time online scoring failures.

    Timeout introduces a minor performance overhead. Enable it only when necessary.

  11. Click Create to proceed with the new deployment.

    You can deploy the same version of the model on a different scoring service, or you can deploy a different version of the model on the same scoring service.

    If Dual Control is enabled, initiating a deployment request does not immediately deploy the model. Instead, the request appears in a Pending approval state. To proceed, a system administrator, different from the requester, must approve the action by following these steps:

    1. Navigate to the Deployments tab and locate the pending request.
    2. From the Action menu, select Open Request for each item that require approval.
    3. In the approval dialog that appears, review the request details and choose Approve if appropriate. Once approved, the deployment becomes available for use.
    Note: All approvals are logged in the MLz audit trace for tracking.
  12. On the Deployments tab, verify that the model is deployed and that tempModelDeploy is listed.

    If necessary, edit the deployment on the Deployments page.

    1. From the Action menu for tempModelDeploy, select by selecting Update.
    2. On the Update deployment page, edit the name, model version, and scoring service of the deployment. If needed, enable or disable the micro-batching inference option, the on-chip AI acceleration option, or both.
    3. Optionally, edit the scoring timeout value if needed. You can remove an existing value, update an existing value, or add a new value if it does not already exist.
    4. Click Update to save your changes.
    Note: If Dual Control is enabled, updates to the deployment request does not immediately reflect in the deployment. Instead, the request appears in a Pending approval state. Once the approver approves the request, the deployment will be updated.
  13. Test the model by selecting Test API from the Action menu for the deployment.
  14. On the Test API page, enter values for a record.
    For example, you might enter the values of different fields in table or JSON format for the tempModel model.
  15. Click Submit.

    If you click Clear, repeat the last step.