Deploying ZAssistantDeploy on your cluster

This section describes the steps for deploying ZAssistantDeploy, including the setup of preconfigured services.

About this task

ZAssistantDeploy comes with a range of preconfigured services, including OpenSearch, authorization, z/OS Topology, and content ingestion.

OpenSearch

OpenSearch enables you to search, analyze, and visualize large volumes of data in real time.
  1. Determine a password for the OpenSearch instance deployed by the operator.
  2. Add the following content to the os-secret.yaml file:
    apiVersion: v1
    kind: Secret
    metadata:
      name: opensearch-creds
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      password: <OPENSEARCH_PASSWORD>
    

    Make sure to update the OpenSearch password.
  3. Create a secret by using the following command:
    oc apply -f os-secret.yaml

Content ingestion

Customers can ingest their content into the search database to improve the AI Assistant’s ability to retrieve information from their uploaded documents, in addition to leveraging IBM documentation.
Note: The content ingestion password is now generated automatically. Any previously configured content ingestion password is no longer required and can be removed.
  1. Generate NATS credentials for the content ingestion service.
    Ensure that you execute this script on an x86 or ARM64 machine.
    #!/bin/bash
    
    # Install NSC if not available
    nsc_bin=""
    if command -v nsc &> /dev/null; then
      nsc_bin="nsc"
    else
      os_name=$(uname -s | tr '[:upper:]' '[:lower:]')
      arch_name=$(uname -m)
      case "$arch_name" in
          x86_64)  arch_name="amd64" ;;
          aarch64|arm64) arch_name="arm64" ;;
      esac
      curl -sL "https://github.com/nats-io/nsc/releases/download/v2.8.6/nsc-${os_name}-${arch_name}.zip" -o /tmp/nsc.zip
      (cd /tmp && unzip -o nsc.zip nsc >/dev/null 2>&1)
      chmod +x /tmp/nsc
      nsc_bin="/tmp/nsc"
    fi
    
    # Set up isolated NSC environment
    nsc_home="/tmp/nsc-data-$$"
    nkeys_path="/tmp/nkeys-$$"
    export NSC_HOME="$nsc_home" NKEYS_PATH="$nkeys_path"
    export XDG_DATA_HOME="/tmp/.local/share-$$" XDG_CONFIG_HOME="/tmp/.config-$$"
    mkdir -p "$nsc_home" "$nkeys_path" "$XDG_DATA_HOME" "$XDG_CONFIG_HOME"
    
    # Generate operator and SYS account
    $nsc_bin add operator wxa4z-operator >/dev/null 2>&1
    $nsc_bin add account SYS >/dev/null 2>&1
    $nsc_bin edit operator --system-account SYS >/dev/null 2>&1
    
    # Extract JWTs
    operator_jwt=$($nsc_bin describe operator -R 2>/dev/null)
    sys_jwt=$($nsc_bin describe account SYS -R 2>/dev/null)
    
    # Helper: extract 'sub' field from a JWT (header.payload.signature)
    _jwt_sub() {
      local jwt="$1"
      local payload
      payload=$(echo "$jwt" | cut -d. -f2)
      # Add base64 padding
      local mod=$(( ${#payload} % 4 ))
      if [ "$mod" -eq 2 ]; then payload="${payload}=="; fi
      if [ "$mod" -eq 3 ]; then payload="${payload}="; fi
      echo "$payload" | base64 -d 2>/dev/null | grep -o '"sub":"[^"]*"' | cut -d'"' -f4
    }
    
    # Extract keys from JWTs
    operator_key=$(_jwt_sub "$operator_jwt")
    sys_key=$(_jwt_sub "$sys_jwt")
    
    # Find seeds by key in nkeys directory
    operator_seed=$(find "$nkeys_path" -name "${operator_key}.nk" -exec cat {} \; 2>/dev/null)
    sys_seed=$(find "$nkeys_path" -name "${sys_key}.nk" -exec cat {} \; 2>/dev/null)
    
    # Validate
    if [ -z "$operator_jwt" ] || [ -z "$operator_seed" ] || [ -z "$sys_jwt" ] || [ -z "$sys_seed" ]; then
      echo "Failed to generate NATS credentials" && exit 1
      rm -rf "$nsc_home" "$nkeys_path" "$XDG_DATA_HOME" "$XDG_CONFIG_HOME"
      return 1
    fi
    
    cat << EOF
    
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-sys-account-credentials
      namespace: wxa4z-zad
    stringData:
      sys-account.jwt: $sys_jwt
      sys-account.seed: $sys_seed
    type: Opaque
    ---
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-operator-credentials
      namespace: wxa4z-zad
    stringData:
      operator.jwt: $operator_jwt
      operator.seed: $operator_seed
    type: Opaque
    
    EOF
    
    # Cleanup
    rm -rf "$nsc_home" "$nkeys_path" "$XDG_DATA_HOME" "$XDG_CONFIG_HOME" /tmp/nsc /tmp/nsc.zip
    
  2. Save the script in a file named create_nats_credentials.sh, then execute the following commands to run it:
    chmod +x create_nats_credentials.sh
    ./create_nats_credentials.sh
    
    After running the preceding command, the script should generate YAML output containing two Secrets: nats-sys-account-credentials and nats-operator-credentials. For example:
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-sys-account-credentials
      namespace: wxa4z-zad
    stringData:
      sys-account.jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMNDYzTVlTTlc3RTZHNFlHUk80NE5CNzVNWkQ2UDJSUE5aMzRGN1REUkZWRE5DVlFSWkRRIiwiaWF0IjoxNzgyMjQ5NjE2LCJpc3MiOiJPQVhTWkI0NFU3Sk1OM1RVUllJSUZOSVpIUUJCSjRUNVhWNzRFV0c2N1ZLRDVGTFE2NFhZMzVBSSIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBQkZPUTZKQ0NLVUdNUVhXVzNQMlg2VlRWTElLNFVaM0ZFQUk3Vk9GTU1aN0c2UEtNSTRUSDJBUiIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJhdXRob3JpemF0aW9uIjp7fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.O_IbHc_0KmxwqyjsP1zrA3RugHLmrTRKW8v5ZHiBVPW4FeDaD7z1FoUnFqV4NWgz2RDgqBz0jX2AM63-D7BUCA
      sys-account.seed: SAAM7H67PUYQUBQUO6GOA3JKPZODAPX5FXZNR32MK2VVR2N46JPSXLFGNE
    type: Opaque
    ---
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-operator-credentials
      namespace: wxa4z-zad
    stringData:
      operator.jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiI2UVRRVTZGQ0RJNEdJV1VNRVVMVjNGWUZQWk9HV0VUR1RDQzZVN1FFUVlKNEkzU0wzUFRBIiwiaWF0IjoxNzgyMjQ5NjE2LCJpc3MiOiJPQVhTWkI0NFU3Sk1OM1RVUllJSUZOSVpIUUJCSjRUNVhWNzRFV0c2N1ZLRDVGTFE2NFhZMzVBSSIsIm5hbWUiOiJ3eGE0ei1vcGVyYXRvciIsInN1YiI6Ik9BWFNaQjQ0VTdKTU4zVFVSWUlJRk5JWkhRQkJKNFQ1WFY3NEVXRzY3VktENUZMUTY0WFkzNUFJIiwibmF0cyI6eyJzeXN0ZW1fYWNjb3VudCI6IkFCRk9RNkpDQ0tVR01RWFdXM1AyWDZWVFZMSUs0VVozRkVBSTdWT0ZNTVo3RzZQS01JNFRIMkFSIiwidHlwZSI6Im9wZXJhdG9yIiwidmVyc2lvbiI6Mn19.Lx-dy5Wo23hZgRGOVeSvTuWuCuV_TWo61Zek8MRWphSQ2B_61ep0bcdReCiT8GLDZ6eONndVpLYeoCbZuyxlDw
      operator.seed: SOAC2VRBBMNO4GKXD2JPX6LONVIZDHQCZPC7J2IYHPQL2ALLA2V7CLV6VI
    type: Opaque
    
  3. Save the YAML generated in Step 5 to a file named nats-credentials.yaml, then create the secret by running the following command:
    oc apply -f nats-credentials.yaml

Authorization Service

The Authorization Service ensures secure access by validating the current login user in watsonx Orchestrate against the z/OS User Management system.
  • A user logs into and interacts with the Z AI agent.
  • watsonx Orchestrate sends the user’s email ID to the Z AI agent , which forwards it to the Authorization Service.
  • The Authorization Service queries the z/OS User Management system to verify the user’s permissions.
  • The z/OS system responds, either granting or denying access based on the user's credentials.
You can use your own Certificate Authority (CA) signed certificates for mutual TLS (mTLS) by following a simple process, which involves:
  • Creating a Certificate Signing Request (CSR)
  • Getting it signed by your internal CA
  • Importing the signed certificate and your CA’s public key into the system

Alternatively, you can use the default IBM Internal CA-signed certificates, which are available for immediate use.

  1. Create a file named wxa4z-authorization-secrets.yaml with the following content, and update the agent authsecret token, STS_SECRET, and P12_PASSWORD values accordingly:
    Note: Provide only the values that are relevant to the agent you are deploying.
    apiVersion: v1
    kind: Secret
    metadata:
      name: wxa4z-authorization-secrets
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      AUTH_SERVICE_BASE_URL: http://wxa4z-authorization-service.wxa4z-zad.svc.cluster.local:8080
      ZOWE_ENABLED: "false"
      CICS_AGENT_TOKEN: <agent authsecret token>
      DB2_AGENT_TOKEN: <agent authsecret token>
      IMS_AGENT_TOKEN: <agent authsecret token>
      UPGRADE_AGENT_TOKEN: <agent authsecret token>
      SUPPORT_AGENT_TOKEN: <agent authsecret token>
      INTELLIMAGIC_AGENT_TOKEN: <agent authsecret token>
      OMEGAMON_AGENT_TOKEN: <agent authsecret token>
      STS_SECRET: <Security Token Service. secret>
      P12_PASSWORD: <password for the Authorization service PKCS#12 (.p12) certificate file>
      PLATFORM_AGENT_SECRET: <determine a value for the platform agent secret>
      JWT_SIGNING_KEY: <determine a value for the authorization service's JWT signing key>
  2. Create a secret by using the following command:
    oc apply -f wxa4z-authorization-secrets.yaml
  3. Run the following command to generate the base64‑encoded value from the .p12 file:
    openssl base64 -in cert.p12 -A

    This .p12 file must include both the client certificate for the Authorization service and the CA certificate required by the Token Exchange service.

  4. Add the following content to the wxa4z-authorization-ca.yaml file to import the P12 certificate for the Authorization service:
    apiVersion: v1
    kind: Secret
    metadata:
      name: wxa4z-authorization-ca
      namespace: wxa4z-zad
    type: Opaque
    data:
      cert.p12: <CERT_P12>

    Update the value of CERT_P12 with the base64‑encoded content of the Authorization service .p12 certificate.

  5. Create a secret by using the following command:
    oc apply -f wxa4z-authorization-ca.yaml

z/OS Topology Service

Serves as a centralized repository for storing and managing the topology of z/OS systems.
Remember: The topology service is disabled by default. To enable it, create the required secret as specified in steps 1 and 2 below, and add the zosTopology section to the deploy-zad.yaml, as described in the Deploy ZAssistantDeploy section.
  1. Add the following content to the wxa4z-zos-topology-secrets.yaml file to configure the credentials for the z/OS Topology service :
    apiVersion: v1
    kind: Secret
    metadata:
      name: wxa4z-zos-topology-secrets       
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      ZOSMF_USERNAME: <ZOSMF_USERNAME>
      ZOSMF_PASSWORD: <ZOSMF_PASSWORD>
      OMEGAMON_TOKEN: <OMEGAMON_TOKEN>
      REDIS_PASSWORD: <REDIS_PASSWORD>

    Update the following fields:
    • ZOSMF_USERNAME: Enter the z/OSMF API username.
    • ZOSMF_PASSWORD: Enter the z/OSMF API password.
    • OMEGAMON_TOKEN: Enter the token from Creating the AIOps integration server token.
    • REDIS_PASSWORD: (Optional) Enter the Redis authentication password.
  2. Create a secret by using the following command:
    oc apply -f wxa4z-zos-topology-secrets.yaml

Wrapper and Software Hub

Create a secret for Wrapper and Software Hub.
  1. Determine a username and password for the search wrapper.
  2. Add the following content to the wrapper-secret.yaml file:
    apiVersion: v1
    stringData:
     username: <wrapper username>
     password: <wrapper password>
    kind: Secret
    metadata: 
     name: wrapper-creds
     namespace: wxa4z-zad
    type: Opaque
  3. Create a secret by using the following command:
    oc apply -f wrapper-secret.yaml
  4. Create a secret for your Software Hub instance.
    1. Create a file named wxa4z-ifm-credentials.yaml and add the following content:
      apiVersion: v1
      kind: Secret
      metadata:
        name: wxa4z-ifm-credentials
        namespace: wxa4z-zad
      type: Opaque
      stringData:
        MODEL_RUNTIME: <openai_protocol|on-prem|cloud>
      
        # set these variables when MODEL_RUNTIME is "openai_protocol"
        LLM_BASE_URL: ""
        LLM_API_KEY: ""
      
        # set these variables when MODEL_RUNTIME is "on-prem" or "cloud"
        CPD_USERNAME: <"cpadmin" (on-prem) or "" (cloud)>
        WATSONX_URL: <Software Hub (on-prem) or SaaS wxai (cloud) URL>
        WATSONX_API_KEY: <Software Hub (on-prem) or IAM (cloud) API key>
        WATSONX_MODEL_ID: <model id>
        WATSONX_SPACE_ID: <Deployment space ID>
        WATSONX_PROJECT_ID: ""

      Update the following fields:
      • MODEL_RUNTIME: Set MODEL_RUNTIME to one of the following values:
        • openai_protocol: Use this option for OpenAI‑protocol‑based inferencing stacks (such as AI Optimizer), and configure the following fields:
          • LLM_BASE_URL: AI inference endpoint URL.
          • LLM_API_KEY: API key for the AI inference endpoint.
          • Note: This configuration is optional and can be left blank for both on-premises and cloud deployments if an OpenAI-compatible inferencing stack is not being used.
        • on-prem: Use this option for on‑premises watsonx Orchestrate or on‑premises watsonx.ai inferencing, and configure the following fields:
          • CPD_USERNAME: Set this to "cpadmin".
          • WATSONX_URL: Obtain the Software Hub URL by using the following command:
            echo https://$(oc -n cpd-instance-1 get route cpd -o jsonpath="{.spec.host}")
          • WATSONX_API_KEY: Provide the Software Hub API key obtained when creating a service instance of watsonx Orchestrate in Software Hub
          • WATSONX_MODEL_ID: Use ibm/granite-4.1-8b-instruct or meta-llama/llama-3-3-70b-instruct, depending on the model you have deployed.
          • WATSONX_SPACE_ID: Leave this field blank as lite_install is set to "true" in the watsonxaiifm Custom Resource in step 7 of the Configure watsonx.ai for Spyre section.
            Note: If you are using x86, you must provide the space ID. You can obtain the deployment space ID by following step 8 in the Configuring the Model Gateway section.
        • cloud: Use this option for a SaaS-based watsonx.ai inferencing, and configure the following fields:
          • CPD_USERNAME: Set this to an empty string ("")
          • WATSONX_URL: Specify the URL of your watsonx.ai SaaS instance (for example, https://us-south.ml.cloud.ibm.com).
          • WATSONX_API_KEY: Provide the IBM Cloud API key associated with your watsonx.ai SaaS instance.
          • WATSONX_MODEL_ID: Specify the model ID.
          • WATSONX_SPACE_ID: If space ID is available, specify a deployment space ID from your watsonx.ai SaaS instance; otherwise, leave this field blank.
          • WATSONX_PROJECT_ID: Leave this field blank if a deployment space ID is provided; otherwise, specify a project ID from your watsonx.ai SaaS instance.
    2. Create a secret by using the following command:
      oc apply -f wxa4z-ifm-credentials.yaml

Knowledge Transformer Secret

Knowledge Transformer is an optional component, currently available as a technical preview, that transforms documents into AI-optimized taxonomies for improved agent retrieval and reasoning. When enabled, it provides a REST API that the zassist CLI uses to process documents. For more information, see Technical preview: Knowledge Transformer. If you plan to enable Knowledge Transformer, create an API key secret for authentication. This key will be used by the zassist CLI tool to authenticate requests to the transform service.
  1. Generate a secure API key for Knowledge Transformer:
    TRANSFORM_API_KEY=$(openssl rand -base64 32)
    echo "Save this API key securely: $TRANSFORM_API_KEY"
  2. Create a file named transform-secret.yaml with the following content:
    
    apiVersion: v1
    kind: Secret
    metadata:
      name: transform-secret
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      TRANSFORM_API_KEY: <TRANSFORM_API_KEY>
    Replace <TRANSFORM_API_KEY> with the API key generated in step 1.
  3. Create the secret:
    
    oc apply -f transform-secret.yaml
  4. Verify the secret was created:
    
    oc get secret transform-secret -n wxa4z-zad
Note: Save the TRANSFORM_API_KEY value securely. You will need it later to configure the zassist CLI tool (see Configuring Knowledge Transformer).

Deploy ZAssistantDeploy

  1. Add the following content to the deploy-zad.yaml file:
    apiVersion: wxa4z.watsonx.ibm.com/v1
    kind: ZAssistantDeploy
    metadata:
      name: zassistantdeploy
      namespace: wxa4z-zad
    spec:
      license: true
    
      global:
        clusterDomain: "<CLUSTER_DOMAIN>"
    
      providerId: "provider"
      cpdNamespace: cpd-instance-1
    
      agentDeploymentGlobal:
        secrets:
          name: wxa4z-ingestion-credentials
    
      opensearch:
        secretName: opensearch-creds
        persistence:
          enabled: true
          storageClass: "ocs-storagecluster-cephfs"
          accessModes:
            - ReadWriteOnce
          size: 24Gi
    
      wrapper:
        secretName: wrapper-creds
        createRoute: true
    
      clientIngestion:
        enabled: true
        tenant:
          id: "provider"
          isProvider: "True"
        ingestionCredentials:
          secretName: "wxa4z-ingestion-credentials"
        createRoute: true
        env:
          AUTH_SERVICE_URL: "http://wxa4z-authorization-service.wxa4z-zad.svc.cluster.local:8080"
    
    
      authorization:
        secrets:
          createSecret: false
        env:
          PORT: ":8080"
          FILE_STORAGE_PATH: "/app/data"
          AGENTS: |-
            [
              {
                "id": "wxa4z:platform:agent",
                "name": "platform",
                "secret": "<ENCODED_PLATFORM_AGENT_SECRET>"
              }
            ]
          ALLOW_DOMAINS: '["ibm.com", "us.ibm.com"]'
          CPD_HOST: "<CPD_HOST>"
    
      bootstrapper:
        enabled: true
        config:
          logLevel: "INFO"
          namespacePrefix: "wxa4z-"
          createNamespace: true
          opensearchUrl: "https://wxa4z-opensearch-cluster.wxa4z-zad.svc.cluster.local:9200"
          imagePullSecret: "icr-io-pull-secret"
          imagePullSecrets: "ibm-stg-icr-io-pull-secret,na-artifactory-pull-secret"
          osCaCertSecret: "wxa4z-opensearch-tls-secret"
          wrapperTlsSecret: "wxa4z-opensearch-wrapper-tls-secret"
          contentIngestionEnabled: "false"
        secrets:
          opensearchUsername: "admin"
          opensearchPassword: "<OPENSEARCH_PASSWORD>"
          wrapperAuthUser: "admin"
          wrapperAuthPassword: "<WRAPPER_PASSWORD>"
    

    Update the following fields:
    • CLUSTER_DOMAIN: Extract the default cluster domain by using the following command:
      oc -n openshift-ingress-operator get ingresscontroller default -o jsonpath="{.status.domain}"
    • OMEGAMON_URL: Specify the URL of your OMEGAMON instance.
    • ZOSMF_BASE_URL: Specify the base URL for your z/OSMF instance.
    • ENCODED_PLATFORM_AGENT_SECRET: Specify the base64‑encoded value of PLATFORM_AGENT_SECRET defined in wxa4z-authorization-secrets. To obtain the base64‑encoded value from this secret, run the following command:
      oc -n wxa4z-zad get secret wxa4z-authorization-secrets -o jsonpath={.data.PLATFORM_AGENT_SECRET}
    • OPENSEARCH_PASSWORD: Specify the password created for the opensearch-creds Secret. To retrieve this value, run the following command:
      oc -n wxa4z-zad get secret opensearch-creds -o go-template='{{.data.password | base64decode}}'
    • WRAPPER_PASSWORD: Specify the password created for the wrapper-creds Secret. To retrieve this value, run the following command:
      oc -n wxa4z-zad get secret wrapper-creds -o go-template='{{.data.password | base64decode}}'
    • CPD_HOST: Specify the hostname for the Software Hub cpd route. To retrieve the hostname, run the following command:
      oc -n cpd-instance-1 get route cpd -o jsonpath={.spec.host}

      z/OS Topology Service

      To enable the z/OS Topology Service, add the following section to deploy-zad.yaml:
      zosTopology:
          enabled: true
          resources:
            requests:
              cpu: "100m"
              memory: "512Mi"
            limits:
              cpu: "500m"
              memory: "1Gi"
          env:
            IS_OMEGAMON_ENABLED: <"true"|"false">
            OMEGAMON_API_URL: <OMEGAMON_URL>
            ZOSMF_BASE_URL: <ZOSMF_BASE_URL>
          pvc:
            storageClass: "ocs-storagecluster-cephfs"
            enabled: true
            size: 8Gi
      

      Update the following fields:
      • OMEGAMON_URL: Specify the URL of your OMEGAMON instance.
      • ZOSMF_BASE_URL: Specify the base URL for your z/OSMF instance.
  2. Deploy the ZAssistantDeploy Service by using the following command:
    oc apply -f deploy-zad.yaml

Results

The deployment of the ZAssistantDeploy Service begins and might take up to 15 minutes or more for the images to download and the deployment to complete. However, you can check the status by running the following command:
oc get pods -n wxa4z-zad