Deploying ZAssistantDeploy on your cluster

This section describes the steps for deploying ZAssistantDeploy, including the setup of preconfigured services.

About this task

ZAssistantDeploy comes with a range of preconfigured services, including OpenSearch, authorization, z/OS Topology, and content ingestion.

OpenSearch

OpenSearch enables you to search, analyze, and visualize large volumes of data in real time.
  1. Determine a password for the OpenSearch instance deployed by the operator.
  2. Add the following content to the os-secret.yaml file:
    apiVersion: v1
    kind: Secret
    metadata:
      name: opensearch-creds
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      password: <OPENSEARCH_PASSWORD>
    

    Make sure to update the OpenSearch password.
  3. Create a secret by using the following command:
    oc apply -f os-secret.yaml

Content ingestion

Customers can ingest their content into the search database to improve the AI Assistant’s ability to retrieve information from their uploaded documents, in addition to leveraging IBM documentation.
  1. Determine an authentication key for the client ingestion server deployed by the operator.
  2. Add the following content to the wxa4z-ingestion-credentials.yaml file.
    apiVersion: v1
    kind: Secret
    metadata:
      name: wxa4z-ingestion-credentials
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      NATS_URL: "nats://wxa4z-nats.wxa4z-zad.svc.cluster.local:4222"
      NATS_PUBLIC_KEY: ""
      NATS_USER_SEED_KEY: ""
      NATS_JWT_TOKEN: ""
      NATS_SIGNING_SEED: ""
      NATS_OPERATOR_FINGERPRINT: ""
      INGESTION_API_KEY: ""
    

  3. Create a secret by using the following command:
    oc apply -f wxa4z-ingestion-credentials.yaml
  4. Generate NATS credentials for the content ingestion service.
    Ensure that you execute this script on an x86 or ARM64 machine.
    #!/bin/bash
    
    # Install NSC if not available
    nsc_bin=""
    if command -v nsc &> /dev/null; then
      nsc_bin="nsc"
    else
      os_name=$(uname -s | tr '[:upper:]' '[:lower:]')
      arch_name=$(uname -m)
      case "$arch_name" in
          x86_64)  arch_name="amd64" ;;
          aarch64|arm64) arch_name="arm64" ;;
      esac
      curl -sL "https://github.com/nats-io/nsc/releases/download/v2.8.6/nsc-${os_name}-${arch_name}.zip" -o /tmp/nsc.zip
      (cd /tmp && unzip -o nsc.zip nsc >/dev/null 2>&1)
      chmod +x /tmp/nsc
      nsc_bin="/tmp/nsc"
    fi
    
    # Set up isolated NSC environment
    nsc_home="/tmp/nsc-data-$$"
    nkeys_path="/tmp/nkeys-$$"
    export NSC_HOME="$nsc_home" NKEYS_PATH="$nkeys_path"
    export XDG_DATA_HOME="/tmp/.local/share-$$" XDG_CONFIG_HOME="/tmp/.config-$$"
    mkdir -p "$nsc_home" "$nkeys_path" "$XDG_DATA_HOME" "$XDG_CONFIG_HOME"
    
    # Generate operator and SYS account
    $nsc_bin add operator wxa4z-operator >/dev/null 2>&1
    $nsc_bin add account SYS >/dev/null 2>&1
    $nsc_bin edit operator --system-account SYS >/dev/null 2>&1
    
    # Extract JWTs
    operator_jwt=$($nsc_bin describe operator -R 2>/dev/null)
    sys_jwt=$($nsc_bin describe account SYS -R 2>/dev/null)
    
    # Helper: extract 'sub' field from a JWT (header.payload.signature)
    _jwt_sub() {
      local jwt="$1"
      local payload
      payload=$(echo "$jwt" | cut -d. -f2)
      # Add base64 padding
      local mod=$(( ${#payload} % 4 ))
      if [ "$mod" -eq 2 ]; then payload="${payload}=="; fi
      if [ "$mod" -eq 3 ]; then payload="${payload}="; fi
      echo "$payload" | base64 -d 2>/dev/null | grep -o '"sub":"[^"]*"' | cut -d'"' -f4
    }
    
    # Extract keys from JWTs
    operator_key=$(_jwt_sub "$operator_jwt")
    sys_key=$(_jwt_sub "$sys_jwt")
    
    # Find seeds by key in nkeys directory
    operator_seed=$(find "$nkeys_path" -name "${operator_key}.nk" -exec cat {} \; 2>/dev/null)
    sys_seed=$(find "$nkeys_path" -name "${sys_key}.nk" -exec cat {} \; 2>/dev/null)
    
    # Validate
    if [ -z "$operator_jwt" ] || [ -z "$operator_seed" ] || [ -z "$sys_jwt" ] || [ -z "$sys_seed" ]; then
      echo "Failed to generate NATS credentials" && exit 1
      rm -rf "$nsc_home" "$nkeys_path" "$XDG_DATA_HOME" "$XDG_CONFIG_HOME"
      return 1
    fi
    
    cat << EOF
    
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-sys-account-credentials
      namespace: wxa4z-zad
    stringData:
      sys-account.jwt: $sys_jwt
      sys-account.seed: $sys_seed
    type: Opaque
    ---
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-operator-credentials
      namespace: wxa4z-zad
    stringData:
      operator.jwt: $operator_jwt
      operator.seed: $operator_seed
    type: Opaque
    
    EOF
    
    # Cleanup
    rm -rf "$nsc_home" "$nkeys_path" "$XDG_DATA_HOME" "$XDG_CONFIG_HOME" /tmp/nsc /tmp/nsc.zip
    
  5. Save the script in a file named create_nats_credentials.sh, then execute the following commands to run it:
    chmod +x create_nats_credentials.sh
    ./create_nats_credentials.sh
    
    After running the preceding command, the script should generate YAML output containing two Secrets: nats-sys-account-credentials and nats-operator-credentials. For example:
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-sys-account-credentials
      namespace: wxa4z-zad
    stringData:
      sys-account.jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMNDYzTVlTTlc3RTZHNFlHUk80NE5CNzVNWkQ2UDJSUE5aMzRGN1REUkZWRE5DVlFSWkRRIiwiaWF0IjoxNzgyMjQ5NjE2LCJpc3MiOiJPQVhTWkI0NFU3Sk1OM1RVUllJSUZOSVpIUUJCSjRUNVhWNzRFV0c2N1ZLRDVGTFE2NFhZMzVBSSIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBQkZPUTZKQ0NLVUdNUVhXVzNQMlg2VlRWTElLNFVaM0ZFQUk3Vk9GTU1aN0c2UEtNSTRUSDJBUiIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJhdXRob3JpemF0aW9uIjp7fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.O_IbHc_0KmxwqyjsP1zrA3RugHLmrTRKW8v5ZHiBVPW4FeDaD7z1FoUnFqV4NWgz2RDgqBz0jX2AM63-D7BUCA
      sys-account.seed: SAAM7H67PUYQUBQUO6GOA3JKPZODAPX5FXZNR32MK2VVR2N46JPSXLFGNE
    type: Opaque
    ---
    kind: Secret
    apiVersion: v1
    metadata:
      name: nats-operator-credentials
      namespace: wxa4z-zad
    stringData:
      operator.jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiI2UVRRVTZGQ0RJNEdJV1VNRVVMVjNGWUZQWk9HV0VUR1RDQzZVN1FFUVlKNEkzU0wzUFRBIiwiaWF0IjoxNzgyMjQ5NjE2LCJpc3MiOiJPQVhTWkI0NFU3Sk1OM1RVUllJSUZOSVpIUUJCSjRUNVhWNzRFV0c2N1ZLRDVGTFE2NFhZMzVBSSIsIm5hbWUiOiJ3eGE0ei1vcGVyYXRvciIsInN1YiI6Ik9BWFNaQjQ0VTdKTU4zVFVSWUlJRk5JWkhRQkJKNFQ1WFY3NEVXRzY3VktENUZMUTY0WFkzNUFJIiwibmF0cyI6eyJzeXN0ZW1fYWNjb3VudCI6IkFCRk9RNkpDQ0tVR01RWFdXM1AyWDZWVFZMSUs0VVozRkVBSTdWT0ZNTVo3RzZQS01JNFRIMkFSIiwidHlwZSI6Im9wZXJhdG9yIiwidmVyc2lvbiI6Mn19.Lx-dy5Wo23hZgRGOVeSvTuWuCuV_TWo61Zek8MRWphSQ2B_61ep0bcdReCiT8GLDZ6eONndVpLYeoCbZuyxlDw
      operator.seed: SOAC2VRBBMNO4GKXD2JPX6LONVIZDHQCZPC7J2IYHPQL2ALLA2V7CLV6VI
    type: Opaque
    
  6. Save the YAML generated in Step 5 to a file named nats-credentials.yaml, then create the secret by running the following command:
    oc apply -f nats-credentials.yaml

Authorization Service

The Authorization Service ensures secure access by validating the current login user in watsonx Orchestrate against the z/OS User Management system.
  • A user logs into and interacts with the Z AI agent.
  • watsonx Orchestrate sends the user’s email ID to the Z AI agent , which forwards it to the Authorization Service.
  • The Authorization Service queries the z/OS User Management system to verify the user’s permissions.
  • The z/OS system responds, either granting or denying access based on the user's credentials.
You can use your own Certificate Authority (CA) signed certificates for mutual TLS (mTLS) by following a simple process, which involves:
  • Creating a Certificate Signing Request (CSR)
  • Getting it signed by your internal CA
  • Importing the signed certificate and your CA’s public key into the system

Alternatively, you can use the default IBM Internal CA-signed certificates, which are available for immediate use.

  1. Create a file named wxa4z-authorization-secrets.yaml with the following content, and update the agent authsecret token, STS_SECRET, and P12_PASSWORD values accordingly:
    Note: Provide only the values that are relevant to the agent you are deploying.
    apiVersion: v1
    kind: Secret
    metadata:
      name: wxa4z-authorization-secrets
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      AUTH_SERVICE_BASE_URL: http://wxa4z-authorization-service.wxa4z-zad.svc.cluster.local:8080
      ZOWE_ENABLED: "false"
      CICS_AGENT_TOKEN: <agent authsecret token>
      DB2_AGENT_TOKEN: <agent authsecret token>
      IMS_AGENT_TOKEN: <agent authsecret token>
      UPGRADE_AGENT_TOKEN: <agent authsecret token>
      SUPPORT_AGENT_TOKEN: <agent authsecret token>
      INTELLIMAGIC_AGENT_TOKEN: <agent authsecret token>
      OMEGAMON_AGENT_TOKEN: <agent authsecret token>
      STS_SECRET: <Security Token Service. secret>
      P12_PASSWORD: <password for the Authorization service PKCS#12 (.p12) certificate file>
      PLATFORM_AGENT_SECRET: <determine a value for the platform agent secret>
      JWT_SIGNING_KEY: <determine a value for the authorization service's JWT signing key>
  2. Create a secret by using the following command:
    oc apply -f wxa4z-authorization-secrets.yaml
  3. Run the following command to generate the base64‑encoded value from the .p12 file:
    openssl base64 -in cert.p12 -A

    This .p12 file must include both the client certificate for the Authorization service and the CA certificate required by the Token Exchange service.

  4. Add the following content to the wxa4z-authorization-ca.yaml file to import the P12 certificate for the Authorization service:
    apiVersion: v1
    kind: Secret
    metadata:
      name: wxa4z-authorization-ca
      namespace: wxa4z-zad
    type: Opaque
    data:
      cert.p12: <CERT_P12>

    Update the value of CERT_P12 with the base64‑encoded content of the Authorization service .p12 certificate.

  5. Create a secret by using the following command:
    oc apply -f wxa4z-authorization-ca.yaml

z/OS Topology Service

Serves as a centralized repository for storing and managing the topology of z/OS systems.
Remember: The topology service is disabled by default. To enable it, create the required secret as specified in steps 1 and 2 below, and add the zosTopology section to the deploy-zad.yaml, as described in the Deploy ZAssistantDeploy section.
  1. Add the following content to the wxa4z-zos-topology-secrets.yaml file to configure the credentials for the z/OS Topology service :
    apiVersion: v1
    kind: Secret
    metadata:
      name: wxa4z-zos-topology-secrets       
      namespace: wxa4z-zad
    type: Opaque
    stringData:
      ZOSMF_USERNAME: <ZOSMF_USERNAME>
      ZOSMF_PASSWORD: <ZOSMF_PASSWORD>
      OMEGAMON_TOKEN: <OMEGAMON_TOKEN>
      REDIS_PASSWORD: <REDIS_PASSWORD>

    Update the following fields:
    • ZOSMF_USERNAME: Enter the z/OSMF API username.
    • ZOSMF_PASSWORD: Enter the z/OSMF API password.
    • OMEGAMON_TOKEN: Enter the token from Creating the AIOps integration server token.
    • REDIS_PASSWORD: (Optional) Enter the Redis authentication password.
  2. Create a secret by using the following command:
    oc apply -f wxa4z-zos-topology-secrets.yaml

Wrapper and Software Hub

Create a secret for Wrapper and Software Hub.
  1. Determine a username and password for the search wrapper.
  2. Add the following content to the wrapper-secret.yaml file:
    apiVersion: v1
    stringData:
     username: <wrapper username>
     password: <wrapper password>
    kind: Secret
    metadata: 
     name: wrapper-creds
     namespace: wxa4z-zad
    type: Opaque
  3. Create a secret by using the following command:
    oc apply -f wrapper-secret.yaml
  4. Create a secret for your Software Hub instance.
    1. Create a file named wxa4z-ifm-credentials.yaml and add the following content:
      apiVersion: v1
      kind: Secret
      metadata:
        name: wxa4z-ifm-credentials
        namespace: wxa4z-zad
      type: Opaque
      stringData:
        MODEL_RUNTIME: <openai_protocol|on-prem|cloud>
      
        # set these variables when MODEL_RUNTIME is "openai_protocol"
        LLM_BASE_URL: <llm base url>
        LLM_API_KEY: <llm api key>
      
        # set these variables when MODEL_RUNTIME is "on-prem" or "cloud"
        CPD_USERNAME: <"cpadmin" (on-prem) or "" (cloud)>
        WATSONX_URL: <Software Hub (on-prem) or SaaS wxai (cloud) URL>
        WATSONX_API_KEY: <Software Hub (on-prem) or IAM (cloud) API key>
        WATSONX_MODEL_ID: <model id>
        WATSONX_SPACE_ID: <Deployment space ID>
        WATSONX_PROJECT_ID: ""

      Update the following fields:
      • MODEL_RUNTIME: Set MODEL_RUNTIME to one of the following values:
        • openai_protocol: Use this option for OpenAI‑protocol‑based inferencing stacks (such as AI Optimizer), and configure the following fields:
          • LLM_BASE_URL: The AI inference URL.
          • LLM_API_KEY: The AI inference API key.
        • on-prem: Use this option for on‑premises watsonx Orchestrate or on‑premises watsonx.ai inferencing, and configure the following fields:
          • CPD_USERNAME: Set this to "cpadmin".
          • WATSONX_URL: Obtain the Software Hub URL by using the following command:
            echo https://$(oc -n cpd-instance-1 get route cpd -o jsonpath="{.spec.host}")
          • WATSONX_API_KEY: Provide the Software Hub API key obtained when creating a service instance of watsonx Orchestrate in Software Hub
          • WATSONX_MODEL_ID: Use ibm/granite-3-3-8b-instruct or meta-llama/llama-3-3-70b-instruct, depending on the model you have deployed.
          • WATSONX_SPACE_ID: Leave this field blank as lite_install is set to "true" in the watsonxaiifm Custom Resource in step 7 of the Configure watsonx.ai for Spyre section.
            Note: If you are using x86, you must provide the space ID. You can obtain the deployment space ID by following step 8 in the Configuring the Model Gateway section.
        • cloud: Use this option for a SaaS-based watsonx.ai inferencing, and configure the following fields:
          • CPD_USERNAME: Set this to an empty string ("")
          • WATSONX_URL: Specify the URL of your watsonx.ai SaaS instance (for example, https://us-south.ml.cloud.ibm.com).
          • WATSONX_API_KEY: Provide the IBM Cloud API key associated with your watsonx.ai SaaS instance.
          • WATSONX_MODEL_ID: Specify the model ID.
          • WATSONX_SPACE_ID: If space ID is available, specify a deployment space ID from your watsonx.ai SaaS instance; otherwise, leave this field blank.
          • WATSONX_PROJECT_ID: Leave this field blank if a deployment space ID is provided; otherwise, specify a project ID from your watsonx.ai SaaS instance.
    2. Create a secret by using the following command:
      oc apply -f wxa4z-ifm-credentials.yaml

Deploy ZAssistantDeploy

  1. Add the following content to the deploy-zad.yaml file:
    apiVersion: wxa4z.watsonx.ibm.com/v1
    kind: ZAssistantDeploy
    metadata:
      name: zassistantdeploy
      namespace: wxa4z-zad
    spec:
      license: true
    
      global:
        clusterDomain: "<CLUSTER_DOMAIN>"
    
      providerId: "provider"
      cpdNamespace: cpd-instance-1
    
      agentDeploymentGlobal:
        secrets:
          name: wxa4z-ingestion-credentials
    
      opensearch:
        secretName: opensearch-creds
        persistence:
          enabled: true
          storageClass: "ocs-storagecluster-cephfs"
          accessModes:
            - ReadWriteOnce
          size: 24Gi
    
      wrapper:
        secretName: wrapper-creds
        createRoute: true
    
      clientIngestion:
        enabled: true
        tenant:
          id: "provider"
          isProvider: "True"
        ingestionCredentials:
          secretName: "wxa4z-ingestion-credentials"
        createRoute: true
        env:
          AUTH_SERVICE_URL: "http://wxa4z-authorization-service.wxa4z-zad.svc.cluster.local:8080"
    
    
      authorization:
        secrets:
          createSecret: false
        env:
          PORT: ":8080"
          FILE_STORAGE_PATH: "/app/data"
          AGENTS: |-
            [
              {
                "id": "wxa4z:platform:agent",
                "name": "platform",
                "secret": "<ENCODED_PLATFORM_AGENT_SECRET>"
              }
            ]
          ALLOW_DOMAINS: '["ibm.com", "us.ibm.com"]'
          CPD_HOST: "<CPD_HOST>"
    
      bootstrapper:
        enabled: true
        config:
          logLevel: "INFO"
          namespacePrefix: "wxa4z-"
          createNamespace: true
          opensearchUrl: "https://wxa4z-opensearch-cluster.wxa4z-zad.svc.cluster.local:9200"
          imagePullSecret: "icr-io-pull-secret"
          imagePullSecrets: "ibm-stg-icr-io-pull-secret,na-artifactory-pull-secret"
          osCaCertSecret: "wxa4z-opensearch-tls-secret"
          wrapperTlsSecret: "wxa4z-opensearch-wrapper-tls-secret"
          contentIngestionEnabled: "false"
        secrets:
          opensearchUsername: "admin"
          opensearchPassword: "<OPENSEARCH_PASSWORD>"
          wrapperAuthUser: "admin"
          wrapperAuthPassword: "<WRAPPER_PASSWORD>"
    

    Update the following fields:
    • CLUSTER_DOMAIN: Extract the default cluster domain by using the following command:
      oc -n openshift-ingress-operator get ingresscontroller default -o jsonpath="{.status.domain}"
    • OMEGAMON_URL: Specify the URL of your OMEGAMON instance.
    • ZOSMF_BASE_URL: Specify the base URL for your z/OSMF instance.
    • ENCODED_PLATFORM_AGENT_SECRET: Specify the base64‑encoded value of PLATFORM_AGENT_SECRET defined in wxa4z-authorization-secrets. To obtain the base64‑encoded value from this secret, run the following command:
      oc -n wxa4z-zad get secret wxa4z-authorization-secrets -o jsonpath={.data.PLATFORM_AGENT_SECRET}
    • OPENSEARCH_PASSWORD: Specify the password created for the opensearch-creds Secret. To retrieve this value, run the following command:
      oc get secret opensearch-creds -o go-template='{{.data.password | base64decode}}'
    • WRAPPER_PASSWORD: Specify the password created for the wrapper-creds Secret. To retrieve this value, run the following command:
      oc get secret wrapper-creds -o go-template='{{.data.password | base64decode}}'
    • CPD_HOST: Specify the hostname for the Software Hub cpd route. To retrieve the hostname, run the following command:
      oc -n cpd-instance-1 get route cpd -o jsonpath={.spec.host}

      z/OS Topology Service

      To enable the z/OS Topology Service, add the following section to deploy-zad.yaml:
      zosTopology:
          enabled: true
          resources:
            requests:
              cpu: "100m"
              memory: "512Mi"
            limits:
              cpu: "500m"
              memory: "1Gi"
          env:
            IS_OMEGAMON_ENABLED: <"true"|"false">
            OMEGAMON_API_URL: <OMEGAMON_URL>
            ZOSMF_BASE_URL: <ZOSMF_BASE_URL>
          pvc:
            storageClass: "ocs-storagecluster-cephfs"
            enabled: true
            size: 8Gi
      

      Update the following fields:
      • OMEGAMON_URL: Specify the URL of your OMEGAMON instance.
      • ZOSMF_BASE_URL: Specify the base URL for your z/OSMF instance.
  2. Deploy the ZAssistantDeploy Service by using the following command:
    oc apply -f deploy-zad.yaml

Results

The deployment of the ZAssistantDeploy Service begins and might take up to 15 minutes or more for the images to download and the deployment to complete. However, you can check the status by running the following command:
oc get pods -n wxa4z-zad