Installing IBM Model Gateway

If you intend to use large language models (LLM) with your IBM® Business Automation Workflow deployment, then it is recommended that you install and use IBM Model Gateway. The Model Gateway provides a unified interface for managing and routing requests to LLMs with built-in OpenTelemetry configuration. OpenTelemetry is the open standard for telemetry.

apiVersion: v1
kind: Secret
metadata:
  name: model-gateway-provider-secret
type: Opaque
data:
  openai-key: <base64-encoded-json>
  watsonx-key: <base64-encoded-json>

openai-key: eyJhcGlLZXkiOiAic2stKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqWlhZIiwidXJsIjogImh0dHBzOi8vYXBpLm9wZW5haS5jb20vdjEifQ==

{
  "apiKey": "sk-*********************************ZXY",
  "url": "https://api.openai.com/v1"
}

1. Create Kubernetes secrets in the operator and instance namespaces to pull the required container images.

   oc create secret docker-registry ibm-entitlement-key \
     --docker-server=cp.icr.io \
     --docker-username=cp \
     --docker-password=<your-entitlement-key> \
     -n <namespace>

2. Optional: Create a secret for your external PostgreSQL instance. If you do not want the script to create the secret from a property file, then create it manually.

   The Kubernetes secret name that is used by the deployment must be model-gateway-postgres-external-secret, and the required secret keys are host, port, username, password, dbname, and parameters.

   oc create secret generic model-gateway-postgres-external-secret \
     --from-literal=host=<postgres-host> \
     --from-literal=port=<postgres-port> \
     --from-literal=username=<db-username> \
     --from-literal=password=<db-password> \
     --from-literal=dbname=<database-name> \
     --from-literal=parameters='sslmode=require' \
     -n <instance-namespace>

   Note: The supported SSL modes are disable, require, verify-ca, and verify-full.

   If verify-ca or verify-full is used, the CA certificate must exist. If client certificate authentication is enabled, both the client certificate and the client key must exist.

3. Run the baw-model-gateway-deployment.sh script to verify all the prerequisites are installed.

   Run the following command from the scripts folder of your downloaded cert-kubernetes.

   ./baw-model-gateway-deployment.sh --verify-only

   The script checks the prerequisites:

   • Checks for required CLI tools (kubectl, helm, oc).
   • Verifies Kubernetes cluster connectivity.
   • Validates namespace existence.
   • Checks for image pull secrets.
   • Verifies storage class availability.
4. Run the baw-model-gateway-deployment.sh script to install the Model Gateway.

   Run the following command from the scripts folder of your downloaded cert-kubernetes.

   ./baw-model-gateway-deployment.sh -n <instance-namespace>

   The interactive mode requires an instance namespace and then takes the following actions:

   1. Reads the CP4BA common configmap (ibm-cp4ba-common-config), which exists in your Business Automation Workflow <instance-namespace> to determine whether operators and services use separate namespaces.
   2. Prompts for license acceptance.
   3. Prompts for storage classes.
   4. Prompts for scale configuration.
   5. Optionally configures custom TLS.
   6. Optionally configures AI providers.
   7. Prompts for the external PostgreSQL details.

      Option 1: Enter the secret that you created (model-gateway-postgres-external-secret).

      Option 2: If you want the script to create the secret, enter a property file (cp4ba_model_gateway.property) that contains values for the following properties.

      MODEL_GATEWAY_POSTGRES_HOST
      MODEL_GATEWAY_POSTGRES_PORT
      MODEL_GATEWAY_POSTGRES_USERNAME
      MODEL_GATEWAY_POSTGRES_PASSWORD
      MODEL_GATEWAY_POSTGRES_DBNAME
      MODEL_GATEWAY_POSTGRES_SSL_MODE
      MODEL_GATEWAY_POSTGRES_CA_CERT_PATH
      MODEL_GATEWAY_POSTGRES_USE_CLIENT_CERT
      MODEL_GATEWAY_POSTGRES_CLIENT_CERT_PATH
      MODEL_GATEWAY_POSTGRES_CLIENT_KEY_PATH

   8. Shows a deployment summary.
   9. Prompts for final confirmation.

   The following example shows the command with the required options.

   ./baw-model-gateway-deployment.sh \
     --accept-license \
     -o cp4ba-operators \
     -n cp4ba-instance \
     -b ocs-storagecluster-ceph-rbd \
     -f ocs-storagecluster-cephfs \
     -s medium

   Tip: Use the -o cp4ba-operators parameter only if you chose to install the Business Automation Workflow operators and the Business Automation Workflow deployments in separate namespaces (separation of duties).

   The following example shows the command with custom namespaces.

   ./baw-model-gateway-deployment.sh \
     --accept-license \
     -o my-operators \
     -n my-instance \
     -b portworx-db \
     -f portworx-shared \
     -s large

   The following example shows the command for a development or test deployment with SQLite.

   ./baw-model-gateway-deployment.sh \
     -n cp4ba-instance \
     --accept-license \
     --lite-install

   The following example shows the command to enable custom TLS.

   ./baw-model-gateway-deployment.sh \
     -n cp4ba-instance \
     --accept-license \
     --use-custom-tls \
     -b ocs-storagecluster-ceph-rbd \
     -f ocs-storagecluster-cephfs

   The following example shows the command with AI Provider configuration.

   ./baw-model-gateway-deployment.sh \
     -n cp4ba-instance \
     --accept-license \
     --configure-providers \
     -b ocs-storagecluster-ceph-rbd \
     -f ocs-storagecluster-cephfs

   The configure-providers parameter instructs the script to prompt you to set the configuration for the selected AI providers. Select your choices from the available options, and provide the Kubernetes secret that is needed to make the connection.

   1. Select the AI providers that you want.
   2. Enter the URL and API key (credentials) for each selected provider.
   3. Configure the models that you want to use from the selected providers.
5. Optional: Monitor the progress of the installation.
   1. Watch the status.

      oc get modelgateway modelgateway-cr -n cp4ba-instance -w

   2. Check the operator logs.

      oc logs -n cp4ba-operators \
        -l control-plane=ibm-cpd-model-gateway-operator \
        --tail=100 -f

   3. Check the Model Gateway logs.

      oc logs -n cp4ba-instance \
        -l app=model-gateway \
        --tail=100 -f

6. Verify that the deployment is ready.
   1. Check the status of the Model Gateway deployment.

      oc get modelgateway -n cp4ba-instance

   2. Check all the pods.

      oc get pods -n cp4ba-instance -l app=model-gateway

   3. Check the PostgreSQL secret.

      oc get secret model-gateway-postgres-external-secret -n <instance-namespace>

   4. Check the AI Provider secret (if configured).

      oc get secret model-gateway-provider-secret -n <instance-namespace>

   5. Check the Redis cluster.

      oc get rediscp -n cp4ba-instance

7. Access the Model Gateway service.

   oc get svc model-gateway -n cp4ba-instance