Setting up the model gateway with code

The model gateway provides a single OpenAI-compatible LLM-as-a-service API, which proxies the routing of requests to model providers and currently offers features such as load balancing.

Note:

The model gateway feature is in preview and available only in the Toronto region.

Before you begin

To use the model gateway on IBM Cloud, you need to provide credentials for each supported model provider that you plan to use and configure environment variables for all your keys.

You can also use environment variables to track other API keys (such as for IBM Cloud IAM) and other configuration values such as URLs. You can export the host URL of the IBM Cloud model gateway to the environment variable GATEWAY_URL and the IAM API key to IBM_CLOUD_APIKEY.
```
export GATEWAY_URL="https://ca-tor.ml.cloud.ibm.com/ml/gateway"
export IBM_CLOUD_APIKEY="xxxx" 
```
To authenticate requests to IBM Cloud services, use IBM Cloud Identity and Access Management (IAM).

To work with the API, authenticate your application or service by including your IBM Cloud IAM access token in API requests.

Attention: Users must be assigned administrator permissions to all watsonx.ai Runtime instances.
Create an IBM Cloud API key. For more information, see IBM Cloud API key.
Provision and link an IBM Secrets Manager instance to securely store and manage your API keys, credentials, and configuration values to use the model gateway.
- To configure by using the UI, see Creating a Secrets Manager instance in the UI.
- To configure by using the CLI, use the following command.
  Replace <region> with ca-tor and <plan> with the pricing plan ID. For details, see Creating a Secrets Manager instance from the CLI.
```
  ibmcloud login
  ibmcloud target -r <region> -g <resource_group_name>
  ibmcloud resource service-instance-create <instance_name> secrets-manager <plan> -p '{"allowed_network": "public-and-private"}'
  ibmcloud resource service-instances # Optional, verifies that the service instance was created successfully.
```
Authorize Secrets Manager to allow an IBM Cloud service to securely access secrets. For details, see Authorizing an IBM Cloud service to access Secrets Manager.

A SecretsReader service role or higher is required on your Secrets Manager instance to authorize. For more information, see Managing IAM access for Secrets Manager.
- To authorize by using the UI, see Creating an authorization in the console.
- To authorize by using the IBM Cloud CLI, use the following command:
```
  ibmcloud login -a https://cloud.ibm.com --apikey ${IBM_CLOUD_APIKEY}
  ibmcloud iam authorization-policy-create SecretsReader \
      --source-service-name pm-20 \
      --target-service-name secrets-manager
```
Note: pm-20 is the service name for watsonx.ai runtime.

For more information, see Using authorizations to grant access between services.

You can now use the IBM_CLOUD_APIKEY as a valid bearer token for the model gateway on IBM Cloud.
```
curl ${GATEWAY_URL}/v1/...  \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${IBM_CLOUD_APIKEY}" \
  ...
```

Procedure

Select a model provider that you want to configure. The model gateway provides access to multiple model providers through a single OpenAI-compatible endpoint.
Define and create a secret in IBM Cloud Secrets Manager for your provider. For more information, see Configuring credentials for model providers in IBM Cloud Secrets Manager.

Configure the provider by using the secret reference by running the following command:

  # Set environment variables
  export SM_CRN="crn:v1:bluemix:public:secrets-manager:us-south:a/your-account-id:your-secret-id::"
  export GATEWAY_URL="https://your-gateway-url"

  # Register the provider using the secret reference
  curl -sS ${GATEWAY_URL}<provider-endpoint> \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $IBM_CLOUD_APIKEY" \
    -d "$(jq -n \
      --arg resource "$SM_CRN" \
      --arg name "<custom-name-for-provider>" \
      '{name: $name, data_reference: {resource: $resource}}')"

For example, use the following command to configure OpenAI as a provider:

# Set environment variables
export SM_CRN="crn:v1:bluemix:public:secrets-manager:us-south:a/your-account-id:your-secret-id::"
export GATEWAY_URL="https://your-gateway-url"

# Register the provider using the secret reference
curl -sS ${GATEWAY_URL}/v1/providers/openai \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $IBM_CLOUD_APIKEY" \
  -d "$(jq -n \
    --arg resource "$SM_CRN" \
    --arg name "openai-provider" \
    '{name: $name, data_reference: {resource: $resource}}')"

Next steps

You can send requests to models through the model gateway. For details, see Inferencing gateway models.