Submitting LLM inference requests by using the AI Optimizer for IBM Z and IBM LinuxONE REST API

After successfully deploying an LLM, you can submit inference requests by specifying tags and other metadata that best describe the model. The AI Optimizer for Z and LinuxONE inference router will intelligently route the requests to the model based on the specified information. You can create and submit an inference request by using the AI Optimizer for Z and LinuxONE REST API.

1. Obtain an authentication token from the IBM Z Appliance Container Infrastructure (zACI) system API.

   An authentication token is a short-lived credential that is usually valid for 30 minutes. You will use the token to create a client API key before it expires.

   Submit an HTTPS POST API request by running the following curl command:

   token=$(curl -k -X POST https://<ssc-lpar-ip>:<port>/api/com.ibm.zaci.system/api-tokens \
      -H "zACI-API: com.ibm.zaci.system/1.0" \
      -H "Accept: application/vnd.ibm.zaci.payload+json;version=1.0" \
      -H "Content-Type: application/vnd.ibm.zaci.payload+json;version=1.0" \
      -d '{
            "kind": "request",
            "parameters": {
              "user": "<username>",
              "password": "<password>"
              }
           }' | jq -r '.parameters.token')
   echo $token

   Where:

   • <ssc-lpar-ip> is the IP address of your AI Optimizer for Z and LinuxONE SSC LPAR.
   • <port> is the port number of your AI Optimizer for Z and LinuxONE SSC LPAR.
   • <username> is your username for the Appliance Manager or the Inference router UI.
   • <password> is your password for the Appliance Manager or the Inference router UI.
   • The -k flag bypasses SSL certificate verification. For production environments, consider using proper SSL certificates and excluding the flag from the command.

   The command will return a JWT token string as shown in the following example:

   eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhZG1pbiI6dHJ1ZSwidXNlciI6InlvZGEiLCxxxxxxx

   Tip: The authentication token expires in 30 minutes. If it takes more time for you to create an API key with the token, you must obtain a new one by repeating this step.
2. Generate an API key for the REST API client that you will use to make inference requests.

   Compared to an authentication token, an API key is a long-lived credential that is usually valid for 180 days or 6 months. Use the authentication token that you just obtained to create an API key. Your REST API client can use the key repeatedly to make inference requests before it expires.

   Submit an HTTPS POST API request by running the following curl command:

   curl -k -X POST https://<ssc-lpar-ip>/aio-auth/create-client-api-key \
     -H "Authorization: Bearer token" \
     -H "Content-Type: application/json" \
     -d '{"client_application_name": "<client_application_name>"}' 

   Where:

   • <token> is the authentication token that you obtained in Step 1.
   • <client_application_name> is the descriptive name of your REST API application. The name must be unique. If you use a name that already exists, the API call will fail. Consider including appending a timestamp or a similar identifier in the name to ensure its uniqueness.

   The command returns a JSON response that contains an API key as shown in the following example:

   {
     "api_key": "qyorzi_obA0WYfprbVvoxxD5Lxxxxxx",
     "client_application_name": "<client_application_name>"
   }

   Save the API key value in a secure location. You will use this key to make inference requests later.

   Tip: The API key is valid for 180 days. After this period, you must create a new API key by repeating Steps 1 and 2.
3. Use the API key to submit an inference request to the LLMs that are deployed in AI Optimizer for Z and LinuxONE.

   Submit an HTTPS POST API request by running the following curl command:

   curl -k -X POST https://<ssc-lpar-ip>/api/com.ibm.aioinference/v1/completions \
     -H "Authorization: Bearer <api_key>" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "<model_name>",
       "prompt": "<prompt>",
       "max_tokens": <max_tokens>
     }' 

   Where:

   • <api_key> is the API key that you generated in Step 2.
   • <model_name> is the name of the LLM that you want to inference, such as ibm/granite-3-3-8b-instruct.
   • <prompt> is the text query that you want to send to the model.
   • <max_tokens> is the maximum number of tokens to generate in the response.

   For example, when you run the following curl command:

   curl -k -X POST https://<ssc-lpar-ip>/api/com.ibm.aioinference/v1/completions \
     -H "Authorization: Bearer qyorzi_obA0WYfprbVvoxxD5Lxxxxxx" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "ibm/granite-3-3-8b-instruct",
       "prompt": "Say hello",
       "max_tokens": 10
     }'

   The returned JSON response will be similar to the following example:

   {
     "id": "cmpl-...",
     "object": "text_completion",
     "created": 1774449556,
     "model": "ibm/granite-3-3-8b-instruct",
     "choices": [
       {
         "text": "Hello! How can I assist you today?",
         "index": 0,
         "finish_reason": "length"
       }
     ],
     "usage": {
       "prompt_tokens": 2,
       "completion_tokens": 10,
       "total_tokens": 12
     }
   }

   The response includes the following options and fields:

   • choices[].text is the generated text from the model.
   • choices[].finish_reason indicates that the text generation stopped at length when <max_tokens> was reached.
   • usage is the token consumption statistics for the request.

   The API key remains valid for 180 days. You can now use the key to make additional inference requests until it expires.

4. If needed, delete an API key that you no longer need to use before it expires.

   You can delete an API key if you no longer need to use it before its expiration or if you want to remove the key from a specific REST API application.

   Remember: Deleting an API key is permanent and cannot be undone. A single deletion operation will remove all API keys from the affected application. This means that deleting a key will effectively revoke the application's access to AI Optimizer for Z and LinuxONE. You must create a new API key if you want to use the same application to submit inference requests again.

   Submit an HTTPS DELETE API request by running the following curl command:

   curl -k -X DELETE https://<ssc-lpar-ip>/aio-auth/delete-client-api-key \
     -H "Authorization: Bearer token" \
     -H "Content-Type: application/json" \
     -d '{"client_application_name": "<client_application_name>"}' 

   Where:

   • token is the authentication token that you can obtain as described in Step 1.
   • <client_application_name> is the name of the REST API application from which you want to remove all API keys.

   The command returns a JSON response that confirms the deletion as shown in the following example:

   {
     "status": "success",
     "message": "Deleted 6 API key(s) for client application",
     "data": {
       "client_application_name": "<client_application_name>",
       "deleted_count": 6
     }
   }

   The deleted_count field indicates the number of API keys that were deleted from the affected application.