Coding and deploying AI services with ibm-watsonx-ai-cli

Use the ibm-watsonx-ai-cli Python package to download, edit, test, and deploy AI services from your local environment.

By using ibm-watsonx-ai-cli, depending on your needs, you can:

  • Test an AI service locally, without deploying it, by passing arguments to a template that was downloaded to your local environment.
  • Deploy an AI service in watsonx.ai and then test it locally, by passing arguments to the deployed service.
  • Download, deploy, and test an app that is based on an AI service.

Prerequisites:

  • Python 3.11
  • The Poetry packaging and dependency management tool must be installed in the local environment. To install Poetry, follow the instructions on the Poetry installation page.

To start working with AI service templates in your local environment, you must first install ibm-watsonx-ai-cli. To install ibm-watsonx-ai-cli, type this command on your local machine: pip install -U ibm-watsonx-ai-cli.

Important:
  • Some templates require a correctly prepared config file (config.toml). If a downloaded template contains config.toml with placeholders, read its description carefully and fill in the required fields.
  • For security reasons, it is recommended not to hard-code your API key or other secrets directly in your scripts or in the config.toml file. Instead, set them up as environment variables and then access the variables from your code. For example: WATSONX_APIKEY = os.environ.get("WATSONX_APIKEY", "").
  • Since version 0.2.0 of ibm-watsonx-ai-cli, the recommended way for passing the API key or other secrets that are required to authenticate in IBM watsonx.ai is to use the .env file. The .env file must be located in the path from which the cli commands are called.
  • The dist folder that contains the template source code distribution is created locally when package extension asset is prepared. Starting with version 0.3.0 of ibm-watsonx-ai-cli, this folder is automatically deleted after the AI service content is successfully deployed.

Example flow for deploying an AI service

To deploy AI service templates by using ibm-watsonx-ai-cli, refer to this example flow:

  1. Create the template in your IDE:

    watsonx-ai template new

    A list of available templates shows on your screen. Type the number that is associated with the template that you want to work with. Then type the name of the folder to be created.

  2. Navigate to the template directory:

    cd <template directory>
cp config.toml.example config.toml

  3. Customize the template code and edit the config.toml file to update placeholder valuesFor easy access to some of the required variables, see the Developer Access page.

  4. Test the template code:

    watsonx-ai template invoke "<your query>"

  5. Create a service:

    watsonx-ai service new

    When the service is deployed, ibm-watsonx-ai-cli returns the ID of the deployment. If you want this ID to be the default deployment ID when you invoke the deployed service, paste it as deployment_id in the config.toml file. If you don't do this, you must provide the deployment ID each time you invoke your deployed service.

  6. Test the service

    watsonx-ai service invoke "<your query>"
Note:

If you want to test your service locally by using a graphical user interface, install the React UI App. For more information, see watsonx.ai AI Service UI app

Example flow for creating an app that is based on an AI service template

To create apps that are based on AI service templates by using ibm-watsonx-ai-cli, refer to this example flow:

  1. Create an app in your IDE:

    watsonx-ai app new

    A list of available apps shows on your screen. Type the number that is associated with the app that you want to work with. Then type the name of the folder to be created.

  2. Navigate to the app directory and then copy your environment variable file:

    cd <created folder with the app>
cp template.env .env

  3. Edit the .env file to update placeholder values. For easy access to some of the required variables, see the Developer Access page. For the WATSONX_BASE_DEPLOYMENT_URL, use the URL to the endpoint of a deployed service.

  4. Start the app:

    watsonx-ai app run

    Alternatively, if you want to make changes to the source code of the app while the app is running, start the app in development mode:

    watsonx-ai app run --dev

    If an app is running in development mode, each time you save your changes the app reloads.

Example flow for running the eval command

Use the eval command to check how an agent performs on a specific dataset. You can include multiple benchmarking files in a single eval command. The results are calculated separately for each file.

The eval command provides several metrics that can be used simultaneously to evaluate the agent’s performance. To get more information about the available metrics, run:

watsonx-ai template eval --help

For the answer_similarity and answer_relevance metrics, you can use an LLM as a judge. You can either provide the model's ID (the model must be available in your cloud environment) or use the llm_as_judge keyword (in this case, one of two predefined models are used).

Here’s an example usage of the eval command:

watsonx-ai template eval --tests test1.jsonl,test2.jsonl --metrics answer_similarity,answer_relevance --evaluator llm_as_judge

Here is a single result for one metric:

{
  "name": "answer_similarity",
  "method": "llm_as_judge",
  "provider": "unitxt",
  "value": 0.6333333333333333,
  "errors": null,
  "additional_info": null,
  "group": "answer_quality",
  "thresholds": [
    {
      "type": "lower_limit",
      "value": 0.7
    }
  ],
  "min": 0.0,
  "max": 1.0,
  "mean": 0.6333333333333333,
  "total_records": 3,
  "record_level_metrics": [
    {
      "name": "answer_similarity",
      "method": "llm_as_judge",
      "provider": "unitxt",
      "value": 1.0,
      "errors": null,
      "additional_info": null,
      "group": "answer_quality",
      "thresholds": [
        {
          "type": "lower_limit",
          "value": 0.7
        }
      ],
      "record_id": "c6ffedaa-b2b1-4ece-bbe6-0d680a470a12",
      "record_timestamp": null
    },
    {
      "name": "answer_similarity",
      "method": "llm_as_judge",
      "provider": "unitxt",
      "value": 0.9,
      "errors": null,
      "additional_info": null,
      "group": "answer_quality",
      "thresholds": [
        {
          "type": "lower_limit",
          "value": 0.7
        }
      ],
      "record_id": "f5c15bfc-80a8-4095-a2c3-54ae1c21dc23",
      "record_timestamp": null
    },
    {
      "name": "answer_similarity",
      "method": "llm_as_judge",
      "provider": "unitxt",
      "value": 0.0,
      "errors": null,
      "additional_info": null,
      "group": "answer_quality",
      "thresholds": [
        {
          "type": "lower_limit",
          "value": 0.7
        }
      ],
      "record_id": "f7dda69a-fd54-4b80-a966-2198c44109d8",
      "record_timestamp": null
    }
  ]
}

Learn more

For more details, see