Tracking prompt templates
Track a prompt template in an AI use case to capture and share facts about the asset to help you meet governance and compliance goals.
Tracking prompt templates
A prompt template is the saved prompt input for a foundation model. A prompt template can include variables so that it can be run with different options. For example, if you have a prompt that summarizes meeting notes for project-X, you can define a variable so that the same prompt can run for project-Y.
You can add a saved prompt template to an AI use case to track the details for the prompt template. In addition to recording details about the prompt template creation information and source model details, the factsheet tracks information from prompt template evaluations to capture performance metrics. You can evaluate prompt templates before or after you start tracking a prompt template.
Before you begin
Before you can track a prompt template, these conditions must be met.
- Be an administrator or editor for the project that contains the prompt template.
- The prompt template must include at least one variable. For more information, see Building reusable prompts.
Watch this video to see how to track a prompt template in an AI use case.
This video provides a visual method to learn the concepts and tasks in this documentation.
Tracking a prompt template in an AI use case
You can add a prompt template to an AI use case from a project or space.
- Open the project or space that contains the prompt template that you want to govern.
- From the action menu for the asset, click View AI use case.
- If this prompt template is not already part of an AI use case, you are prompted to Track in AI use case. When you start tracking a prompt template, it is locked and you can no longer edit it. To make changes, you must create
a new prompt template.
- Select an existing AI use case or follow the prompts to create a new one.
- Choose an existing approach or create a new approach. An approach represents one facet of a complete solution. Each approach creates a version set for all assets in the same approach.
- Choose a version numbering scheme. All the assets in an approach share a common version. Choose from:
- Experimental if you plan to update frequently.
- Stable if the assets are not changing rapidly.
- Custom if you want to start a new version number. Version numbering must follow a schema of major.minor.patch.
When tracking is enabled, all collaborators for the use case can review details for the prompt template.
Details are captured for each lifecycle stage for a prompt template.
- Develop provides information about how the prompt is defined, including the prompt itself, creation date, foundation model that is used, prompt parameters set, and variables defined.
- Evaluate displays the dimension metrics from evaluating your prompt template.
- Operate provides details that are related to how the prompt template is deployed for productive use.
Viewing the factsheet for a tracked prompt template
Click the name of the prompt template in an AI use case to view the associated factsheet.
The factsheet for a prompt template collects this type of data:
- Governance collects basic information such as the name of the AI use case, the description, and the approach name and version data.
- Foundation model displays the name of the foundation model, the license ID, and the model publisher.
- Prompt template shows the prompt name, ID, prompt input, and variables.
- Prompt parameters collect the configuration options for the prompt template, including the decoding method and stopping criteria.
- Evaluation displays the data from evaluation, including alerts, and metric data from the evaluation. For example, this prompt template shows the metrics data for quality evaluations on the prompt template. One threshold alert was triggered by the evaluation:
- Validate shows the data for how the prompt template was evaluated, including the data set used for the validation, alerts triggered, and evaluation metric data.
- Attachments shows information about attachments that support the use case.
Moving a prompt template through lifecycle stages
When a prompt template is tracked, you can see details from creating the prompt template, and evaluating performance against appropriate metrics. The next stage in the lifecycle is to _validate the prompt template. This involves testing the prompt template with new data. If you are the prompt engineer who is tasked with validating the asset, follow these steps to validate the prompt template and capture the validation data in the associated factsheet.
- From the project containing the prompt template, export the project to a compressed ZIP file.
- Create a new project and populate it with the exported ZIP file.
- Upload validation data, evaluate the prompt template, and save the results to the validation project.
- From the project, promote the prompt template to a new or existing deployment space that is designated as a Production stage. The stage is assigned when the space is created and cannot be updated, so create a new space if you do not have a production space available.
- After you promote the prompt template to a deployment space, you can configure continuous monitoring.
- Details from monitoring the prompt template in a production space are displayed in the Operate lifecycle stage of the AI use case.
Tracking detached prompt templates from externally-hosted large language models
As part of your governance process, you can track prompt templates for generative AI models that are not created or hosted by IBM. For example, you can evaluate a prompt template for a Google Vertex AI model, and Azure OpenAI model, or an AWS Bedrock model. To learn how to create a detached prompt template for an externally-hosted model and save it to a project or space, see Evaluating prompt templates for non-IBM foundation models.
To explore the evaluation results, you can create a detached deployment in a deployment space to connect to the remote prompt template, then evaluate the detached prompt template with dimensions for the task type. For more information, see Creating a detached deployment for an external prompt.
After the prompt template is saved to a product or space, you track it in a use case in the same way you track a prompt template from a watsonx.ai foundation model.
This example shows a detached prompt template that is being tracked in an AI use case.
Viewing the factsheet for the detached prompt templates shows information about where the source model is hosted.
Learn more
- Follow the tutorial Quick start: Evaluate and track a prompt template to evaluate and track a sample prompt template.
- See Deploying a prompt template for details on preparing a prompt template for production.
- See Evaluating prompt templates for details on evaluating a prompt template for dimensions such as accuracy or to test for the presence of hateful or abusive speech.
Parent topic: Tracking assets in an AI use case