Quick start: Evaluate and track a prompt template

Take this tutorial to learn how to evaluate and track a prompt template. You can evaluate prompt templates in projects or deployment spaces to measure the performance of foundation model tasks and understand how your model generates responses. Then, you can track the prompt template in an AI use case to capture and share facts about the asset to help you meet governance and compliance goals.

Required services: watsonx.governance; watsonx.ai; Watson Machine Learning

Your basic workflow includes these tasks:

Open a project that contains the prompt template to evaluate. Projects are where you can collaborate with others to work with assets.
Evaluate a prompt template using test data.
Review the results on the AI Factsheet.
Track the evaluated prompt template in an AI use case.
Deploy and test your evaluated prompt template.

Read about prompt templates

With watsonx.governance, you can evaluate prompt templates in projects to measure how effectively your foundation models generate responses for the following task types:

Classification
Summarization
Generation
Question answering
Entity extraction

Watch a video about evaluating and tracking a prompt template

Watch Video Watch this video to preview the steps in this tutorial. There might be slight differences in the user interface shown in the video. The video is intended to be a companion to the written tutorial.

This video provides a visual method to learn the concepts and tasks in this documentation.

Try a tutorial about evaluating and tracking a prompt template

In this tutorial, you will complete these tasks:

Task 1: Create the workspaces
Task 2: Create an inventory and AI use case
Task 3: Evaluate the sample prompt template
Task 4: Start tracking the prompt template
Task 5: Import the tracked assets for validation
Task 6: Validate the prompt template
Task 7: Deploy the prompt template

Tips for completing this tutorial

Here are some tips for successfully completing this tutorial.

Get help in the community

If you need help with this tutorial, you can ask a question or find an answer in the watsonx Community discussion forum.

Set up your browser windows

For the optimal experience completing this tutorial, open Cloud Pak for Data in one browser window, and keep this tutorial page open in another browser window to switch easily between the two applications. Consider arranging the two browser windows side-by-side to make it easier to follow along.

Side-by-side tutorial and UI

Tip: If you encounter a guided tour while completing this tutorial in the user interface, click Maybe later.

Task 1: Create the workspaces

To complete this tutorial, you need three workspaces:

Development phase: A development project to store the assets that you develop, evaluate, and track.
Validation phase: A validation project to store the assets that are ready to be validated.
Operation phase: A production deployment space to store the validated assets and deployments.

Task 1a: Create the development project based on a sample

You need a project to store the prompt template and the evaluation. Follow these steps to create a project based on a sample:

Download the getting-started-with-watsonx-governance.zip file.
From the Navigation Menu , choose Projects > All projects.
On the Projects page, click New project.
Select Local file.
Drag the previously downloaded ZIP file on to the page.
On the Create a project page, copy and paste the project name and add an optional description for the project.
```
Getting started with watsonx.governace
```
Click Create.
Click View new project to verify that the project and assets were created successfully.
Click the Assets tab to view the project's assets.

Check your progress

The following image shows the development project Assets tab. You are now ready to create the inventory and AI use case.

Task 1b: Create a validation project

Typically, the prompt engineer evaluates the prompt with test data, and the validation engineer validates the prompt. The validation engineer has access to the validation data that prompt engineers might not have. In this case, validation data occurs in a different project. Follow these steps to create an empty project. Later, you import assets from the development project into the validation project.

From the Navigation Menu , choose Projects > All projects.
On the Projects page, click New project.
For the project name, type:
```
Validation project
```
Click Create.

Check your progress

The following image shows the empty validation project.

Task 1c: Create a deployment space

You need to create a deployment space now, so you can later promote the prompt template to that deployment space. Follow these steps to create the deployment space:

From the Navigation Menu , choose Deployments.
Click New deployment space.
For the Space name, copy and paste the following text:
```
Insurance claims deployment space
```
For the Deployment stage, select Production.

Important: You must select Production for the Deployment stage if you wish to move the deployment from the Evaluation stage to the Operation stage.
Click Create.
When the space is created, click View new space.

Check your progress

The following image shows the deployment space.

Task 2: Create an inventory and AI use case

An inventory is for storing and reviewing AI use cases. AI use cases collect governance facts for AI assets that your organization tracks. You can view all the AI use cases in an inventory.

Task 2a: Create an inventory

Follow these steps to create an inventory:

From the Navigation Menu , choose AI governance > AI use cases.

Note: If this is your first time working with AI use cases, then you are prompted to Complete setup.
Click the Manage inventories icon .

If you have an existing inventory, then you can skip to Create an AI use case to use that inventory for this tutorial. Otherwise, continue with the following steps to create an inventory.
Click New inventory.
For the name, copy and paste the following text:
```
Golden Bank Insurance Inventory
```
For the description, copy and paste the following text:
```
Inventory for insurance related processing
```
Clear the Add collaborators after creation option. You can restrict access at the inventory and AI use case level.
Click Create.
Close the Manage inventories page.

Check your progress

The following image shows the inventory. You are now ready to create an AI use case.

Task 2b: Create an AI use case

This tutorial uses OpenPages to create and manage AI use cases. If you are not using OpenPages, then refer to Setting up an AI use case for the steps to create a use case without OpenPages.

An AI use case is a defined business problem that you can solve with the help of AI. Usually these are defined before any AI asset gets developed. Follow these steps to create an AI use case with OpenPages:

Click New AI use case.
For the Name, copy and paste the following text:
```
Insurance claims processing AI use case
```
For the Owner field, select your user name.

For the Description and for the Purpose, type:

Use case for evaluating the prompt templates for insurance claims processing for Golden Bank

For the Primary Business Entity, click Add.
1. Select Catalogs.
2. Click Done.
Click Save.
In the Third Party Link field, click the link to open the AI use case in the watsonx user interface.

Check your progress

The following image shows the AI use case. You are now ready to track the prompt template.

Task 2c: Associate the workspaces with the use case

Follow these steps to associate the workspaces with this use case:

Note: You can create new projects and deployment spaces from within the AI use case.

Scroll to the Associated workspaces section.
Under the Development phase, click Associate workspace.
1. Select the Getting started with watsonx.governance project.
2. Click Save.
Under the Validation phase, click Associate workspace.
1. Select the Validation project.
2. Click Save.
Under the Operation phase, click Associate workspace.
1. Select Insurance claims deployment space.
2. Click Save.

Check your progress

The following image shows the AI use case with all associated workspaces.

Task 3: Evaluate the sample prompt template

The sample project contains a few prompt templates and CSV files used as test data. Follow these steps to review and evaluate one of the sample prompt templates:

Task 3a: Edit the sample prompt template in the Prompt Lab

View the prompt template to see how it is structured.

From the Navigation Menu , choose Projects > All projects.
Select the Getting started with watsonx.governance project.
Click the Assets tab.
Click Insurance claim summarization to open the prompt template in Prompt Lab, and then click Edit.
Click the Prompt variables icon .

Note: To run evaluations, you must create at least one prompt variable.
Scroll to the Try section. Notice the {input} variable in the Input field. You must include the prompt variable as input for testing your prompt. A prompt variable is a placeholder keyword that you include in the static text of your prompt at creation time and replace with text dynamically at run time.

Check your progress

The following image shows the Prompt Lab.

Task 3b: Evaluate the prompt template

Now you are ready to evaluate the prompt template.

Click the Evaluate icon .
If prompted, click Associate a service instance to select the service to use for the evaluation.
1. Select the appropriate service.
2. Click Associate.
Expand the Generative AI Quality section to see a list of dimensions. The available metrics depend on the task type of the prompt. For example, summarization has different metrics than classification.
Click Next.
Select the test data:
1. Click Select from project.
2. Select Project file > Insurance claim summarization test data.csv.
3. Click Select.
4. For the Input column, select Insurance_Claim.
5. For the Reference output column, select Summary.
6. Click Next.
Click Evaluate. When the evaluation completes, you see the test results on the Evaluate tab.
Click the AI Factsheet tab.
1. View the information on each of the sections on the tab.
2. Click Development > Getting started with watsonx.governance > Test results to see the test results again.

Check your progress

The following image shows the results of the evaluation. Now you can start tracking the prompt template in an AI use case.

Task 4: Start tracking the prompt template

You can track your prompt template in an AI use case to report the development and test process to your peers. Follow these steps to start tracking the prompt template:

On the AI Factsheet tab, click the Governance page.
Click Track an AI use case.
Notice that the associated AI use case is Insurance claims processing AI use case.
Select an approach. An approach is one facet of the solution to the business problem represented by the AI use case. For example, you might create approaches to track several prompt templates in a use case.
Click Next.
If you are using OpenPages, you are prompted to define an asset record. Select New asset record, and click Next.
For the model version, select Experimental.
Accept the default value for the version number.
Click Next.
Review the information, and then click Track asset.
When model tracking successfully begins, click the View details icon to open the AI use case.
Click the Lifecycle tab to see the prompt template is in the Development phase. As the prompt template moves through the AI lifecycle, it will move through these phases:
- Development phase: AI assets that have been developed in a project environment.
- Validation phase: AI assets that have been deployed in a space or project for validation.
- Operation phase: AI assets deployed in a space for operation.

Check your progress

The following image shows the Lifecycle tab in the AI use case with the prompt template in the Development phase. You are now ready to continue to the Validation phase.

Task 5: Import the tracked assets for validation

As noted in Task 1, typically, the prompt engineer evaluates the prompt with test data, and the validation engineer validates the prompt. The validation engineer has access to the validation data that prompt engineers might not have. In this case, validation data occurs in a different project. Follow these steps to export the development project and import those assets into the validation project that you created in Task 1 to move the asset into the Validation phase of the AI lifecycle:

Task 5a: Export the sample project

Follow these steps to export the development project:

From the Navigation Menu , choose Projects > All projects.
Select the Getting started with watsonx.governance project.
Click the Import/Export icon > Export project.
Check the box to select all assets.
Click Export.
Click Continue export to acknowledge that the assets might contain credentials.
Wait to be prompted for the project file name, and type validation-project.zip, and then click Save.
When the project export completes, click Back to project.

Check your progress

The following image shows the Export project page.

Task 5b: Import the assets into the validation project

Follow these steps to import the assets from the development project into the validation project:

From the Navigation Menu , choose Projects > All projects.
Open the Validation project.
Click the Import/Export icon > Import project.
Click Browse.
Select the validation-project.zip, and click Open.
Select the option to indicate agreement: I understand that some types of assets overwrite existing asets with the same name and type.
Click Import.
When the assets import successfully, click the Refresh icon to see the imported assets.

Check your progress

The following image shows the validation project Assets tab. You are now ready to evaluate the sample prompt template in the validation project.

Task 6: Validate the prompt template

Now you are ready to evaluate the prompt template in this validation project using the same evaluation process as before. Use the same test data set for evaluation. And select the same Input and Output columns as before. Follow these steps to validate the prompt template:

Click the Assets tab in the Validation project.
From the Overflow menu for the Insurance claim summarization prompt template, select Evaluate.
Click Evaluate to start the evaluation.
Repeat the steps in Task 3a: Evaluate the prompt template to evaluate the Claims processing summarization prompt template in the Validation project.
Click the AI Factsheet tab when the evaluation is complete.
View both sets of test results:
1. Click Development > Getting started with watsonx.governance > Test results.
2. Click Validation > Validation project > Test results.

Check your progress

The following image shows the validation test results. You are now ready to promote the prompt template to a deployment space, and then deploy the prompt template.

Task 7: Deploy the prompt template

To deploy the prompt template, you need promote it the deployment space that you created in Task 1. Then, in the deployment space, you can create a deployment and test the deployed prompt template.

Task 7a: Promote the prompt template to a deployment space

You promote the prompt template to a deployment space in preparation for deploying it. Follow these steps to prompte the prompt template:

Click Validation project project in the navigation trail to return to the Assets tab.
From the Overflow menu for the Insurance claim summarization prompt template, select Promote to space.
For the Target space, select the Insurance claims deployment space deployment space from the list.
Check the option to Go to the space after promoting the prompt template.
Click Promote.

Check your progress

The following image shows the prompt template in the deployment space. You are now ready to create a deployment.

Task 7b: Deploy the prompt template

Now you can create an online deployment of the prompt template from inside the deployment space. Follow these steps to create a deployment:

From the Insurance claims summarization asset page in the deployment space, select New deployment.
For the deployment name, copy and paste the following text:
```
Insurance claims summarization deployment
```
Click Create.

Check your progress

The following image shows the deployed prompt template.

Task 7c: View the deployed prompt template

Follow these steps to view the deployed prompt template in its current phase of the lifecycle:

View the deployment when it is ready. The API reference tab provides information for you to use the prompt template deployment in your application.
Click the Test tab. The Test tab allows you to submit an instruction and Input to test the deployment.
Click Generate. Close the results window.
Click the AI Factsheet tab.
Scroll down to the bottom of the AI Factsheet page, and click the arrow for more details.
Review the information in the Development, Validation, and Operation phases for the AI Factsheet for the deployed prompt template.
Scroll to the top of the page, and click the View details icon to open the AI use case.
In the use case, click the Lifecycle tab. You can see that the prompt template is now in the Operation phase.
Click the Insurance claim summarization prompt template in the Operation phase. When you are done, click Cancel.
Click the Insurance claims summarization deployment prompt template deployment in the Operation phase. When you are done, click Cancel.

Check your progress

The following image shows the prompt template prompt template in the Operation phase of the lifecycle.

Next steps

Try one of the other tutorials:

Additional resources

View more videos.

Parent topic: Quick start tutorials

Prerequisites

Draft content - tasks for review servers

Task 1

To complete this tutorial, you need three workspaces:

Development phase: A development project to store the assets that you develop, evaluate, and track.
Validation phase: A validation project to store the assets that are ready to be validated.
Operation phase: A production deployment space to store the validated assets and deployments.

Task 1a: Create the development project based on a sample

You need a project to store the prompt template and the evaluation. Follow these steps to create a project based on a sample:

Download the getting-started-with-watsonx-governance.zip file.
From the Navigation Menu , choose Projects > All projects.
On the Projects page, click New project.
Select Local file.
Drag the previously downloaded ZIP file on to the page.
On the Create a project page, copy and paste the project name and add an optional description for the project.
```
Getting started with watsonx.governace
```
Click Create.
Click View new project to verify that the project and assets were created successfully.
Click the Assets tab to view the project's assets.

Check your progress

The following image shows the development project Assets tab. You are now ready to create the inventory and AI use case.

Task 1b: Create a validation project

From the Navigation Menu , choose Projects > All projects.
On the Projects page, click New project.
For the project name, type:
```
Validation project
```
Click Create.

Check your progress

The following image shows the empty validation project.

Task 1c: Create a deployment space

You need to create a deployment space now, so you can later promote the prompt template to that deployment space. Follow these steps to create the deployment space:

From the Navigation Menu , choose Deployments.
Click New deployment space.
For the Space name, copy and paste the following text:
```
Insurance claims deployment space
```
For the Deployment stage, select Production.

Important: You must select Production for the Deployment stage if you wish to move the deployment from the Evaluation stage to the Operation stage.
Click Create.
When the space is created, click View new space.

Check your progress

The following image shows the deployment space.

Task 2

An inventory is for storing and reviewing AI use cases. AI use cases collect governance facts for AI assets that your organization tracks. You can view all the AI use cases in an inventory.

Task 2a: Create an inventory

Follow these steps to create an inventory:

From the Navigation Menu , choose AI governance > AI use cases.

Note: If this is your first time working with AI use cases, then you are prompted to Complete setup.
Click the Manage inventories icon .

If you have an existing inventory, then you can skip to Create an AI use case to use that inventory for this tutorial. Otherwise, continue with the following steps to create an inventory.
Click New inventory.
For the name, copy and paste the following text:
```
Golden Bank Insurance Inventory
```
For the description, copy and paste the following text:
```
Inventory for insurance related processing
```
Clear the Add collaborators after creation option. You can restrict access at the inventory and AI use case level.
Click Create.
Close the Manage inventories page.

Check your progress

The following image shows the inventory. You are now ready to create an AI use case.

Task 2b: Create an AI use case

This tutorial uses OpenPages to create and manage AI use cases. If you are not using OpenPages, then refer to Setting up an AI use case for the steps to create a use case without OpenPages.

Click New AI use case.
For the Name, copy and paste the following text:
```
Insurance claims processing AI use case
```
For the Owner field, select your user name.

For the Description and for the Purpose, type:

Use case for evaluating the prompt templates for insurance claims processing for Golden Bank

For the Primary Business Entity, click Add.
1. Select Catalogs.
2. Click Done.
Click Save.
In the Third Party Link field, click the link to open the AI use case in the watsonx user interface.

Check your progress

The following image shows the AI use case. You are now ready to track the prompt template.

Task 2c: Associate the workspaces with the use case

Follow these steps to associate the workspaces with this use case:

Note: You can create new projects and deployment spaces from within the AI use case.

Scroll to the Associated workspaces section.
Under the Development phase, click Associate workspace.
1. Select the Getting started with watsonx.governance project.
2. Click Save.
Under the Validation phase, click Associate workspace.
1. Select the Validation project.
2. Click Save.
Under the Operation phase, click Associate workspace.
1. Select Insurance claims deployment space.
2. Click Save.

Check your progress

The following image shows the AI use case with all associated workspaces.

Task 3

The sample project contains a few prompt templates and CSV files used as test data. Follow these steps to review and evaluate one of the sample prompt templates:

Task 3a: Edit the sample prompt template in the Prompt Lab

View the prompt template to see how it is structured.

From the Navigation Menu , choose Projects > All projects.
Select the Getting started with watsonx.governance project.
Click the Assets tab.
Click Insurance claim summarization to open the prompt template in Prompt Lab, and then click Edit.
Click the Prompt variables icon .

Note: To run evaluations, you must create at least one prompt variable.
Scroll to the Try section. Notice the {input} variable in the Input field. You must include the prompt variable as input for testing your prompt. A prompt variable is a placeholder keyword that you include in the static text of your prompt at creation time and replace with text dynamically at run time.

Check your progress

The following image shows the Prompt Lab.

Task 3b: Evaluate the prompt template

Now you are ready to evaluate the prompt template.

Click the Evaluate icon .
If prompted, click Associate a service instance to select the service to use for the evaluation.
1. Select the appropriate service.
2. Click Associate.
Expand the Generative AI Quality section to see a list of dimensions. The available metrics depend on the task type of the prompt. For example, summarization has different metrics than classification.
Click Next.
Select the test data:
1. Click Select from project.
2. Select Project file > Insurance claim summarization test data.csv.
3. Click Select.
4. For the Input column, select Insurance_Claim.
5. For the Reference output column, select Summary.
6. Click Next.
Click Evaluate. When the evaluation completes, you see the test results on the Evaluate tab.
Click the AI Factsheet tab.
1. View the information on each of the sections on the tab.
2. Click Development > Getting started with watsonx.governance > Test results to see the test results again.

Check your progress

The following image shows the results of the evaluation. Now you can start tracking the prompt template in an AI use case.

Task 4

You can track your prompt template in an AI use case to report the development and test process to your peers. Follow these steps to start tracking the prompt template:

On the AI Factsheet tab, click the Governance page.
Click Track an AI use case.
Notice that the associated AI use case is Insurance claims processing AI use case.
Select an approach. An approach is one facet of the solution to the business problem represented by the AI use case. For example, you might create approaches to track several prompt templates in a use case.
Click Next.
If you are using OpenPages, you are prompted to define an asset record. Select New asset record, and click Next.
For the model version, select Experimental.
Accept the default value for the version number.
Click Next.
Review the information, and then click Track asset.
When model tracking successfully begins, click the View details icon to open the AI use case.
Click the Lifecycle tab to see the prompt template is in the Development phase. As the prompt template moves through the AI lifecycle, it will move through these phases:
- Development phase: AI assets that have been developed in a project environment.
- Validation phase: AI assets that have been deployed in a space or project for validation.
- Operation phase: AI assets deployed in a space for operation.

Check your progress

The following image shows the Lifecycle tab in the AI use case with the prompt template in the Development phase. You are now ready to continue to the Validation phase.

Task 5

Task 5a: Export the sample project

Follow these steps to export the development project:

From the Navigation Menu , choose Projects > All projects.
Select the Getting started with watsonx.governance project.
Click the Import/Export icon > Export project.
Check the box to select all assets.
Click Export.
Click Continue export to acknowledge that the assets might contain credentials.
Wait to be prompted for the project file name, and type validation-project.zip, and then click Save.
When the project export completes, click Back to project.

Check your progress

The following image shows the Export project page.

Task 5b: Import the assets into the validation project

Follow these steps to import the assets from the development project into the validation project:

From the Navigation Menu , choose Projects > All projects.
Open the Validation project.
Click the Import/Export icon > Import project.
Click Browse.
Select the validation-project.zip, and click Open.
Select the option to indicate agreement: I understand that some types of assets overwrite existing asets with the same name and type.
Click Import.
When the assets import successfully, click the Refresh icon to see the imported assets.

Check your progress

The following image shows the validation project Assets tab. You are now ready to evaluate the sample prompt template in the validation project.

Task 6

Click the Assets tab in the Validation project.
From the Overflow menu for the Insurance claim summarization prompt template, select Evaluate.
Click Evaluate to start the evaluation.
Repeat the steps in Task 3a: Evaluate the prompt template to evaluate the Claims processing summarization prompt template in the Validation project.
Click the AI Factsheet tab when the evaluation is complete.
View both sets of test results:
1. Click Development > Getting started with watsonx.governance > Test results.
2. Click Validation > Validation project > Test results.

Check your progress

The following image shows the validation test results. You are now ready to promote the prompt template to a deployment space, and then deploy the prompt template.

Task 7

To deploy the prompt template, you need promote it the deployment space that you created in Task 1. Then, in the deployment space, you can create a deployment and test the deployed prompt template.

Task 7a: Promote the prompt template to a deployment space

You promote the prompt template to a deployment space in preparation for deploying it. Follow these steps to prompte the prompt template:

Click Validation project project in the navigation trail to return to the Assets tab.
From the Overflow menu for the Insurance claim summarization prompt template, select Promote to space.
For the Target space, select the Insurance claims deployment space deployment space from the list.
Check the option to Go to the space after promoting the prompt template.
Click Promote.

Check your progress

The following image shows the prompt template in the deployment space. You are now ready to create a deployment.

Task 7b: Deploy the prompt template

Now you can create an online deployment of the prompt template from inside the deployment space. Follow these steps to create a deployment:

From the Insurance claims summarization asset page in the deployment space, select New deployment.
For the deployment name, copy and paste the following text:
```
Insurance claims summarization deployment
```
Click Create.

Check your progress

The following image shows the deployed prompt template.

Task 7c: View the deployed prompt template

Follow these steps to view the deployed prompt template in its current phase of the lifecycle:

View the deployment when it is ready. The API reference tab provides information for you to use the prompt template deployment in your application.
Click the Test tab. The Test tab allows you to submit an instruction and Input to test the deployment.
Click Generate. Close the results window.
Click the AI Factsheet tab.
Scroll down to the bottom of the AI Factsheet page, and click the arrow for more details.
Review the information in the Development, Validation, and Operation phases for the AI Factsheet for the deployed prompt template.
Scroll to the top of the page, and click the View details icon to open the AI use case.
In the use case, click the Lifecycle tab. You can see that the prompt template is now in the Operation phase.
Click the Insurance claim summarization prompt template in the Operation phase. When you are done, click Cancel.
Click the Insurance claims summarization deployment prompt template deployment in the Operation phase. When you are done, click Cancel.

Check your progress

The following image shows the prompt template prompt template in the Operation phase of the lifecycle.