Payload logging

You can enable payload logging in the Watson OpenScale or watsonx.governance service to configure model evaluations.

The following sections describe how you can enable payload logging in the Watson OpenScale or watsonx.governance service:

Payload logging in Watson OpenScale

To manage payload data for configuring fairness and drift evaluations and explainability, Watson OpenScale must log your payload data in the data mart. When you add deployments for evaluations or provide model details, Watson OpenScale can automatically send a scoring request to log your model transactions if all of the required model details are detected. If Watson OpenScale doesn't detect all of the required model details or if you're evaluating image or unstructured text models, you must manually enable payload logging.

Manual payload logging

To manually log payload data in Watson OpenScale, you can use the Watson OpenScale Python SDK to enable payload logging.

You can use the following steps to enable payload logging with the Watson OpenScale Python SDK:

  • Import and initiate the API clients:

    from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator
    from ibm_watson_openscale import APIClient
    
    service_credentials = {
                 "url": "***",
                 "username": "***",
                 "password": "***"
                 }
    
    authenticator = CloudPakForDataAuthenticator(
         url=service_credentials["url"],
         username=service_credentials["username"],
         password=service_credentials["password"],
         disable_ssl_verification=True # Based on the SSL status of your cluster
     )
    wos_client = APIClient(authenticator=authenticator, service_url=service_credentials["url"])
    
  • Fetch the payload data set ID:

import time

time.sleep(5)
payload_data_set_id = None
payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, 
                                                target_target_id=subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if payload_data_set_id is None:
    print("Payload data set not found. Please check subscription status.")
else:
    print("Payload data set id: ", payload_data_set_id)
  • Store the payload data in the payload logging table:
import uuid
from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord

print("Performing explicit payload logging.....")
wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
           scoring_id=str(uuid.uuid4()),
           request=openscale_input,
           response=openscale_output,
           response_time=460
)])
time.sleep(5)
pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
print("Number of records in the payload logging table: {}".format(pl_records_count))

wos_client.data_sets.show_records(data_set_id=payload_data_set_id)

Payload logging in watsonx.governance

To manage payload data for configuring drift v2, generative AI quality, and model health evaluations, watsonx.governance must log your payload data in the payload logging table.

Generative AI quality evaluations use payload data to generate results for the following task types when you evaluate prompt templates:

  • Text summarization
  • Content generation
  • Question answering

Drift v2 and model health evaluations use payload data to generate results for the following task types when you evaluate prompt templates:

  • Text classification
  • Text summarization
  • Content generation
  • Entity extraction
  • Question answering

You can log your payload data with the payload logging endpoint or by uploading a CSV file. For more information, see Sending model transactions.

Parent topic: Managing payload data in Watson OpenScale