Monitoring Amazon SageMaker AI

Amazon SageMaker AI provides capabilities for building, training, and deploying generative AI applications using foundation models. This guide shows you how to instrument an application using Amazon SageMaker AI endpoints with OpenLLMetry to send telemetry data to Instana.

Prerequisites

Make sure that the following prerequisites are met:

Python 3.8 or later
Amazon Web Services (AWS) credentials with SageMaker access
A deployed SageMaker AI endpoint
An Instana backend account with Generative AI observability enabled
Review of Getting Started guide on agent and agentless modes

Instrumenting your Amazon SageMaker AI application

Install the required packages.

pip install boto3 pandas numpy traceloop-sdk

Export your AWS credentials.

export AWS_ACCESS_KEY_ID="<your-access-key-id>"
export AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"

To create or manage AWS access keys, see Amazon IAM documentation.

Create a Python file with the following code.

import boto3
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow
import pandas as pd

# Initialize OpenLLMetry
Traceloop.init(app_name="sagemaker_example", disable_batch=True)

# Initialize SageMaker Runtime client
runtime = boto3.Session().client('sagemaker-runtime', region_name="us-east-1")

@workflow(name="sagemaker_example")
def invoke_sagemaker_endpoint(endpoint):
    # Sample input data (1 row with 12 features)
    X_test = [[1, 8.2, 10.5, 1.5, 9.6, 1.3, 11.2, 2.9, 100, 11, 12.0, 13]]

    # Convert data to CSV format
    df = pd.DataFrame(X_test)
    csv_input = df.to_csv(index=False, header=False)
    csv_bytes = csv_input.encode('utf-8')

    # Invoke the SageMaker endpoint
    response = runtime.invoke_endpoint(
        EndpointName=endpoint,
        ContentType='text/csv',
        Body=csv_bytes
    )

    # Read and decode the prediction result
    return response['Body'].read().decode('utf-8')

# Replace with your SageMaker endpoint name
invoke_sagemaker_endpoint('<sagemaker-endpoint>')

Note:

The input data format depends on your deployed model's requirements. Adjust the data structure based on your specific model.

Run your Amazon SageMaker AI application.
```
python3 sagemaker_app.py
```
The application will send data to your SageMaker AI endpoint and display the prediction results. OpenLLMetry automatically captures traces for each endpoint invocation and sends them to Instana.
View data on Instana.

After running your application, data will appear in the Instana Gen AI observability dashboard showing:
- Endpoint name
- Request and response payloads
- Inference latency
- Error information (if any)

Troubleshooting

For common issues such as traces not appearing or connection errors, see Troubleshooting.

Authentication errors

If you encounter authentication errors:

Check whether your AWS credentials are set correctly
Check whether your IAM user has the necessary SageMaker permissions
Make sure that your credentials are not expired
Verify the AWS region matches your endpoint's region

Endpoint not found errors

If you encounter endpoint not found errors:

Verify the endpoint name is correct
Check whether the endpoint is deployed and in "InService" status
Make sure that the endpoint is in the same region as your client configuration
Confirm your IAM policy includes sagemaker:InvokeEndpoint permission

Model input errors

If you encounter model input errors:

Verify the input data format matches your model's expectations
Check whether the content type is correct for your model
Make sure that data types and dimensions are correct
Review your model's input schema in the SageMaker console

Next steps

Explore LLM providers supported by Instana
Learn about cost calculation for your AI usage
Set up alerts for your Amazon SageMaker AI endpoint invocations
Review Amazon SageMaker AI documentation