Monitoring LLMs from model providers
Instana monitors LLMs from model providers by collecting traces, metrics, and logs to provide real-time insights into performance, efficiency, and cost. Track key metrics including model inference latency, error rates, resource utilization, token usage, and costs to identify bottlenecks and optimize AI operations.
The following list outlines the supported LLM providers:
Protecting sensitive data
When working with LLMs, you may need to process Personally Identifiable Information (PII) such as names, email addresses, or financial data. To protect sensitive data and maintain compliance, see Anonymizing PII data for guidance on implementing PII anonymization in your LLM applications.
Troubleshooting
Information on troubleshooting SSL issues.
SSL issues
If SSL handshake or any other issues occur on your LLM applications, such as the following error:
Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.You need to set OTEL_EXPORTER_OTLP_INSECURE=true for your LLM applications to export the data to the gRPC endpoint without using TLS.
Module issues
If module 'lib' or any other issues occur on your LLM applications, such as the following error:
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'.Install the following OpenSSL packages:
pip3 install --upgrade cryptography pyopenssl