Monitoring AI model platforms

AI model platforms are critical for deploying, managing, and scaling machine learning (ML) and large language models (LLMs) in production environments. Platforms like watsonx enable organizations to build, train, and serve AI models efficiently, handling tasks such as inference requests, data integration, and model governance. Monitoring these platforms is critical to make sure optimal performance, reliability, and cost efficiency. Key metrics to track include model inference latency, error rates, resource utilization (for example, CPU, GPU, and memory), input/output tokens usage, and corresponding cost. By effectively monitoring, you can identify bottlenecks, detect anomalies, and make sure that AI models deliver accurate and timely results. This precision is crucial for maintaining user trust and achieving business goals.

Instana enables comprehensive monitoring of AI model platforms by collecting traces, metrics, and logs to provide real-time insights into their performance, efficiency, and cost. Instana monitors the underlying LLMs, and identifies abnormal trends such as increased response times or failures while tracking usage costs. By integrating with OpenTelemetry (OTel), Instana collects telemetry data from AI agents and LLMs, enabling the organizations to diagnose issues, optimize performance, and maintain cost-effective AI operations within a unified observability platform.

The following list outlines the supported AI model platforms:

Adding LLM security

Revealing Personally Identifiable Information (PII) to LLMs can lead to serious security and privacy risks. These risks include violating contractual obligations and increased chances of data leakage or a data breach.

Anonymization of PII data

You can anonymize PII data by replacing the original data with artificial data or rearranging data set attributes in ways that differ from their original form.

LLM-Guard ensures interactions with LLMs remain safe and secure.

To install LLM Guard, run the following command in your terminal:

pip3 install llm-guard

The following code snippet is an example of adding security (that uses llm_guard) to the application:


from llm_guard.input_scanners import Anonymize
from llm_guard.input_scanners.anonymize_helpers import BERT_LARGE_NER_CONF
from llm_guard.output_scanners import Deanonymize
from llm_guard.vault import Vault

vault = Vault()
Traceloop.init(app_name="watsonx_chat_security_service")

#call anonymize() on prompt to replace the sensitive data with synthetic data 
@workflow(name="anonymize")
def anonymize(input: str):
  scanner = Anonymize(vault, preamble="Insert before prompt", allowed_names=["John Doe"], hidden_names=["Test LLC"],recognizer_conf=BERT_LARGE_NER_CONF, language="en")
  sanitized_prompt, is_valid, risk_score = scanner.scan(prompt)
  return sanitized_prompt

#call deanonymize() on the response to replace the synthetic data with original data 
@workflow(name="deanonymize")
def deanonymize(sanitized_prompt: str, answer: str):
  scanner = Deanonymize(vault)
  sanitized_model_output, is_valid, risk_score = scanner.scan(sanitized_prompt, answer)
  return sanitized_model_output

Troubleshooting

SSL issues

In the case of SSL handshake issues or similar ones with your LLM applications, such as the following error:

Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

You need to set OTEL_EXPORTER_OTLP_INSECURE=true for your LLM applications to export the data to the gRPC endpoint without using TLS.

Module issues

In the case of module 'lib' issue with your LLM applications, such as the following error:

AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'.

You need to install the following OpenSSL packages:

pip3 install --upgrade cryptography pyopenssl