Offloading analytics data to Langfuse exporter

Configure the DataPower Interact Gateway Collector to export LLM observability telemetry to Langfuse for AI model monitoring and analysis.

1. Identify the namespace where the collector is deployed.

   kubectl get ns

   Sample output:

   NAME                    STATUS   AGE
   default                 Active   5h6m
   fyre-ci-293280          Active   5h2m

   Note: Lists all namespaces and their deployments, including the DataPower Interact Gateway. In this example, fyre-ci-293280 is the namespace in which the DataPower Interact Gateway is deployed.
2. Create a Kubernetes secret containing the Langfuse configuration.

   kubectl create secret generic langfuse-credentials \
     --from-literal=LANGFUSE_ENDPOINT='https://cloud.langfuse.com/api/public/ingestion' \
     --from-literal=LANGFUSE_SECRET_KEY='sk-lf-...' \
     -n <namespace>

   Important: Replace https://cloud.langfuse.com/api/public/ingestion and sk-lf-... with your actual Langfuse endpoint and secret key.

   Output:

   secret/langfuse-credentials created

3. Verify that the secret was created successfully.

   kubectl get secret langfuse-credentials -n <namespace>

4. Identify the collector deployment.

   kubectl get deployment -n <namespace> | grep collector

   Sample output:

   idig-nanogw-nanogw-analytics-collector   1/1     1     1     5h

5. Edit the collector deployment to inject the secret as environment variables.

   kubectl edit deployment <deployment_name> -n <namespace>

   1. Locate the containers section for the analytics-collector container.
   2. Add the following envFrom section under the container definition:

      envFrom:
        - secretRef:
            name: langfuse-credentials

   3. Save and exit the editor.

      Sample deployment configuration:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: idig-nanogw-nanogw-analytics-collector
        namespace: <namespace>
      spec:
        replicas: 1
        selector:
          matchLabels:
            app.kubernetes.io/name: nanogw-analytics-collector
        template:
          metadata:
            labels:
              app.kubernetes.io/name: nanogw-analytics-collector
          spec:
            containers:
            - name: analytics-collector
              image: your-image:tag
              envFrom:
              - secretRef:
                  name: langfuse-credentials

6. Identify the collector ConfigMap.

   kubectl get configmap -n <namespace> | grep collector

   Sample output:

   idig-nanogw-collector-config                       1      5h11m

7. Edit the collector ConfigMap to add the Langfuse exporter configuration.

   kubectl edit configmap <configmap_name> -n <namespace>

   1. Add the following exporter configuration under the exporters section:

      exporters:
        otlphttp/langfuse:
          endpoint: '${LANGFUSE_ENDPOINT}'
          headers:
            Authorization: 'Bearer ${LANGFUSE_SECRET_KEY}'
          tls:
            insecure: false

   2. Update the service pipelines to include the Langfuse exporter:

      service:
        pipelines:
          traces/langfuse:
            receivers: [otlp]
            processors: [batch/apic]
            exporters: [otlphttp/langfuse]
      
          logs/langfuse:
            receivers: [otlp]
            processors: [batch/apic]
            exporters: [otlphttp/langfuse]

   3. Save the ConfigMap.

      Sample ConfigMap structure:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: idig-nanogw-collector-config
        namespace: <namespace>
      data:
        config.yaml: |
          receivers:
            ...
      
          processors:
            batch/apic:
              ...
      
          exporters:
            otlphttp/langfuse:
              endpoint: '${LANGFUSE_ENDPOINT}'
              headers:
                Authorization: 'Bearer ${LANGFUSE_SECRET_KEY}'
              tls:
                insecure: false
      
          service:
            pipelines:
              traces/langfuse:
                receivers: [otlp]
                processors: [batch/apic]
                exporters: [otlphttp/langfuse]
      
              logs/langfuse:
                receivers: [otlp]
                processors: [batch/apic]
                exporters: [otlphttp/langfuse]

8. Restart the collector deployment to apply the changes.

   kubectl rollout restart deployment <deployment_name> -n <namespace>

   Output:

   deployment.apps/idig-nanogw-nanogw-analytics-collector restarted

9. Verify that the collector pods restart successfully.

   kubectl get pods -n <namespace> | grep collector

   Sample output:

   idig-nanogw-nanogw-analytics-collector-7d4f8ccf45-bk468   1/1   Running   0   30s

   The OpenTelemetry Collector is now configured to export telemetry data to Langfuse. Traces and logs will be sent to your Langfuse instance for LLM observability and analysis.