Monitoring GPU

You can install the NVIDIA GPU Operator on your GPU environment that helps manage and collect GPU metrics. Enable components such as NVIDIA drivers (to enable CUDA), Kubernetes device plugin for GPUs, and the NVIDIA Container Toolkit if you need them. For more information, see NVIDIA GPU Operator.

helm install gpu-operator \
   --repo https://helm.ngc.nvidia.com/nvidia \
   --namespace gpu-operator \
   --create-namespace \
   -set driver.enabled=false \
   -set toolkit.enabled=false \
   -set devicePlugin.enabled=false \
   -set mig.strategy=single \
   gpu-operator
 

The NVIDIA GPU Operator installs the NVIDIA Data Center GPU Manager (DCGM) by default. DCGM Exporter is a tool to NVIDIA DCGM that allows users to gather GPU metrics and understand workload behavior or monitor GPUs in clusters.

DCGM Exporter exposes GPU metrics at an HTTP endpoint (/metrics) for monitoring solutions. For more information, see DCGM Exporter.

Instana supports MIG (Multi-Instance GPU) through the DCGM Exporter that provides visibility into partitioned GPU usage. Metrics can be collected for each MIG instance independently, helping you monitor and allocate GPU resources more effectively in shared environments.

You can forward the OpenTelemetry data of the GPU to an Instana agent or Instana backend by using the OpenTelemetry Collector.

Instana supports the following models to collect the GPU data:

• Agent model: For this pattern, the GPU data is sent to the Instana agent first, and the agent helps aggregate the data and send to the Instana backend.
• Agentless model: For this pattern, the GPU data is sent to the Instana backend directly without going through the agent.

Instead of specifying static DCGM exporter endpoints, you can configure the OpenTelemetry Collector to automatically discover DCGM exporter pods in your Kubernetes cluster by using kubernetes_sd_configs. This approach removes the need to hard-code individual pod addresses.

receivers:
  otlp:
    protocols:
      grpc: {}
  prometheus/nvidia-dcgm:
    config:
      scrape_configs:
        - job_name: nvidia-dcgm
          scrape_interval: 10s
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: ['nvidia-gpu-operator']
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_app]
              regex: nvidia-dcgm-exporter
              action: keep
            - source_labels: [__meta_kubernetes_pod_container_port_name]
              regex: metrics
              action: keep
            - source_labels: [__meta_kubernetes_pod_node_name]
              target_label: node_name
              action: replace
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: 'DCGM_FI_DEV_(GPU_UTIL|MEM_COPY_UTIL|FB_USED|FB_FREE|GPU_TEMP|POWER_USAGE|SM_CLOCK)|DCGM_FI_PROF_GR_ENGINE_ACTIVE'
              action: keep
processors:
  batch: {}
  transform:
    metric_statements:
      - context: datapoint
        statements:
          - set(resource.attributes["service.name"], Concat(["nvidia-dcgm@", resource.attributes["k8s.node.name"]], "")
          - set(resource.attributes["service.instance.id"], attributes["node_name"])
  resource:
    attributes:
      - action: insert
        from_attribute: net.host.name
        key: server.address
      - action: insert
        from_attribute: net.host.port
        key: server.port
      - action: update
        key: service.name
        value: nvidia-dcgm
      - action: insert
        key: INSTANA_PLUGIN
        value: dcgm

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: "http://<instana-grpc-host>:443"
    headers:
      x-instana-key: <api-key>
      x-instana-host: <instana-host>
  otlphttp:
    endpoint: "https://<instana-http-host>:443"
    headers:
      x-instana-key: <api-key>
      x-instana-host: <instana-host>

service:
  pipelines:
    metrics/nvidia-dcgm:
      receivers:
        - prometheus/nvidia-dcgm
      processors:
        - transform
        - batch
        - resource
      exporters:
        - otlp
        - debug
        - otlphttp
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

After you install OpenTelemetry (OTel) Data Collector, you can view the GPU metrics in the Instana UI.

1. Open the Instana UI, and click Infrastructure. Then, click Analyze Infrastructure.
2. Select OTEL Dcgm from the list of types of the entities.
3. Click the entity instance of OTEL Dcgm entity type to open the associated dashboard.

GPU metrics