NVIDIA GPUs

The collector monitors NVIDIA GPUs via NVML using the go-nvml library, which loads libnvidia-ml.so at runtime. No CUDA toolkit or DCGM daemon is needed.

Requirements

• Linux with NVIDIA GPU drivers installed
• libnvidia-ml.so present on the host (installed with the NVIDIA driver)
• For Docker: NVIDIA Container Toolkit

Collected metrics

Metric	Description
hw.gpu.utilization	Compute, encoder, and decoder utilization (0.0–1.0) via hw.gpu.task
hw.gpu.memory.utilization	Fraction of GPU memory used (usage / limit)
hw.gpu.memory.controller.utilization	Memory controller busy fraction (extension)
hw.gpu.memory.limit	Total VRAM (bytes)
hw.gpu.memory.usage	Used VRAM (bytes)
hw.gpu.memory.free	Free VRAM (bytes)
hw.temperature	Die and memory temperature (°C) via hw.sensor_location
hw.fan.speed_ratio	Fan speed as fraction of max (NVML %; rpm not available)
hw.power	Current power draw (W)
hw.power.limit	Power cap (W)
hw.energy	Cumulative energy (J)
hw.gpu.speed	Clock frequency in Hz (hw.gpu.clock_domain=graphics/sm/memory)
hw.status	Hardware status (ok / degraded / failed)
hw.errors	ECC correctable/uncorrectable errors and PCIe replay errors

Docker

docker run -d \
  --name otel-gpu-collector \
  --gpus all \
  --pid=host \
  -e OTEL_SERVICE_NAME=my-app \
  -e OTEL_RESOURCE_ATTRIBUTES='deployment.environment=production' \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
  ghcr.io/thinkfleetai/otel-gpu-collector:latest

--pid=host is required for per-process GPU attribution (cmdline, PID, zombie state).

Docker Compose

services:
  otel-gpu-collector:
    image: ghcr.io/thinkfleetai/otel-gpu-collector:latest
    pid: host
    environment:
      OTEL_SERVICE_NAME: my-app
      OTEL_RESOURCE_ATTRIBUTES: deployment.environment=production
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: always

Kubernetes (DaemonSet)

To monitor GPUs on every node in a cluster, deploy the collector as a DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-gpu-collector
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: otel-gpu-collector
  template:
    metadata:
      labels:
        app: otel-gpu-collector
    spec:
      hostPID: true
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: collector
          image: ghcr.io/thinkfleetai/otel-gpu-collector:latest
          env:
            - name: OTEL_SERVICE_NAME
              value: gpu-collector
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: deployment.environment=production
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector.monitoring.svc.cluster.local:4318
          resources:
            limits:
              nvidia.com/gpu: 1
          securityContext:
            privileged: false

Note

The collector does not need privileged mode for NVML device metrics — just access to libnvidia-ml.so. Per-process attribution needs Docker --pid=host or Kubernetes hostPID: true. eBPF tracing / stream-sync occupancy requires CAP_BPF + CAP_PERFMON (or privileged) and raised memlock (--ulimit memlock=-1:-1).

---

Metrics reference

Full metrics list with types, units, and attributes

Configuration

All environment variables and defaults