Overview
The OpenTelemetry GPU Collector is a lightweight, single-binary metrics collector written in Go. It exports GPU hardware telemetry, host system metrics, and process metrics via OpenTelemetry (OTLP) - with no Python dependencies, no DCGM daemon, and no vendor-specific agents.
It is fully configured via standard OpenTelemetry environment variables and follows the OTel semantic conventions for hardware metrics. Use collector modes (OTEL_GPU_COLLECTOR_MODE, default all) to turn optional features on by use case.
Resource identity models this process as a GPU metrics agent (host + accelerator telemetry), not an application service: telemetry.sdk.*, telemetry.distro.name=opentelemetry-gpu-collector, telemetry.distro.version, plus host/k8s/cloud attributes. Set OTEL_SERVICE_NAME only if your backend requires service.name.
- OpenTelemetry-native - uses standard
OTEL_*env vars, exports via OTLP gRPC or HTTP to any OTel-compatible backend - Cross-vendor GPU support - NVIDIA (NVML), AMD, and Intel on Linux and Windows from a single binary
- OTel semantic conventions -
hw.gpu.*metric names,hw.id/hw.name/hw.vendorattributes per spec - Zero dependencies - no DCGM, no Python, no CUDA toolkit needed at runtime for hardware metrics
- Resilient - continues exporting host metrics even when no GPUs are present; retries GPU discovery every 30s
What it collects
Section titled “What it collects”Utilization, memory, temperature, power, energy, clocks, PCIe/interconnect throughput, throttle, ECC/XID/RAS — NVIDIA, AMD, and Intel on Linux and Windows
Per-process VRAM/util plus cmdline, OS state (zombie), owner, uptime, and LLM framework labels (vllm, ollama, …). Kubernetes pod name/namespace via PodResources
Kernel launch counts, grid/block sizes, graph replays, memory allocations, and memory copies via uprobes on libcudart.so (runtime) and libcuda.so (driver) — covers llama.cpp/Ollama as well as PyTorch (on by default on Linux; discovers libs from /proc with host PID — no CUDA mount)
GPU vendor support
Section titled “GPU vendor support”| Vendor | Linux | Windows |
|---|---|---|
| NVIDIA | NVML (+ MIG, NVLink, XID events) | NVML (nvml.dll, NVLink) + PDH process attribution (MIG not available on Windows) |
| AMD | sysfs/hwmon/DRM + libamd_smi (PCIe/XGMI/throttle/RAS when present) | DXGI + PDH (incl. encode/decode) + ADL (util/temp/clocks/fan) |
| Intel | DRM/hwmon + Level Zero Sysman (util/VRAM/power/media) | DXGI + PDH (incl. encode/decode) + Level Zero Sysman |
Platform support
Section titled “Platform support”| Feature | Linux | macOS | Windows |
|---|---|---|---|
| System metrics (CPU, memory, disk, network) | Yes | Yes | Yes |
| Process metrics (CPU, memory, threads, FDs) | Yes | Yes | Yes |
| GPU device + process metrics | Yes | - | Yes |
| eBPF CUDA tracing / occupancy | Yes | - | - |
How it works
Section titled “How it works”Host Metrics (all platforms via gopsutil) +-- CPU utilization, memory, disk I/O, filesystem, network +-- Process: self CPU, memory, threads, FDs, Go runtime
GPU Metrics (Linux + Windows) +-- Linux: PCI scan → NVML / sysfs / DRM (+ eBPF CUDA, default on) +-- Windows: NVML.dll / DXGI + PDH (no eBPF)
Export +-- OTel SDK --> OTLP gRPC/HTTP --> your OTel collector / backendOn Linux, the collector scans /sys/bus/pci/devices/ for GPU class codes. On Windows, it loads nvml.dll when present and enumerates adapters via DXGI.
Each detected GPU is handed to its vendor-specific backend. NVIDIA uses NVML. Linux AMD/Intel read sysfs/hwmon; Windows AMD/Intel use DXGI memory totals and PDH for utilization and per-process attribution.
Observable gauge and counter instruments are registered with the OTel SDK meter. On each collection tick, the SDK calls back into the collector to read fresh values from each GPU.
Metrics are exported via OTLP to any compatible backend - Shield360, Grafana, Datadog, New Relic, or a standard OTel Collector.