Installation

Docker (recommended)

The easiest way to run the collector. The image is published to GitHub Container Registry and supports linux/amd64 and linux/arm64.

docker pull ghcr.io/thinkfleetai/otel-gpu-collector:latest

For per-process GPU attribution (cmdline, PID, zombie/process.state, owner), run with host PID namespace access: Docker --pid=host, Compose pid: host, or Kubernetes hostPID: true. Device-level hw.gpu.* metrics work without it.

Tags

Tag	Description
latest	Most recent release
1.2.3	Specific version
1.2	Latest patch of a minor version

---

Binary from the image

On Linux, copy the binary out of the published image instead of running it in a container:

docker create --name otel-gpu-collector-copy ghcr.io/thinkfleetai/otel-gpu-collector:latest
docker cp otel-gpu-collector-copy:/usr/local/bin/opentelemetry-gpu-collector ./opentelemetry-gpu-collector
docker rm otel-gpu-collector-copy

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector

Pull the image for the host’s architecture (linux/amd64 or linux/arm64; add --platform to choose another). The binary is built against glibc (Debian bookworm) and loads libnvidia-ml.so from the host for NVIDIA GPUs.

Note

GPU device and process metrics work on Linux and Windows (NVIDIA via NVML; AMD/Intel via sysfs on Linux or DXGI+PDH on Windows). eBPF CUDA tracing and occupancy are Linux only. Windows and macOS binaries are not published for download yet: ask support@shield360.ai if you need one.

---

Kubernetes DaemonSet

Run one collector per GPU node. Use the OpenTelemetry-recommended pattern: downward API → K8S_NODE_NAME → OTEL_RESOURCE_ATTRIBUTES with host.name and k8s.node.name. On GKE, AKS, and EKS the collector also auto-detects k8s.cluster.name, cloud.provider, and host.type (instance type) from the Kubernetes Node object and/or cloud metadata.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-gpu-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-gpu-collector-node-get
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-gpu-collector-node-get
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-gpu-collector-node-get
subjects:
  - kind: ServiceAccount
    name: otel-gpu-collector
    namespace: default # change to your namespace
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-gpu-collector
spec:
  selector:
    matchLabels:
      app: otel-gpu-collector
  template:
    metadata:
      labels:
        app: otel-gpu-collector
    spec:
      serviceAccountName: otel-gpu-collector
      hostPID: true
      containers:
        - name: otel-gpu-collector
          image: ghcr.io/thinkfleetai/otel-gpu-collector:latest
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: OTEL_SERVICE_NAME
              value: otel-gpu-collector
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "host.name=$(K8S_NODE_NAME),k8s.node.name=$(K8S_NODE_NAME)"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector:4317"
            # Optional for on-prem / self-managed clusters:
            # - name: K8S_CLUSTER_NAME
            #   value: my-cluster
            # eBPF CUDA tracing is on by default on Linux; set false to disable:
            # - name: OTEL_GPU_EBPF_ENABLED
            #   value: "false"
          volumeMounts:
            - name: pod-resources
              mountPath: /var/lib/kubelet/pod-resources
              readOnly: true
            # AMD/Intel DRM:
            # - name: dri
            #   mountPath: /dev/dri
          securityContext:
            capabilities:
              add: ["SYS_ADMIN"] # or privileged / CAP_BPF+CAP_PERFMON for eBPF
      volumes:
        - name: pod-resources
          hostPath:
            path: /var/lib/kubelet/pod-resources
        # - name: dri
        #   hostPath:
        #     path: /dev/dri

Note

hostPID: true (same role as Docker --pid=host) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under /proc. Without it, device-level hw.gpu.* metrics still work. nodes/get enables OpenCost-style host.type / cloud.provider discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1). pods/list is what SHIELD360_K8S_POD_LOOKUP uses to turn a process’s cgroup pod UID into k8s.pod.name. The list call is field-selected to this node; the permission is cluster-scoped.

See Configuration for identity env vars and detection order.

---

Upgrade

Docker

docker pull ghcr.io/thinkfleetai/otel-gpu-collector:latest
docker stop otel-gpu-collector
docker rm otel-gpu-collector
# re-run with same flags

Binary

Copy the new binary out of the new image (see Binary from the image), replace the existing file, and restart the process.

Uninstall

Docker

docker stop otel-gpu-collector
docker rm otel-gpu-collector
docker rmi ghcr.io/thinkfleetai/otel-gpu-collector:latest

Binary

rm /usr/local/bin/opentelemetry-gpu-collector

---

Troubleshooting

No GPU metrics - collector starts but reports no hw.gpu.* metrics

• Confirm the host has a supported GPU: lspci | grep -E 'VGA|3D|Display'
• For NVIDIA: verify libnvidia-ml.so is present: ldconfig -p | grep nvidia-ml
• For Docker: ensure --gpus all (NVIDIA) or --device /dev/dri (AMD/Intel) is passed
• Check logs: docker logs otel-gpu-collector for "discovered GPU" entries

eBPF tracing not working

• On Linux it is enabled by default; confirm it is not disabled via OTEL_GPU_EBPF_ENABLED=false
• Check kernel version: uname -r (requires 5.8+)
• The process needs CAP_BPF and CAP_PERFMON, or run as root
• For Docker: --cap-add CAP_BPF --cap-add CAP_PERFMON, --pid=host, and --ulimit memlock=-1:-1 (BPF maps need locked memory; no CUDA mount needed)
• For Kubernetes: hostPID: true plus BPF capabilities (or SYS_ADMIN); raise memlock if map create returns EPERM
• If no CUDA process is running yet, the collector rescans /proc every 30s
• Verify a workload has loaded CUDA: grep -E 'libcudart|libcuda.so' /proc/*/maps 2>/dev/null | head

Connection refused / no data reaching backend

• Verify OTEL_EXPORTER_OTLP_ENDPOINT is reachable from the container: curl http://<endpoint>/health
• For Docker networking: use the host IP or service name, not localhost
• Check if gRPC vs HTTP/protobuf matches the backend: set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf for HTTP backends (port 4318)

Intel GPU not detected

• Verify the i915 or Xe driver is loaded: lsmod | grep -E 'i915|xe'
• Check DRM entries exist: ls /sys/class/drm/
• Requires Linux kernel 5.10+ for sysfs metric exposure
• Fan speed requires kernel 6.16+