Configuration

The collector is configured entirely via environment variables. All variables follow the standard OpenTelemetry SDK configuration spec where applicable.

Collector mode

OTEL_GPU_COLLECTOR_MODE selects a use-case preset (all | light | serving | training | deep). Default: all. Presets only fill unset feature envs; explicit flags always win. See Modes for the full matrix, cost notes, and hosting checklist.

Variable	Default	Description
OTEL_GPU_COLLECTOR_MODE	all	Feature preset. Invalid values warn and fall back to all

Environment Variables

Variable	Default	Description
OTEL_EXPORTER_OTLP_ENDPOINT	(required)	OTLP endpoint URL, e.g. http://localhost:4318
OTEL_EXPORTER_OTLP_HEADERS		Auth headers in key=val,key2=val2 format
OTEL_EXPORTER_OTLP_PROTOCOL	grpc	grpc or http/protobuf
OTEL_SERVICE_NAME	(unset)	Optional. Only set if your backend requires service.name. This agent is not an application service — product identity uses telemetry.distro.name=opentelemetry-gpu-collector and telemetry.distro.version
OTEL_RESOURCE_ATTRIBUTES	deployment.environment=default	Resource attributes (key=val,...). Prefer setting host.name, k8s.*, cloud.provider, host.type, cloud.region here — overrides auto-detect. Org tags such as team or datacenter are not auto-detected; set them here.
OTEL_METRIC_EXPORT_INTERVAL	60000	Metric polling interval in milliseconds. For self-hosted LLM hosts, 15000 is recommended
OTEL_GPU_EBPF_ENABLED	mode-dependent (true on Linux for serving+)	eBPF CUDA activity tracing + stream-sync occupancy (Linux/NVIDIA only). Discovers libcudart and libcuda via FS + /proc maps (fleet-friendly with host PID). Soft-fails without caps; set false to disable
SHIELD360_HOST_METRICS	true	Collect system + collector-process host metrics. Set false for a GPU-only light footprint
OTEL_GPU_FS_TYPES_EXCLUDE	squashfs,erofs,iso9660,cramfs,romfs,cd9660,CDFS,UDF	Filesystem types excluded from system.filesystem.* metrics (case-sensitive). Default skips image-based and optical filesystems that are 100% full by construction (e.g. snap mounts). Set to an empty string to report all types
OTEL_GPU_PROCESS_CMDLINE	true	Export truncated process.command_line on GPU process metrics
OTEL_GPU_PROCESS_CMDLINE_MAX_LEN	512	Max characters for process.command_line
OTEL_GPU_ALLOCATED_UTIL_THRESHOLD	0.05	Util threshold (0–1) used with process memory for hw.gpu.allocated
OTEL_GPU_INTERCONNECT_ENABLED	true	Export NVLink/XGMI interconnect throughput when available
K8S_NODE_NAME		Kubernetes node name via downward API (spec.nodeName). Also accepts OTEL_RESOURCE_ATTRIBUTES_NODE_NAME (Operator) or legacy NODE_NAME
K8S_CLUSTER_NAME		Explicit cluster name when cloud auto-detect fails (on-prem). Alias: SHIELD360_K8S_CLUSTER_NAME. Only applied in Kubernetes
SHIELD360_K8S_NODE_LOOKUP	true	When false, skip GET /api/v1/nodes/$K8S_NODE_NAME for instance-type / provider discovery
SHIELD360_K8S_POD_RESOURCES	true in K8s	Use kubelet PodResources socket; joins GPU UUID → k8s.pod.name / namespace / container
SHIELD360_K8S_POD_LOOKUP	true in K8s when K8S_NODE_NAME is set	List pods on this node via the Kubernetes API (needs list on pods) for UID/container-id joins
POD_RESOURCES_SOCKET	OS default	Override kubelet PodResources socket / named pipe path
SHIELD360_CLOUD_DETECT	true	When false, skip AWS/GCP/Azure IMDS probes (recommended on bare metal to avoid link-local timeouts)
OTEL_GPU_DCGM_ENABLED	mode-dependent (true for serving+)	Optional NVIDIA DCGM profiling (soft-skipped without libdcgm / when DCGM cannot init)
OTEL_GPU_DCGM_LIB_PATH	/lib64/libdcgm.so	Path to libdcgm
OTEL_GPU_DCGM_ADDRESS		Empty = embedded; else hostengine address
OTEL_GPU_DCGM_INTERVAL	10	DCGM sample interval in seconds (warns if < 10s; DCP multiplexing)
OTEL_GPU_DCGM_PREFER	mode-dependent (true for serving+)	When true and DCGM is up: DCGM owns overlapping hw.gpu.io / interconnect / power / util / SM clock / hw.gpu.memory.controller.utilization; NVML suppresses those series. Spec hw.gpu.memory.utilization (usage/limit) stays on the vendor path
OTEL_GPU_DCGM_FIELDS	50,100,155,203,204,1001–1012	Comma-separated DCGM field IDs (155=power, 203=GPU util, 204=mem util, 1001–1012=DCP)
OTEL_GPU_RDC_ENABLED	mode-dependent (true for all)	Optional AMD RDC profiling (soft-skipped without librdc)
OTEL_GPU_RDC_LIB_PATH	librdc.so	Path / soname for librdc
OTEL_GPU_KINETO_ENABLED	mode-dependent (true for deep/all)	On-demand Kineto IPC + control profiling
OTEL_GPU_PROMETHEUS_ADDR		When set (e.g. :9464), serve Prometheus /metrics alongside OTLP
OTEL_GPU_CONTROL_ADDR	mode-dependent (127.0.0.1:1919 for deep/all)	Loopback control HTTP; empty disables
OTEL_GPU_CONTROL_TOKEN		Optional Bearer token for control API (required if allow-remote)
OTEL_GPU_CONTROL_ALLOW_REMOTE	false	Allow non-loopback bind for multi-node fan-out (requires token)
OTEL_HOST_CPU_HIGHRES	mode-dependent (true for deep/all)	100ms CPU ring buffer for GET /v1/cpu/highres (requires control addr; Linux)
OTEL_HOST_PMU_ENABLED	mode-dependent (true for training+)	Core CPU PMU counters (hw.cpu.instructions, …)
OTEL_HOST_PMU_EVENTS	instructions,cycles or +memory_bandwidth in training+	Include memory_bandwidth / uncore for IMC hw.cpu.memory.io
OTEL_HOST_INTEL_PT_ENABLED	mode-dependent (true for deep/all)	On-demand Intel PT via POST /v1/profile/cpu/pt (needs perf + intel_pt)
OTEL_HOST_INTEL_PT_MAX_DURATION_MS	2000	Hard cap for PT capture duration
OTEL_HOST_INTEL_PT_MAX_CPUS	4	Max CPUs traced per PT capture
OTEL_HOST_INTEL_PT_MAX_BUFFER_PAGES	64	AUX mmap pages passed to perf -m
OTEL_TPU_ENABLED	mode-dependent (true for all)	Scrape TPU Prometheus endpoint
OTEL_HOST_KVM_ENABLED	mode-dependent (true for all)	KVM exit / VM metrics
OTEL_HOST_INTERRUPTS_ENABLED	mode-dependent (true for all)	Host interrupt counters
OTEL_HOST_NIC_ENABLED	mode-dependent (true for training+)	Per-NIC hw.network.* (ethtool/sysfs). When enabled, system.network.* is skipped to avoid duplicates
OTEL_HOST_RDMA_ENABLED	mode-dependent (true for training+)	RDMA device counters

Host, Kubernetes, and cloud identity

Resource attributes follow OpenTelemetry semantic conventions for host, K8s, and cloud:

Attribute	When set
host.name	Always (from OTEL_RESOURCE_ATTRIBUTES, or K8S_NODE_NAME → GCE hostname in K8s → OS hostname)
k8s.node.name	When a node env is set (K8S_NODE_NAME / Operator / legacy NODE_NAME)
k8s.cluster.name	In Kubernetes only: K8S_CLUSTER_NAME / OTEL_RESOURCE_ATTRIBUTES → GKE / AKS / EKS metadata
cloud.provider	Auto: K8s Node.spec.providerID → AWS/GCP/Azure IMDS → DMI vendor hint
cloud.platform	e.g. aws_eks, gcp_kubernetes_engine, aws_ec2, azure_aks
host.type	Instance type (e.g. g4dn.xlarge, a2-highgpu-1g, Standard_NC6s_v3) from node labels or IMDS
cloud.region / cloud.availability_zone	Topology labels or IMDS
cloud.account.id	AWS account / GCP project / Azure subscription when available
host.id	Cloud instance ID from providerID or IMDS
shield360.host.type.source	Which tier filled host.type: k8s_label, imds, or dmi

Discovery order (later tiers only fill missing fields; never blocks startup; never emits "unknown"):

1. Explicit OTEL_RESOURCE_ATTRIBUTES (wins via SDK WithFromEnv)
2. Kubernetes Node GET (needs get on nodes + K8S_NODE_NAME) — OpenCost-style labels / providerID
3. Parallel AWS / GCP / Azure instance metadata (short timeout)
4. DMI sys_vendor hint for provider only

Use cloud.provider + host.type + cloud.region as join keys for future UI cost attribution. The collector does not compute prices.

Kubernetes is detected via KUBERNETES_SERVICE_HOST. On GKE, AKS, and EKS the cluster name is read from the instance metadata service (short timeout; failures are ignored). Self-managed clusters should set k8s.cluster.name via OTEL_RESOURCE_ATTRIBUTES or K8S_CLUSTER_NAME.

Recommended Kubernetes DaemonSet (OTel-native)

env:
  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "host.name=$(K8S_NODE_NAME),k8s.node.name=$(K8S_NODE_NAME)"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector:4317"
  # Optional when cloud auto-detect is unavailable (on-prem / self-managed):
  # - name: K8S_CLUSTER_NAME
  #   value: my-cluster
  # Or append to OTEL_RESOURCE_ATTRIBUTES:
  #   ,k8s.cluster.name=my-cluster,cloud.provider=aws,host.type=g4dn.xlarge,cloud.region=us-east-1

If you only set K8S_NODE_NAME (without packing it into OTEL_RESOURCE_ATTRIBUTES), the collector still maps it to host.name and k8s.node.name automatically.

For K8s node label / providerID discovery, grant the DaemonSet ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-gpu-collector-node-get
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-gpu-collector-node-get
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-gpu-collector-node-get
subjects:
  - kind: ServiceAccount
    name: otel-gpu-collector
    namespace: monitoring

Note

On EKS, pods without hostNetwork may fail IMDSv2 when the node httpPutResponseHopLimit is 1. Prefer K8s node lookup (above), raise the hop limit to 2+, or run with hostNetwork: true. Timeouts are soft-fail and do not stop the collector.

Common configurations

Minimal - send to a local OTel Collector

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector

Production - with service name, environment, and auth header

OTEL_SERVICE_NAME=gpu-worker \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=ml \
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.example.com:4317 \
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer\ my-token \
OTEL_METRIC_EXPORT_INTERVAL=30000 \
./opentelemetry-gpu-collector

HTTP/protobuf instead of gRPC

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
./opentelemetry-gpu-collector

Disable eBPF CUDA tracing

On Linux, eBPF CUDA tracing is on by default. It discovers libcudart and libcuda from the filesystem and from /proc/*/maps (no CUDA volume mount required when Docker --pid=host / Kubernetes hostPID: true is set). Soft-fails without CAP_BPF + CAP_PERFMON (or root). Containers typically also need --ulimit memlock=-1:-1 so BPF maps can be created. Set false to skip:

OTEL_GPU_EBPF_ENABLED=false \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector

eBPF activity/occupancy is NVIDIA/CUDA only. AMD and Intel use the same host-PID process attribution for DRM fdinfo metrics; they do not need libcudart.

Notes

• OTEL_METRIC_EXPORT_INTERVAL is in milliseconds per the OTel spec. For a 30-second interval, set 30000.
• deployment.environment is extracted from OTEL_RESOURCE_ATTRIBUTES and attached as a resource attribute. Any key-value pairs in OTEL_RESOURCE_ATTRIBUTES are also forwarded to the OTel SDK resource via resource.WithFromEnv().
• If OTEL_EXPORTER_OTLP_ENDPOINT is not set, the collector starts but no metrics are exported. Check the logs for a warning.
• Auto-detected host.name / k8s.* attributes are logged at startup as resolved resource identity.