Metrics Reference

All metric names and attributes follow the OpenTelemetry semantic conventions for hardware and system metrics. See the semconv registry for the full instrument list.

GPU Hardware Telemetry

Collected for each detected GPU on Linux and Windows. Availability depends on vendor, OS, and GPU model.

Device metrics

Metric	Type	Unit	Description	NVIDIA	AMD	Intel
hw.gpu.utilization	Gauge	1	GPU compute/encoder/decoder utilization (0.0–1.0)	Yes	Yes	Yes*
hw.gpu.memory.utilization	Gauge	1	Fraction of GPU memory used (usage / limit)	Yes	Yes	Yes*
hw.gpu.memory.controller.utilization	Gauge	1	Memory controller busy fraction (extension; NVML/AMD)	Yes	Yes	-
hw.gpu.memory.limit	UpDownCounter	By	Total GPU memory	Yes	Yes	Yes*
hw.gpu.memory.usage	UpDownCounter	By	Used GPU memory	Yes	Yes	Yes*
hw.gpu.memory.free	UpDownCounter	By	Free GPU memory	Yes	Yes	Yes*
hw.temperature	Gauge	Cel	Die or memory temperature	Yes	Yes	Yes
hw.fan.speed	Gauge	rpm	Fan speed	-†	Yes	Yes*
hw.fan.speed_ratio	Gauge	1	Fan speed as fraction of max	Yes	-	-
hw.power	Gauge	W	Current power draw	Yes	Yes	Yes
hw.power.limit	Gauge	W	Power limit/cap	Yes	Yes	Yes
hw.energy	Counter	J	Cumulative energy consumed	Yes	Yes	Yes
hw.gpu.speed	Gauge	Hz	Clock frequency (hw.gpu.clock_domain)	Yes	Yes	Yes*
hw.gpu.io	Counter	By	Cumulative PCIe I/O bytes	Yes	Yes¶	Soft‡
hw.gpu.interconnect.io	Counter	By	Cumulative NVLink / XGMI bytes	Yes§	Yes¶	-
hw.status	UpDownCounter	1	Hardware status (up / throttled states)	Yes	Yes	Yes
hw.gpu.allocated	Gauge	1	1 when process memory or util ≥ threshold	Yes	Yes	Yes
hw.gpu.idle	Gauge	1	Idle ratio (1 - utilization) when util known	Yes	Yes	Yes*
hw.errors	Counter	{error}	ECC, PCIe, XID, and RAS error counts	Yes	Yes (RAS + PCIe replay¶)	-

* Intel support depends on driver (i915/Xe) and kernel version.

† NVIDIA NVML reports fan speed as a percentage — use hw.fan.speed_ratio; hw.fan.speed (rpm) is omitted for NVIDIA.

§ Requires OTEL_GPU_INTERCONNECT_ENABLED (default true).

¶ AMD XGMI / PCIe when the driver exposes counters.

‡ Soft / best-effort when the kernel exposes the counters.

* Intel device util/VRAM/power/throttle/media use Level Zero Sysman when available (Linux/Windows); otherwise DRM/hwmon/PDH subset. Encoder/decoder util comes from media engine groups or Windows PDH VideoEncode/VideoDecode.

† NVIDIA NVML exposes fan speed as a percentage, not RPM, so hw.fan.speed is not emitted for NVIDIA — use hw.fan.speed_ratio instead.

‡ Soft-omitted when the vendor library/driver path does not expose the counter. Core util/memory/power still collect without those libraries. Windows AMD has no PCIe/XGMI throughput API (ADL).

§ NVLink on Linux and Windows NVIDIA when the GPU exposes enabled NVLink links (rates after the second sample).

¶ AMD Linux via libamd_smi (amdsmi_get_gpu_pci_throughput, amdsmi_get_link_metrics, amdsmi_get_violation_status) when present; soft-omitted on VMs/consumer cards without those counters. AMD media util (MmActivity) is exported as hw.gpu.task=encoder when the driver only exposes a combined VCN counter (decoder stays unset).

Attributes

All GPU metrics carry these base attributes:

Attribute	Description	Example
hw.id	Unique device identifier (required by spec)	GPU-a1b2c3d4-5678-...
hw.name	Product name	NVIDIA A100-SXM4-80GB
hw.vendor	Vendor name	nvidia, amd, intel
gpu.index	Zero-based device index	0, 1
gpu.pci_address	PCI bus address	0000:01:00.0

Additional per-metric attributes:

Metric	Attribute	Values
hw.gpu.utilization	hw.gpu.task	general, encoder, decoder
hw.temperature	hw.sensor_location	die, memory
hw.gpu.speed	hw.gpu.clock_domain	graphics, memory, sm
hw.gpu.io	network.io.direction	receive, transmit
hw.gpu.interconnect.io	network.io.direction	receive, transmit
hw.gpu.interconnect.io	hw.gpu.interconnect.type	nvlink, xgmi, other
hw.status	status / throttle attrs	up / throttled + reason labels
hw.errors	error.type	corrected, uncorrected, pcie_replay, xid, ras_corrected, ras_uncorrected
hw.errors	hw.type	gpu
MIG devices	gpu.mig.enabled, gpu.mig.device_id, gpu.parent.uuid, gpu.mig.instance_id, gpu.mig.profile	Linux NVIDIA only (MIG is not available on Windows)
Partitions (RDC)	hw.parent, hw.gpu.partition	AMD compute partitions when available

---

System Metrics

Collected on all platforms (Linux, macOS, Windows) via gopsutil. Follows the OTel semantic conventions for system metrics.

Metric	Type	Unit	Description	Attributes
system.cpu.utilization	Gauge	1	CPU utilization per logical core (0.0–1.0)	cpu.logical_number
system.cpu.logical.count	UpDownCounter	{cpu}	Number of logical CPU cores
system.memory.usage	UpDownCounter	By	Memory bytes by state	system.memory.state={used,free,cached,buffers}
system.memory.utilization	Gauge	1	Memory utilization (0.0–1.0)
system.disk.io	Counter	By	Disk I/O bytes	system.device, disk.io.direction={read,write}
system.disk.operations	Counter	{operation}	Disk I/O operations	system.device, disk.io.direction={read,write}
system.filesystem.usage	UpDownCounter	By	Filesystem space by state	system.device, system.filesystem.mountpoint, system.filesystem.type, system.filesystem.state={used,free}
system.filesystem.utilization	Gauge	1	Filesystem utilization (0.0–1.0)	system.device, system.filesystem.mountpoint, system.filesystem.type
system.network.io	Counter	By	Network I/O bytes	network.interface.name, network.io.direction={receive,transmit}
system.network.errors	Counter	{error}	Network errors	network.interface.name, network.io.direction={receive,transmit}

Note

system.memory.state values cached and buffers are only reported on Linux. Loopback interfaces (lo, lo0) are excluded from network metrics.

---

Process Metrics

Self-monitoring of the collector process. Follows the OTel semantic conventions for process metrics.

Metric	Type	Unit	Description	Attributes
process.cpu.time	Counter	s	Cumulative CPU time	cpu.mode={user,system}
process.cpu.utilization	Gauge	1	CPU utilization (0.0–1.0)
process.memory.usage	UpDownCounter	By	Resident memory (RSS)
process.memory.virtual	UpDownCounter	By	Virtual memory size
process.thread.count	UpDownCounter	{thread}	OS thread count
process.unix.file_descriptor.count	UpDownCounter	{file_descriptor}	Open file descriptors (Linux/macOS)
process.runtime.go.goroutines	Gauge	{goroutine}	Go goroutine count
process.runtime.go.mem.heap_alloc	Gauge	By	Go heap memory allocated

---

Per-process / per-pod GPU attribution

Collected on each scrape from NVML (NVIDIA) or DRM fdinfo (AMD/Intel). Requires host PID visibility — Docker --pid=host / Compose pid: host, or Kubernetes hostPID: true — so /proc shows workload PIDs. Without it, device-level hw.gpu.* metrics still work, but per-process cmdline, state (including zombies), and GPU process metrics will be missing or incomplete.

These attributes support views like zombie-process triage: filter process.state == "zombie" and show process.command_line, process.pid, process.owner, and process.uptime (age).

Metric	Type	Unit	Description	NVIDIA	AMD	Intel
process.gpu.memory.usage	UpDownCounter	By	GPU memory used by a process on a device	Yes	Yes	Yes
process.gpu.memory.utilization	Gauge	1	Process memory / device memory limit (0–1); omitted when limit unknown	Yes	Yes	Yes
process.gpu.utilization	Gauge	1	Per-process GPU utilization (0.0–1.0)	Yes	Yes	Yes
process.uptime	Gauge	s	Process uptime for GPU-attributed PIDs	Yes	Yes	Yes

Attributes

Attribute	Description
process.pid	Host process ID
process.executable.name	Short name from /proc/<pid>/comm (or Windows image basename)
process.command_line	Truncated command line (see OTEL_GPU_PROCESS_CMDLINE)
process.state	running, sleeping, zombie, stopped, dead, unknown
process.owner / process.owner.userid	Username and UID/SID
process.workload.kind	llm_inference, llm_training, or other
process.workload.framework	vllm, ollama, llama.cpp, sglang, tgi, triton, ray, pytorch, unknown
hw.id, hw.name, hw.vendor, gpu.index, gpu.pci_address	Same device attrs as hw.gpu.*
hw.gpu.task	On utilization only: general, encoder, decoder
k8s.pod.uid	From cgroup path when present
k8s.pod.name, k8s.namespace.name, k8s.container.name	Via kubelet PodResources (GPU device join) and/or pod API lookup
container.id	Runtime container ID when resolvable

Note

process.start_time is not exported as a metric attribute (cardinality). Use process.uptime for lifetime. On MIG devices, NVML often provides memory only (no per-process SM util). Process util may not sum to device util (time-slicing / sampling windows differ). Use process.state == zombie to find defunct processes still holding GPU contexts.

---

eBPF CUDA Metrics

Enable with OTEL_GPU_EBPF_ENABLED (on by default on Linux for serving+ / all modes; off in light). Soft-fails without caps. Discovers libcudart.so* (CUDA runtime API) and libcuda.so* (CUDA driver API) from common install paths and from /proc/*/maps (with Docker --pid=host / Kubernetes hostPID: true), so fleet DaemonSets do not need a CUDA toolkit mount. Driver-API probes extend coverage to frameworks that never load libcudart (llama.cpp, Ollama). Attaches uprobes/uretprobes for launches (cudaLaunchKernel, cudaLaunchKernelExC, cooperative, cuLaunchKernel, cuLaunchKernelEx), graph replays (cudaGraphLaunch, cuGraphLaunch), alloc/free, memcpy, stream/device/event sync, and cudaSetDevice. PIDs that map libcudart skip driver launch/graph events so runtime wrappers are not double-counted. AMD/Intel do not use this path — their per-process metrics come from DRM fdinfo.

Activity metrics

Metric	Type	Unit	Description	Attributes
gpu.kernel.launch.calls	Counter	{call}	CUDA kernel launch count (runtime and driver APIs)	process.pid, process.executable.name, cuda.kernel.name, hw.id / gpu.index when known
gpu.kernel.grid.size	Histogram	{thread}	Total threads in grid per launch	same
gpu.kernel.block.size	Histogram	{thread}	Threads per block per launch	same
gpu.kernel.shared_memory	Histogram	By	Dynamic shared memory per launch (cudaLaunchKernel sharedMem)	same
gpu.kernel.duration	Histogram	s	Model estimate of launch→sync duration (shared store with occupancy; cardinality-capped names)	same + optional cuda.launch.kind=graph
gpu.graph.launch.calls	Counter	{call}	CUDA graph replay invocations (cudaGraphLaunch / cuGraphLaunch). Counts replays, not the kernels executed inside each replay — that number is not observable at this API-tracing layer	process.pid, process.executable.name, hw.id / gpu.index when known
gpu.memory.allocations	Counter	By	Bytes allocated via cudaMalloc	process.pid, process.executable.name, hw.id / gpu.index when known
gpu.memory.copies	Histogram	By	Bytes per cudaMemcpyAsync	process.pid, cuda.memcpy.kind, hw.id / gpu.index when known

hw.id is set after cudaSetDevice for that thread, or immediately on single-GPU hosts. On multi-GPU hosts before cudaSetDevice, join via process.pid as before. Linux/NVIDIA (CUDA) only — same eBPF path for Kubernetes and non-Kubernetes when host PID is visible. AMD/Intel use device/process metrics via DRM fdinfo, not these kernel series.

Stream-sync occupancy model

These are CPU-side model estimates, not hardware SM occupancy (warps resident / max warps). Spans run from kernel launch to sync API return; thread-seconds are clamped to NVML CUDA core count and normalized so multi-process totals do not exceed device capacity.

Metric	Type	Unit	Description
process.gpu.core.usage	Gauge	{cores}	Normalized avg CUDA cores from launch→sync thread-seconds
process.gpu.estimated.sm_active	Gauge	1	Fraction of interval with any launch→sync span (gpu.measurement.source=ebpf_model)
hw.gpu.core.limit	Gauge	{cores}	NVML CUDA core count
hw.gpu.estimated.sm_active	Gauge	1	Device-wide union of launch→sync spans

Prefer NVML process.gpu.utilization for sampled SM%. These estimates are not DCGM hw.gpu.sm.utilization / hw.gpu.sm.occupancy. Use them when NVML process util is unavailable or when correlating launch→sync activity with device util.

Known limits: no GPU completion signal (late sync inflates usage); cudaGraphLaunch / cuGraphLaunch is one composite span and gpu.graph.launch.calls counts replays, not kernels (no per-node kernels inside a graph); cudaStreamWaitEvent wait edges are not modeled; sync cudaMemcpy and default-stream sync are treated as device-wide (legacy default-stream semantics).

---

DCGM profiling (optional)

Enable with OTEL_GPU_DCGM_ENABLED=true. Soft-fails when libdcgm is missing. DCGM is optional — continuous clocks (graphics / sm / memory), util, and power work from NVML alone. Metrics carry gpu.measurement.source=dcgm. Set OTEL_GPU_DCGM_PREFER=true so DCGM owns overlapping IO / power / util / graphics clock / idle (NVML suppresses those series — single producer). Memory clock stays on NVML.

Metric	Type	Unit	Attributes
hw.gpu.engine.utilization	Gauge	1	hw.gpu.engine
hw.gpu.sm.utilization	Gauge	1	DCP SM active
hw.gpu.sm.occupancy	Gauge	1	DCP SM occupancy
hw.gpu.pipe.utilization	Gauge	1	hw.gpu.pipe=tensor|fp16|fp32|fp64
hw.gpu.memory.bandwidth.utilization	Gauge	1
hw.gpu.io / hw.gpu.interconnect.io	Counter	By	Prefer only; rate-integrated PCIe / NVLink
hw.gpu.speed	Gauge	Hz	Prefer only; hw.gpu.clock_domain=graphics (field 100)
hw.gpu.idle	Gauge	1	Prefer only; derived from Prefer util
hw.power / hw.gpu.utilization / hw.gpu.memory.controller.utilization	Gauge	Prefer only	Fields 155 / 203 / 204 (204 → controller util)
shield360.collector.gpu.dcgm.sample_valid	UpDownCounter	1	0 on blank sample

Control plane: POST /v1/dcgm/pause and POST /v1/dcgm/resume (loopback OTEL_GPU_CONTROL_ADDR).

---

AMD RDC profiling (optional)

Enable with OTEL_GPU_RDC_ENABLED=true. Soft-fails when librdc is missing. Metrics carry gpu.measurement.source=rdc. Partition samples may set hw.parent.

Metric	Type	Unit	Notes
hw.gpu.sm.utilization	Gauge	1	SM/CU active
hw.gpu.sm.occupancy	Gauge	1	CU / SM occupancy
hw.gpu.pipe.utilization	Gauge	1	fp16 / fp32 / fp64 from EVAL_FLOPS percent
hw.gpu.simd.utilization	Gauge	1	SIMD utilization

---

Kineto on-demand profiling (optional)

Enable with OTEL_GPU_KINETO_ENABLED=true. Does not emit continuous metrics. Use the control API:

• POST /v1/profile/gpu — match GPU PIDs and write on-demand libkineto config
• Requires OTEL_GPU_CONTROL_ADDR (loopback by default; set OTEL_GPU_CONTROL_ALLOW_REMOTE=true + token for multi-node)

Self-metric: shield360.collector.profile.requests.

Multi-node fan-out (no always-on collector cost):

go run ./cmd/profile-fanout --hosts node1,node2 --port 1919 --token "$TOKEN" --duration-ms 500
# or SLURM:
go run ./cmd/profile-fanout --job-id 12345 --port 1919 --token "$TOKEN"

---

Intel PT on-demand (optional)

Enable with OTEL_HOST_INTEL_PT_ENABLED=true and a control address. Does not run continuously.

• POST /v1/profile/cpu/pt with {"duration_ms":500}
• Requires Linux intel_pt PMU and perf in PATH
• Caps: duration, CPU count, AUX buffer pages (see configuration)

Fan-out: profile-fanout --profile cpu-pt ...

---

CPU PMU / uncore memory IO (optional)

Enable with OTEL_HOST_PMU_ENABLED=true.

Metric	Notes
hw.cpu.instructions / hw.cpu.cycles / cache / branch / TLB	Core PMU
hw.cpu.memory.io	When memory_bandwidth or uncore is in OTEL_HOST_PMU_EVENTS; IMC CAS×64 bytes

---

TPU scrape (optional)

Enable with OTEL_TPU_ENABLED=true. Scrapes a Prometheus-format TPU endpoint (OTEL_TPU_ENDPOINT, default http://127.0.0.1:2112/metrics) and re-exports allowlisted series. Soft-fails when the endpoint is unreachable.

---

NIC / RDMA (optional)

Flag	Default	Metrics surface
OTEL_HOST_NIC_ENABLED	false	Per-NIC hardware counters beyond basic system.network.*
OTEL_HOST_RDMA_ENABLED	false	RDMA device counters (OTEL_HOST_RDMA_COUNTERS allowlist)

Soft-omitted when sysfs/counters are unavailable.

---

Capability matrix

Feature	NVIDIA	AMD	Intel
Device hw.gpu.* / spec hw.*	Yes	Yes	Partial
process.gpu.memory.usage / memory.utilization / utilization	NVML	DRM fdinfo	DRM fdinfo
Pod UID on process metrics	cgroup	cgroup	cgroup
eBPF activity + occupancy	Yes (default on Linux)	—	—
DCGM profiling	Optional	—	—
RDC profiling	—	Optional (soft-skip without librdc)	—