Ingest pipeline

Every span and log record passes through three steps before it is stored. They run in the bundled OpenTelemetry Collector for the platform’s own database, and in the platform itself for a project on its own ClickHouse, from the same rule files, so rows read the same wherever they land.

Step	What it does
Map vendor attribute names	Copies OpenLLMetry (llm.*), OpenInference (llm.token_count.*), Vercel AI SDK (ai.*) and older OpenTelemetry GenAI names (gen_ai.usage.prompt_tokens, gen_ai.system) to the GenAI names the dashboards read. Nothing already present is overwritten, and the vendor attribute is kept.
Scrub secrets	Replaces API keys, cloud access keys, JWTs, private keys and the credentials in connection strings with [REDACTED], using the same patterns the shield360 CLI applies on each machine.
Price spans	Puts gen_ai.usage.cost on spans that report a model and token counts but no cost, and records how the price was found in shield360.pricing.basis (exact, or family when a model name with no version takes the newest priced model of its family; a versioned model with no row of its own is left unpriced rather than priced at another version’s rate). A cost the SDK set is kept unless you choose otherwise.

Settings

Open Settings → Collection & deployment and find What happens to telemetry on the way in. Each step has a switch. Under Additional redaction patterns you can add your own RE2 regular expressions, for example an employee-id format or an internal ticket reference; matches are replaced whole. A leading (?i) is supported; lookaround and backreferences are not, because the collector’s engine does not have them.

Settings are saved per project database. For the database the bundled collector writes to, a save is rendered as the collector’s processor configuration and sent to every collector through the OpAMP server, without a restart. The card lists each collector and whether it applied the configuration; a collector that failed shows the reason it reported.

Note

Only the processor’s own section of configuration ever travels to a collector. Nothing set here can add a receiver, change an exporter or redirect data.

Series derived at ingest

The collector also turns spans into usage series as they arrive: calls and latency per service, operation and model, and LLM requests, tokens and cost per service, environment, model and provider. Dashboard queries for requests over time and for requests by model, provider or application read these series when they cover the selected window, and fall back to the span table otherwise. Nothing changes in what you see; wide windows answer faster.

Where the rules live

• assets/ingest/attribute-mappings.json — the vendor-to-GenAI groups
• assets/ingest/redaction-patterns.json — the built-in secret patterns
• assets/pricing.json — list prices per model

The image copies them to /app/ingest/, the platform bundles the same files, and CI keeps every copy identical.

Next steps

• Collection policy — what the endpoint collector may collect on each machine
• Privacy and security — what leaves a machine, and what never does