Long ranges, rollups and background queries

Spans are kept for 730 hours (about 30 days). A 30-day chart or count read straight from them has to scan every span in the range, and any filter or total that uses a span attribute — model, provider, cost, tokens — reads the attribute data of every one of them. On a busy project that takes longer than a page can wait.

Shield360 can keep three smaller stores beside the spans, in each project’s own database, and use them automatically. They are rolled out by an operator in stages (see Rollout); until then every view reads the spans as before.

What is kept

Store	What it holds	Kept
Hourly rollups	Per hour (UTC), service, span name, status, environment, operation, model and provider: span count, error count, duration total, minimum, maximum and percentiles (p50, p90, p95, p99), cost and tokens	400 days
Daily rollups	The same per UTC day, built from the hourly rollups once a day has ended	400 days
Trace index	Per span: time, trace and span IDs, service, span name, status, environment, operation, model and provider	As long as the spans (730 hours)

None of them holds prompts, responses, tool arguments or any other attribute value about a person. Data retention settings apply before anything is stored. Cost and token values that are not finite numbers (nan, inf) count as 0. Because the rollups are kept for 400 days, charts and counts on the 3M range and longer show the whole range, not only the last 30 days of spans.

They are only ever created in databases the platform manages — a project’s database it provisioned, or its own bundled database. A ClickHouse you connected yourself is never changed; its views always read the spans.

How a view chooses where to read

48 hours or less: read from the spans.
Longer ranges: summary charts, list totals, Home’s request, error and application counts, grouped tables and structured dashboard widgets read the rollups. The first and last partial hour of the range are still read from the spans, so the edges are exact to the second. Whole UTC days come from the daily rollups when the chart’s buckets are UTC days or longer; otherwise from the hourly rollups.
List pages pick the page’s rows from the trace index, then read only those rows in full.

Counts, sums and averages match what the spans give (sums of decimal cost can differ in the last digits). Percentiles from the rollups are estimates, as percentiles from the spans already are.

A view reads the spans instead when the rollups cannot answer it exactly: a Max Cost filter, Generation issues or Agent loops chips, an agent version scope, a custom filter on an attribute the rollups do not keep, a chart in a time zone that is not a whole number of hours from UTC, a range the rollups do not hold completely (not yet backfilled, or marked stale), or a data source with its own traces table name. Raw SQL dashboard widgets always read the tables their SQL names.

Keeping them correct

Backfill. A background job rebuilds the days written before the views existed, one UTC day at a time, newest first. Each day is rebuilt in a staging table of its own, swapped in atomically, and used only once it holds exactly as many spans as the raw table. A day whose raw partition is about to be dropped under its TTL is left alone; once gone, the rollups are the whole answer for it.
Reconciliation. Every hour the job counts each day the rollups are trusted with against the raw spans. A day that differs — after a restore, a dropped partition, a view that failed on an insert — is marked stale at once, so views stop using it, and is rebuilt.
Inserts never fail because of a view. Writes into managed databases (the platform’s own write path and the bundled collector) carry materialized_views_ignore_errors=1, so a failing view never fails an insert — which would otherwise be retried and store the spans twice. Reconciliation rebuilds anything the view missed.
Changed definitions. Each view is named after a hash of its definition. A release that changes what a view computes replaces it on the next migration and resets what it covered; the backfill rebuilds.

Background queries

Some searches are larger than a page can wait for — every error across a busy month, say. When a Traces or Exceptions search runs past 15 seconds, or stops because it was too large or too slow, Shield360 offers Run in background.

A background query:

uses exactly the filters, range and environment you had on screen;
is queued, and run by the scheduled worker — a few at a time across the platform and at most two per organisation — read-only, a page of spans at a time, with its own time limit;
shows a progress bar in Background queries (next to the filters): spans written of spans matched;
can be cancelled at any time;
produces a gzipped JSON Lines file (.ndjson.gz) — one span per line, newest first, without span events and links — that you can download for 7 days. Results are capped at 1,000,000 spans; the drawer says when a result was cut off.

You are told when one finishes while you are on the Telemetry page. Only you can see, cancel or download your background queries, within the organisation and project you started them in, and only while you can still read that data source. Each person can have three queued or running at once. Every download is recorded in the audit log. Results are written straight into the export store (EXPORT_STORAGE: encrypted local files, or S3 with server-side encryption) and are removed when they expire, when the organisation is deleted, and when a person’s data is erased.

Rollout

Each stage has its own switch, off by default, and can be turned back off on its own.

Switch	Effect when `on`
`SHIELD360_TRACE_ROLLUP_VIEWS`	The migrate step creates the tables and views on managed databases. Set to anything else, the next migrate step drops the views (the kill switch) and forgets what they covered.
`SHIELD360_TRACE_ROLLUP_BACKFILL`	The scheduled job backfills, reconciles and builds the daily table.
`SHIELD360_TRACE_ROLLUPS`	Views read the rollups and the index. Takes effect at once, no deploy.

Collector. Deploy the collector configuration with materialized_views_ignore_errors=1 in its ClickHouse endpoint (included in this release’s configuration). Check inserts still flow.
Background logins. The migrate step gives every provisioned project a background ClickHouse login (<user>_bg) with its own two-thread profile and its own quota, so backfills and background queries never use the quota ingest and the pages share (SHIELD360_CLICKHOUSE_BACKGROUND_USERS=off stops this).
Views. Set SHIELD360_TRACE_ROLLUP_VIEWS=on and run the migrate step. Check its log for skipped databases (a schema that differs from the exporter’s is skipped, with the reason).
Backfill. Set SHIELD360_TRACE_ROLLUP_BACKFILL=on. Watch the job’s log (logs/telemetry/rollups-backfill.log): days rebuilt, months, stale days, mismatches. Statements carry query ids starting s360_rollup_. Tune with SHIELD360_ROLLUP_BACKFILL_DAYS_PER_RUN, SHIELD360_ROLLUP_BACKFILL_MAX_THREADS and SHIELD360_ROLLUP_BACKFILL_MAX_MEMORY_BYTES.
Reads. Set SHIELD360_TRACE_ROLLUPS=on once the backfill has covered the ranges people use (or straight away: uncovered ranges read the spans). The day the views were created is rebuilt only once it has ended, so long ranges that reach today use the rollups from the next day on.

After restoring a ClickHouse backup, reset a database’s rollup coverage so nothing is trusted until it is rebuilt:

curl -X POST "$API_URL/api/telemetry-rollups/reset" \
  -H "X-CRON-JOB: $CRON_JOB_SECRET" -H "Content-Type: application/json" \
  -d '{"databaseConfigId": "<config id>"}'

Reconciliation would also find the difference within the hour; the reset makes it immediate.

To back out, set SHIELD360_TRACE_ROLLUPS=off (reads, at once), SHIELD360_TRACE_ROLLUP_BACKFILL=off, and SHIELD360_TRACE_ROLLUP_VIEWS=off with a migrate run (drops the views). The tables can then be dropped by hand: shield360_trace_rollup_1h, shield360_trace_rollup_1d, shield360_trace_index, shield360_trace_rollup_state.