Configuration

The Configuration tab on Evaluations (/evaluations?tab=configuration) sets the judge model and Auto Evaluation schedule used for online LLM evaluation. The page opens with a Manual & Auto info card explaining both paths. (/evaluations/settings and /settings/evaluation redirect here.)

Evaluation engine

Under the Configuration card, choose:

Field	Label	Notes
Provider	Provider	From Costs → Manage models
Model	Model	List from the selected provider, or a custom model name (e.g. gpt-4o-mini or custom model name)
API key	API Key (Vault)	Select a Vault secret, or Create new

Save with Save Changes or Create Config depending on whether a config already exists.

Auto Evaluation

Auto Evaluation runs enabled evaluators against newly ingested LLM traces on a schedule - no manual click or SDK call required.

	Manual	Auto
Trigger	Run Evaluation on a trace, or SDK	Cron schedule
Scope	One trace (or one SDK call)	New LLM traces since the last run
Sampling	None	Configurable sample rate
Result tagging	source: manual	source: auto (or skipped when sampled out)

Open Configuration

Go to Build → Testing → Evaluations → Configuration.

Choose the judge model

Select Provider, Model, and API Key (Vault).

Enable auto evaluation

Toggle Enable auto evaluation.

Set sample rate

Enter Sample rate (%) - the percentage of eligible traces to evaluate (e.g. 10 for 10%). Sampling is deterministic for a given trace.

Set Cron schedule

Enter a standard cron expression (e.g. */15 * * * * every 15 minutes).

Save

Click Save Changes / Create Config.

Which types actually run is controlled on the Evaluators tab (and via Rule Engine links). Manual evaluation from a trace still uses this same configured judge.

When content is not stored: evaluated as it arrives

When your organization does not store prompts and responses (Data retention), stored traces hold nothing to judge. Auto Evaluation then evaluates each eligible span as it arrives and discards the content once it has been scored. The cron schedule still runs but reads no stored spans, so nothing is judged twice. When content storage is on, Auto Evaluation reads stored traces on the schedule, as above.

How it works:

1. When telemetry reaches Shield360’s ingest, the content is handed to the evaluation before data retention drops it. Only spans an evaluation applies to are kept for it: the same operations (gen_ai.operation.name = chat) and the same deterministic sample as the scheduled path. Rule Engine matches choose the context the judge is given, as before.
2. The content waits in the receiving server’s memory only — never on disk or in a database — for at most 10 minutes. Up to 2 evaluations run at once for an environment.
3. The judge (the provider, model and key configured above) is told not to quote the content. Before anything is stored, the explanation is cleaned: anything in quotation marks, any run of five or more words from the prompt or response, and any long token from them (an id, a key, a URL) is replaced with […], and secrets and personal data are masked. The label is reduced to a snake_case word.
4. The score, label, verdict and cleaned explanation are stored against the span and trace ids with source: in_flight, as any evaluation is, with its cost. The content is let go.

Coverage on the Analytics tab shows, for the period, how many spans were evaluated, skipped by the sample, not evaluated because the content expired before evaluation (the 10 minutes passed or the buffer was full), failed, or were over budget. A trace’s Evaluation tab says which of these happened to that span. A span that was not evaluated cannot be evaluated later: its content is gone. Spans waiting when a server stops are lost without being counted.

Caps

With Auto Evaluation on, three caps apply to evaluating as spans arrive:

Field	Default	Effect
Evaluations per minute	60	Spans over it wait their turn (per server)
Evaluations per day	2,000	Spans over it are not evaluated (recorded as over budget)
Daily evaluation cost cap (USD)	0 (none)	Checked against the metered cost of today’s in-flight evaluations

The daily caps count since midnight UTC, are checked before the judge is called, and are shared by every server.

SDK apps

The SDKs do not capture message content unless you ask them to. For an SDK app’s spans to be evaluated, turn content capture on in the SDK (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true, or the SDK option). With storage off, the content is used for scoring and then discarded; it is not stored.

Note

Only telemetry that reaches Shield360’s ingest (/api/ingest/otlp, and enrolled machines) is evaluated as it arrives. Telemetry sent straight to a collector on ports 4317/4318 has its content dropped by the collector and is not evaluated. Set SHIELD360_INFLIGHT_EVALUATIONS=false on the server to turn in-flight evaluation off.

After a restart

Active Auto Evaluation schedules are restored when Shield360 starts again - no manual re-setup after a redeploy.

Manual evaluation reminder

The Manual Evaluation card points you to Telemetry: open a trace and run evaluation from its Evaluation tab, or use Go to Traces. See LLM-as-a-Judge.

---

Evaluators

Choose which types Auto Evaluation runs

Analytics

Review pass rates and auto execution volume