Configuration
The Configuration tab on Evaluations (/evaluations?tab=configuration) sets the judge model and Auto Evaluation schedule used for online LLM evaluation. The page opens with a Manual & Auto info card explaining both paths. (/evaluations/settings and /settings/evaluation redirect here.)
Evaluation engine
Section titled “Evaluation engine”Under the Configuration card, choose:
| Field | Label | Notes |
|---|---|---|
| Provider | Provider | From Costs → Manage models |
| Model | Model | List from the selected provider, or a custom model name (e.g. gpt-4o-mini or custom model name) |
| API key | API Key (Vault) | Select a Vault secret, or Create new |
Save with Save Changes or Create Config depending on whether a config already exists.
Auto Evaluation
Section titled “Auto Evaluation”Auto Evaluation runs enabled evaluators against newly ingested LLM traces on a schedule - no manual click or SDK call required.
| Manual | Auto | |
|---|---|---|
| Trigger | Run Evaluation on a trace, or SDK | Cron schedule |
| Scope | One trace (or one SDK call) | New LLM traces since the last run |
| Sampling | None | Configurable sample rate |
| Result tagging | source: manual | source: auto (or skipped when sampled out) |
Go to Monitor → Evaluations → Configuration.
Select Provider, Model, and API Key (Vault).
Toggle Enable auto evaluation.
Enter Sample rate (%) - the percentage of eligible traces to evaluate (e.g. 10 for 10%). Sampling is deterministic for a given trace.
Enter a standard cron expression (e.g. */15 * * * * every 15 minutes).
Click Save Changes / Create Config.
Which types actually run is controlled on the Evaluators tab (and via Rule Engine links). Manual evaluation from a trace still uses this same configured judge.
After a restart
Section titled “After a restart”Active Auto Evaluation schedules are restored when Shield360 starts again - no manual re-setup after a redeploy.
Manual evaluation reminder
Section titled “Manual evaluation reminder”The Manual Evaluation card points you to Telemetry: open a trace and run evaluation from its Evaluation tab, or use Go to Traces. See LLM-as-a-Judge.