Overview

Evaluations (/evaluations) is Shield360’s LLM evaluation surface under Monitor. Score production traces with an LLM-as-a-judge, enable built-in or custom evaluators, schedule Auto Evaluation, and review pass-rate analytics - or run the same criteria programmatically via the SDK for offline CI/CD gates.

Open it from Monitor → Evaluations. The page has three tabs:

Analytics

Pass rates, executions, cost, and evaluator results over time

Evaluators

Enable built-in types or create custom evaluators

Configuration

Judge model, Vault API key, Auto Evaluation schedule and sampling

Default tab is Analytics. Use ?tab=evaluators or ?tab=configuration for the others.

Find the right feature

If you want to…	Use this
See pass rates and spend for evals	Analytics
Turn on hallucination, bias, toxicity, or a custom type	Evaluators
Schedule Auto Evaluation or set the judge model	Configuration
Score one trace on demand	LLM-as-a-Judge from the trace Evaluation tab
Rate a trace yourself	Manual Feedback
Gate CI/CD on quality	Programmatic evaluations

flowchart TB;
    subgraph Online["Online evaluation - production traces"]
        direction LR;
        A[LLM / Agent call] -->|creates| B[Trace in Shield360];
        B -->|Auto Eval schedule or manual Run Evaluation| C[LLM Judge];
        C -->|score + classification + reasoning| D[Analytics and trace detail];
    end
    subgraph Offline["Offline evaluation - CI/CD"]
        direction LR;
        E[Dataset] -->|SDK evaluate call| F[LLM Judge];
        F -->|score| G[Pass / fail gate];
    end

Methods

LLM-as-a-Judge

Automated scoring on live traces

Programmatic evals

SDK / CI offline evaluation

Manual Feedback

Good / Bad / Neutral human ratings