Evaluations

Automated AI evaluation to assess and monitor the quality, safety, and performance of your LLM outputs across development and production environments. Open Monitor → Evaluations for Analytics, Evaluators, and Configuration. For evaluation types and custom evaluators, see Evaluators.

Find the right feature

If you want to…	Use this Shield360 feature
Automatically score every production trace	Configuration (Auto Evaluation)
Score one specific trace on demand	LLM-as-a-Judge from that trace’s Evaluation tab
Rate a trace yourself instead of an LLM judge	Manual Feedback - Good / Bad / Neutral plus a comment
Judge responses against your own ground truth, not just the model’s knowledge	Rule Engine + Context
Evaluate on criteria beyond the 11 built-in types	Custom evaluators
Only run certain evaluators for certain models, providers, or environments	Evaluators + Rule Engine
Test prompt or model changes before shipping	Programmatic evaluations via the SDK
Block a deploy on a quality regression	Programmatic evaluations in a CI/CD pipeline
See pass-rate trends across evaluators and time	Analytics

flowchart LR;
    A[LLM / Agent call] -->|creates| B[Trace in Shield360];
    B -->|Auto Eval schedule or manual Run Evaluation| C[LLM Judge];
    C -->|score + classification + reasoning| D[Trace detail & Dashboard];

Pick a judge model

Go to Evaluations → Settings, choose a provider and model to act as the judge (OpenAI, Anthropic, Google, Mistral, and 7+ others), and add its API key from Vault.

Enable evaluators

Switch to the Evaluation Types tab and turn on the evaluators you want. Hallucination, Bias, and Toxicity are enabled by default; Relevance, Coherence, Safety, and 5 others are opt-in.

Turn on Auto Evaluation

Back in Settings, enable Auto Evaluation with a cron schedule so every new trace gets scored automatically - or skip this and click Run Evaluation from any trace’s Evaluation tab to score it on demand.

Review results

Open any trace’s Evaluation tab for its score, classification, and reasoning, or check the Evaluations dashboard for aggregate trends across models and time.

flowchart LR;
    E[Dataset] -->|SDK evaluate call| F[LLM Judge];
    F -->|score| G[Pass / fail gate];

Get an API key

In Shield360, go to Settings → API Keys and create a key. Offline evaluations call your Shield360 server, so you’ll need this key plus your Shield360 URL.

Run an evaluation

Python
Typescript

import shield360

shield360.init(
    shield360_url="http://localhost:3000",
    shield360_api_key="shield360-xxxxx",
)

result = shield360.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Lyon.",
    contexts=["Paris is the capital and largest city of France."],
)

assert result.passed, f"Evaluation failed: {result.failed_evals}"

import shield360, { isPassed, getFailedEvals } from "shield360"

shield360.init({
  shield360Url: "http://localhost:3000",
  shield360ApiKey: "shield360-xxxxx",
})

const result = await shield360.eval({
  prompt: "What is the capital of France?",
  response: "The capital of France is Lyon.",
  contexts: ["Paris is the capital and largest city of France."],
})

console.log(isPassed(result), getFailedEvals(result))

This uses the exact same evaluators, custom types, and Rule Engine context matching configured in your dashboard - see the Programmatic evaluations quickstart for batch runs and CI/CD gating.

Evaluators

Built-in and custom evaluation types

Configuration

Auto Evaluation schedule and judge model

LLM-as-a-Judge

Use advanced LLMs to evaluate AI application quality with automated scoring

Programmatic evaluations

Quick start guide for implementing custom evaluations in your code