What is AI Evaluation?
AI evaluation is the process of scoring LLM outputs for quality, safety, and correctness - using an LLM-as-a-judge, rule-based checks, or human review - so you can catch bad responses before or after they reach users.
Why does AI evaluation matter?
Section titled “Why does AI evaluation matter?”An LLM can produce a fluent, confident-sounding response that’s factually wrong, biased, or off-topic, without ever raising an error. AI evaluation gives you a quality signal traditional monitoring can’t: a score for hallucination, bias, toxicity, relevance, and more, attached to the exact prompt version and model that produced the response.
What is LLM-as-a-judge?
Section titled “What is LLM-as-a-judge?”LLM-as-a-judge uses a separate LLM call to grade a response against a rubric - for example, “does this response contain a hallucinated fact?” - and returns a score and explanation. It scales further than human review and can run automatically on every response in production.
What’s the difference between online and offline evaluation?
Section titled “What’s the difference between online and offline evaluation?”- Online evaluation runs automatically against live production traces - manually triggered, or on an automatic schedule with sampling to control cost.
- Offline evaluation runs programmatically, typically in CI/CD, against a fixed dataset before you ship a prompt or model change - catching regressions before they reach production.
How does Shield360 provide AI evaluation?
Section titled “How does Shield360 provide AI evaluation?”Shield360 includes 11 built-in evaluation types (hallucination, bias, toxicity, and more), runs them automatically in production from Configuration (Auto Evaluation), and exposes the same evaluators programmatically via the SDK for offline testing in CI/CD.