Manual Feedback

Not every quality signal needs an LLM judge. Manual Feedback lets a human rate a specific trace directly, alongside the automated evaluations already running on it.

How it works

1. Open any LLM trace in Telemetry and click its Evaluation tab.
2. In the Feedback panel, rate the response Good, Bad, or Neutral.
3. Optionally add a comment explaining the rating.
4. Click Submit Feedback.

Your feedback is stored as its own evaluation entry (manual_feedback) in the same evaluation pipeline as the automated evaluators - so it appears in the trace’s evaluation list right alongside Hallucination, Bias, and any other evaluator results, and rolls up into Evaluations → Analytics.

Note

Score follows the same convention as the automated evaluators (low score = no issue): Good stores 0, Neutral stores 0.5, and Bad stores 1. Your comment is saved as the entry’s explanation.

When to use it

• Spot-checking responses an automated evaluator didn’t flag, or disagreeing with one that did
• Collecting reviewer or subject-matter-expert sign-off on specific traces
• Capturing qualitative notes that don’t fit a built-in or custom evaluation type

---

Evaluators

Built-in and custom evaluation types

LLM-as-a-Judge

Automated scoring for hallucination, bias, toxicity, and more

Configuration

Auto Evaluation schedule and judge model