Programmatic Evaluations

This guide shows how to run Shield360 evaluations programmatically from your own code - the same evaluation engine, evaluators, and Rule Engine used for online/auto evaluations, called directly via the SDK.

Get an API key

In Shield360, go to Settings → API Keys and create a key. Programmatic evaluations run against your Shield360 server, so you’ll need this key plus your Shield360 URL.

Run an evaluation

Python
Typescript

import shield360

shield360.init(
    shield360_url="http://localhost:3000",
    shield360_api_key="shield360-xxxxx",
)

result = shield360.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Lyon.",
    contexts=["Paris is the capital and largest city of France."],
)

assert result.passed, f"Evaluation failed: {result.failed_evals}"

import shield360, { isPassed, getFailedEvals } from "shield360"

shield360.init({
  shield360Url: "http://localhost:3000",
  shield360ApiKey: "shield360-xxxxx",
})

const result = await shield360.eval({
  prompt: "What is the capital of France?",
  response: "The capital of France is Lyon.",
  contexts: ["Paris is the capital and largest city of France."],
})

console.log(isPassed(result), getFailedEvals(result))

You can also set the URL and API key via SHIELD360_URL / SHIELD360_API_KEY environment variables instead of init().

Evaluate a whole dataset

Pass a list of prompt/response pairs to run them concurrently - useful as a CI/CD quality gate:

Python
Typescript

batch_result = shield360.eval_batch(dataset=[
    {"prompt": "What is 2+2?", "response": "2+2 equals 4."},
    {"prompt": "Who wrote Hamlet?", "response": "Hamlet was written by Charles Dickens."},
])

assert batch_result.all_passed, f"Pass rate: {batch_result.pass_rate:.0%}"

import shield360, { isAllPassed } from "shield360"

const batchResult = await shield360.evalBatch({
  dataset: [
    { prompt: "What is 2+2?", response: "2+2 equals 4." },
    { prompt: "Who wrote Hamlet?", response: "Hamlet was written by Charles Dickens." },
  ],
})

console.log(isAllPassed(batchResult))

Evaluations run programmatically use the exact same evaluators, custom types, and Rule Engine context matching configured in your Shield360 dashboard - trace attributes like service.name and deployment.environment are auto-resolved from shield360.init() for rule matching, or you can pass your own via the attributes parameter.

Full SDK reference

Parameters, result objects, batch evaluation, and Rule Engine attribute resolution in depth

CI/CD integration

Use evaluations as quality gates in your test suite or pipeline

LLM-as-a-judge

Automatically add evaluation scoring to production traces

Integrations

60+ AI integrations with automatic instrumentation and performance tracking

Destinations

Send telemetry to Datadog, Grafana, New Relic, and other observability stacks