Evaluations

Overview

The Shield360 SDK provides server-side evaluations via shield360.eval() (Python) and shield360.eval() (JS/TS). Evaluations use the same engine, rules, contexts, and custom eval types configured in the Shield360 dashboard - working identically for development (offline) and production (online) stages.

Quick Start

Run your first offline evaluation in 3 lines of code.

Batch Evaluation

Evaluate multiple prompt/response pairs concurrently.

Attributes & Rules

Auto-resolve OTel attributes for context-aware evaluations.

---

Offline Evaluations

Offline evaluations run on the Shield360 server using the same evaluation engine as online/auto evaluations. The SDK sends your prompt and response to the server, which runs LLM-as-judge evaluation and returns structured results.

Prerequisites

1. A running Shield360 instance with evaluation configured in the dashboard.
2. A Shield360 API key (create one in the dashboard under Settings > API Keys).

Quick Start

Python

import shield360

# Option 1: Configure once via init()
shield360.init(
    shield360_url="http://localhost:3000",
    shield360_api_key="shield360-xxxxx",
)

# Run evaluation
result = shield360.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Lyon.",
    contexts=["Paris is the capital and largest city of France."],
)

# Use in assertions
assert result.passed, f"Evaluation failed: {result.failed_evals}"

# Option 2: Pass credentials directly (overrides init/env vars)
result = shield360.eval(
    prompt="Explain quantum computing",
    response="Quantum computers use qubits...",
    shield360_url="http://localhost:3000",
    shield360_api_key="shield360-xxxxx",
)

TypeScript / JavaScript

import shield360, { isPassed, getFailedEvals } from 'shield360';

// Option 1: Configure once via init()
shield360.init({
  shield360Url: 'http://localhost:3000',
  shield360ApiKey: 'shield360-xxxxx',
});

// Run evaluation
const result = await shield360.eval({
  prompt: 'What is the capital of France?',
  response: 'The capital of France is Lyon.',
  contexts: ['Paris is the capital and largest city of France.'],
});

// Use in assertions
console.log(result.success);       // true
console.log(isPassed(result));     // false - hallucination detected
console.log(getFailedEvals(result)); // [{ type: 'hallucination', ... }]

// Option 2: Pass credentials directly (overrides init/env vars)
const result = await shield360.eval({
  prompt: 'Explain quantum computing',
  response: 'Quantum computers use qubits...',
  shield360Url: 'http://localhost:3000',
  shield360ApiKey: 'shield360-xxxxx',
});

You can also configure via environment variables:

export SHIELD360_URL="http://localhost:3000"
export SHIELD360_API_KEY="shield360-xxxxx"

shield360.eval() / shield360.eval({}) Parameters

Python

Parameter	Type	Description	Default
prompt	str	The user prompt sent to the LLM. Required.	-
response	str	The LLM’s response to evaluate. Required.	-
contexts	list[str]	Ground truth context for the evaluation.	None
eval_types	list[str]	Specific eval types to run (e.g. ["hallucination", "toxicity"]). Runs all enabled types if omitted.	None
attributes	dict	Trace attributes for rule engine matching (overrides auto-resolved attributes).	None
threshold_score	float	Score threshold for verdict determination.	0.5
store_results	bool	Whether to store results in the Shield360 database.	True
run_id	str	Identifier to group related evaluations.	None
metadata	dict	Custom key-value metadata to attach to results.	None
shield360_api_key	str	API key (overrides init() and env var).	None
shield360_url	str	Server URL (overrides init() and env var).	None
print_results	bool	Print formatted summary to terminal.	True

TypeScript / JavaScript

Parameter	Type	Description	Default
prompt	string	The user prompt sent to the LLM. Required.	-
response	string	The LLM’s response to evaluate. Required.	-
contexts	string[]	Ground truth context for the evaluation.	undefined
evalTypes	string[]	Specific eval types to run (e.g. ["hallucination", "toxicity"]). Runs all enabled types if omitted.	undefined
attributes	Record<string, string | number | boolean>	Trace attributes for rule engine matching (overrides auto-resolved attributes).	undefined
thresholdScore	number	Score threshold for verdict determination.	0.5
storeResults	boolean	Whether to store results in the Shield360 database.	true
runId	string	Identifier to group related evaluations.	undefined
metadata	Record<string, string>	Custom key-value metadata to attach to results.	undefined
shield360ApiKey	string	API key (overrides init() and env var).	undefined
shield360Url	string	Server URL (overrides init() and env var).	undefined
printResults	boolean	Print formatted summary to terminal.	true

Result Object

shield360.eval() returns an OfflineEvalResult with these properties:

Property	Type	Description
success	bool	Whether the evaluation completed without errors.
passed	bool	True if no evaluation types returned a “yes” verdict.
evaluations	list[OfflineEvaluation]	Individual evaluation results per type.
failed_evals	list[OfflineEvaluation]	Evaluations that returned a “yes” verdict.
context_applied	ContextInfo	Information about rule-matched context.
metadata	dict	Model, run ID, token usage, and cost metadata.
error	str	Error message if success is False.

Each OfflineEvaluation contains:

Field	Type	Description
type	str	The evaluation type (e.g. “hallucination”).
score	float	The evaluation score (0.0 to 1.0).
verdict	str	“yes” if detected, “no” otherwise.
classification	str	Category of the detection or “none”.
explanation	str	Brief explanation of the evaluation result.

Selecting Evaluation Types

Run specific evaluation types instead of all enabled ones:

Python

result = shield360.eval(
    prompt="Discuss workplace equality",
    response="Older workers can't learn new tech.",
    eval_types=["bias", "toxicity"],
)

TypeScript / JavaScript

const result = await shield360.eval({
  prompt: 'Discuss workplace equality',
  response: "Older workers can't learn new tech.",
  evalTypes: ['bias', 'toxicity'],
});

Discover Available Types

Python

types = shield360.get_eval_types()
for t in types:
    print(f"{t.id}: {t.label} (custom={t.is_custom}, enabled={t.enabled})")

TypeScript / JavaScript

const types = await shield360.getEvalTypes();
for (const t of types) {
  console.log(`${t.id}: ${t.label} (custom=${t.isCustom}, enabled=${t.enabled})`);
}

Batch Evaluation

Evaluate multiple prompt/response pairs concurrently:

Python

dataset = [
    {
        "prompt": "What is 2+2?",
        "response": "2+2 equals 4.",
        "contexts": ["Basic arithmetic."],
    },
    {
        "prompt": "Who wrote Hamlet?",
        "response": "Hamlet was written by Charles Dickens.",
    },
    {
        "prompt": "Describe gravity",
        "response": "Gravity is the force of attraction between masses.",
        "eval_types": ["hallucination"],
    },
]

batch_result = shield360.eval_batch(
    dataset=dataset,
    eval_types=["hallucination", "toxicity"],
    max_concurrent=5,
)

print(f"Pass rate: {batch_result.pass_rate:.0%}")
assert batch_result.all_passed

TypeScript / JavaScript

import shield360, { isAllPassed, getPassRate } from 'shield360';

const batchResult = await shield360.evalBatch({
  dataset: [
    {
      prompt: 'What is 2+2?',
      response: '2+2 equals 4.',
      contexts: ['Basic arithmetic.'],
    },
    {
      prompt: 'Who wrote Hamlet?',
      response: 'Hamlet was written by Charles Dickens.',
    },
    {
      prompt: 'Describe gravity',
      response: 'Gravity is the force of attraction between masses.',
      evalTypes: ['hallucination'],
    },
  ],
  evalTypes: ['hallucination', 'toxicity'],
  maxConcurrent: 5,
});

console.log(`Pass rate: ${(getPassRate(batchResult) * 100).toFixed(0)}%`);
console.log(`All passed: ${isAllPassed(batchResult)}`);

shield360.eval_batch() Parameters

Parameter	Type	Description	Default
dataset	list[dict]	List of items with prompt and response keys. Required.	-
eval_types	list[str]	Default eval types (can be overridden per item).	None
attributes	dict	Default attributes (can be overridden per item).	None
threshold_score	float	Default threshold score.	0.5
store_results	bool	Store all results in the database.	True
run_id	str	Group all batch evaluations under this ID. Auto-generated if omitted.	None
max_concurrent	int	Maximum number of concurrent evaluations.	5
print_results	bool	Print aggregate summary to terminal.	True

Automatic Attribute Resolution

The SDK automatically resolves trace attributes for rule engine matching, enabling context-aware evaluations without extra configuration. The resolution order (last wins):

1. OTEL_RESOURCE_ATTRIBUTES environment variable
2. OTEL_SERVICE_NAME environment variable
3. SHIELD360_ENVIRONMENT / OTEL_DEPLOYMENT_ENVIRONMENT environment variable
4. shield360.init() configuration (application_name, environment)
5. Explicit attributes parameter (highest priority)

Python

import shield360

# These are auto-detected for rule matching:
shield360.init(
    application_name="my-chatbot",
    environment="staging",
)

# Rules configured in the dashboard for service.name="my-chatbot"
# and deployment.environment="staging" will automatically match.
result = shield360.eval(
    prompt="Hello",
    response="Hi there!",
)

# Override auto-resolved attributes:
result = shield360.eval(
    prompt="Hello",
    response="Hi there!",
    attributes={
        "service.name": "different-service",
        "custom.tag": "experiment-v2",
    },
)

TypeScript / JavaScript

import shield360 from 'shield360';

// These are auto-detected for rule matching:
shield360.init({
  applicationName: 'my-chatbot',
  environment: 'staging',
});

// Rules configured in the dashboard for service.name="my-chatbot"
// and deployment.environment="staging" will automatically match.
const result = await shield360.eval({
  prompt: 'Hello',
  response: 'Hi there!',
});

// Override auto-resolved attributes:
const result2 = await shield360.eval({
  prompt: 'Hello',
  response: 'Hi there!',
  attributes: {
    'service.name': 'different-service',
    'custom.tag': 'experiment-v2',
  },
});

CI/CD Integration

Use offline evaluations in your test suite or CI pipeline:

Python

import shield360
import pytest

def test_no_hallucination():
    result = shield360.eval(
        prompt="What year did WW2 end?",
        response="World War 2 ended in 1945.",
        eval_types=["hallucination"],
        print_results=False,
    )
    assert result.passed, f"Hallucination detected: {result.failed_evals}"

def test_batch_quality():
    dataset = load_test_cases()  # your test data
    result = shield360.eval_batch(
        dataset=dataset,
        print_results=False,
    )
    assert result.pass_rate >= 0.95, f"Pass rate too low: {result.pass_rate:.0%}"

TypeScript / JavaScript

import shield360, { isPassed, getFailedEvals, isAllPassed, getPassRate } from 'shield360';
import { describe, test, expect } from 'vitest'; // or jest

describe('LLM quality', () => {
  test('no hallucination', async () => {
    const result = await shield360.eval({
      prompt: 'What year did WW2 end?',
      response: 'World War 2 ended in 1945.',
      evalTypes: ['hallucination'],
      printResults: false,
    });
    expect(isPassed(result)).toBe(true);
  });

  test('batch quality', async () => {
    const result = await shield360.evalBatch({
      dataset: loadTestCases(),
      printResults: false,
    });
    expect(getPassRate(result)).toBeGreaterThanOrEqual(0.95);
  });
});

Configuration Precedence

For shield360_api_key and shield360_url, the resolution order is:

1. Explicit function parameter (highest priority)
2. shield360.init() configuration
3. SHIELD360_API_KEY / SHIELD360_URL environment variables

---

Deploy Shield360

Deployment options for scalable LLM monitoring infrastructure

Online Evaluations

Get started with evaluating your LLM responses in 2 simple steps on Shield360

Destinations

Send telemetry to Datadog, Grafana, New Relic, and other observability stacks