Get started with AI Observability

Tip

Shield360 automatically instruments LLMs, VectorDBs, MCP, and frameworks by default.

This guide demonstrates how to implement real-time cost tracking, token usage monitoring, hallucination detection, and latency optimization for your AI applications with OpenTelemetry traces and metrics.

flowchart TB;
    subgraph " "
        direction LR;
        subgraph " "
            direction LR;
            Shield360_SDK[Shield360 SDK] -->|Sends Traces & Metrics| OTC[OpenTelemetry Collector];
            OTC -->|Stores Data| ClickHouseDB[ClickHouse];
        end
        subgraph " "
            direction RL;
            Shield360_UI[Shield360] -->|Pulls Data| ClickHouseDB;
        end
    end

Deploy Shield360

Git clone Shield360 repository

git clone git@github.com:ThinkfleetAI/Shield360.git

Start Docker Compose

From the root directory of the Shield360 distribution, Run the below command:

docker compose up -d

Tip

Prefer Kubernetes, or already have ClickHouse/OpenTelemetry Collector running? See Self-Host Shield360 for Helm charts and other deployment options.

Install Shield360 SDK

Python

pip install shield360

Typescript

npm install shield360

Instrument your AI application

Python

Manual instrumentation

Via function parameters

import shield360

shield360.init(otlp_endpoint="http://127.0.0.1:4318")

Examples:

from openai import OpenAI
import shield360

shield360.init(otlp_endpoint="http://127.0.0.1:4318")

client = OpenAI(
    api_key="YOUR_OPENAI_KEY"
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is LLM Observability?",
        }
    ],
    model="gpt-3.5-turbo",
)

import os
from anthropic import Anthropic
import shield360

shield360.init(otlp_endpoint="http://127.0.0.1:4318")

client = Anthropic(
    # This is the default and can be omitted
    api_key=os.environ.get("ANTHROPIC_API_KEY"),
)

message = client.messages.create(
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Hello, What is LLM Observability?",
        }
    ],
    model="claude-3-opus-20240229",
)

import cohere
import shield360

shield360.init(otlp_endpoint="http://127.0.0.1:4318")

co = cohere.Client(
    api_key="YOUR_API_KEY",
)

chat = co.chat(
    message="hello world!",
    model="command"
)

from litellm import completion
import os
import shield360

shield360.init(otlp_endpoint="http://127.0.0.1:4318")

os.environ["HUGGINGFACE_API_KEY"] = "huggingface_api_key"

# e.g. Call 'WizardLM/WizardCoder-Python-34B-V1.0' hosted on HF Inference endpoints
response = completion(
  model="huggingface/WizardLM/WizardCoder-Python-34B-V1.0",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  api_base="https://my-endpoint.huggingface.cloud"
)

from langchain_core.messages import HumanMessage, SystemMessage
import shield360

shield360.init(otlp_endpoint="http://127.0.0.1:4318")

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o-mini")

messages = [
    SystemMessage(content="Translate the following from English into Italian"),
    HumanMessage(content="hi!"),
]

model.invoke(messages)

import ollama
import shield360

shield360.init(otlp_endpoint="http://127.0.0.1:4318")

response = ollama.chat(model='llama3.1', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])

Via environment variables

Add the following two lines to your application code:

import shield360

shield360.init()

Run the following command to configure the OTEL export endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT = "http://127.0.0.1:4318"

Examples:

from openai import OpenAI
import shield360

shield360.init()

client = OpenAI(
    api_key="YOUR_OPENAI_KEY"
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is LLM Observability?",
        }
    ],
    model="gpt-3.5-turbo",
)

import os
from anthropic import Anthropic
import shield360

shield360.init()

client = Anthropic(
    # This is the default and can be omitted
    api_key=os.environ.get("ANTHROPIC_API_KEY"),
)

message = client.messages.create(
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Hello, What is LLM Observability?",
        }
    ],
    model="claude-3-opus-20240229",
)

import cohere
import shield360

shield360.init()

co = cohere.Client(
    api_key="YOUR_API_KEY",
)

chat = co.chat(
    message="hello world!",
    model="command"
)

from litellm import completion
import os
import shield360

shield360.init()

os.environ["HUGGINGFACE_API_KEY"] = "huggingface_api_key"

# e.g. Call 'WizardLM/WizardCoder-Python-34B-V1.0' hosted on HF Inference endpoints
response = completion(
  model="huggingface/WizardLM/WizardCoder-Python-34B-V1.0",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  api_base="https://my-endpoint.huggingface.cloud"
)

from langchain_core.messages import HumanMessage, SystemMessage
import shield360

shield360.init()

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o-mini")

messages = [
    SystemMessage(content="Translate the following from English into Italian"),
    HumanMessage(content="hi!"),
]

model.invoke(messages)

import ollama
import shield360

shield360.init()

response = ollama.chat(model='llama3.1', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])

Zero-code instrumentation

Via CLI arguments

# Install Shield360
pip install shield360

# Configure via CLI arguments
shield360-instrument \
  --service-name my-ai-app \
  --environment production \
  --otlp-endpoint http://127.0.0.1:4318 \
  python your_app.py

Via environment variables

# Configure via environment variables
export OTEL_SERVICE_NAME=my-ai-app
export OTEL_DEPLOYMENT_ENVIRONMENT=production
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318

# Run with zero code changes
shield360-instrument python your_app.py

Typescript

SDK Integration

Via function parameters

import Shield360 from "shield360"

Shield360.init({ otlpEndpoint: "http://127.0.0.1:4318" })

Examples:

import Shield360 from "shield360"

Shield360.init({ otlpEndpoint: "http://127.0.0.1:4318" })

async function main() {
  const OpenAI = await import("openai").then((e) => e.default);
  const openai = new OpenAI({
      apiKey: process.env.OPENAI_API_KEY,
  });
  const completion = await openai.chat.completions.create({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: "What is LLM Observability?" }],
  });

  console.log(completion?.choices?.[0]);
}

main();

import Shield360 from "shield360"

Shield360.init({ otlpEndpoint: "http://127.0.0.1:4318" })

async function main() {
  const Anthropic = await import("@anthropic-ai/sdk").then((e) => e.default);
  const anthropic = new Anthropic({
      apiKey: process.env.ANTHROPIC_API_KEY,
  });
  const message = await anthropic.messages.create({
      max_tokens: 1024,
      messages: [{ role: "user", content: "Hello, What is LLM Observability?" }],
      model: "claude-3-opus-20240229",
  });

  console.log(message);
}

main();

import Shield360 from "shield360"

Shield360.init({ otlpEndpoint: "http://127.0.0.1:4318" })

async function main() {
  const { CohereClient } = await import("cohere-ai");
  const cohere = new CohereClient({
      token: process.env.COHERE_API_KEY,
  });
  const chat = await cohere.chat({
      message: "hello world!",
      model: "command",
  });

  console.log(chat);
}

main();

import Shield360 from "shield360"

Shield360.init({ otlpEndpoint: "http://127.0.0.1:4318" })

async function main() {
  const { ChatOpenAI } = await import("@langchain/openai");
  const { HumanMessage, SystemMessage } = await import("@langchain/core/messages");

  const model = new ChatOpenAI({ model: "gpt-4o-mini" });

  const messages = [
      new SystemMessage("Translate the following from English into Italian"),
      new HumanMessage("hi!"),
  ];

  await model.invoke(messages);
}

main();

import Shield360 from "shield360"

Shield360.init({ otlpEndpoint: "http://127.0.0.1:4318" })

async function main() {
  const ollama = await import("ollama").then((e) => e.default);
  const response = await ollama.chat({
      model: "llama3.1",
      messages: [{ role: "user", content: "Why is the sky blue?" }],
  });

  console.log(response);
}

main();

Tip

Using static top-level imports instead (e.g. import OpenAI from "openai")? Static imports are hoisted and run before Shield360.init(), so the automatic patch can miss the module. Pass it into instrumentations so Shield360 patches it directly:

import shield360 from "shield360"
import OpenAI from "openai"

shield360.init({
  otlpEndpoint: "http://127.0.0.1:4318",
  instrumentations: {
    openai: OpenAI,
  }
})

async function main() {
  const openai = new OpenAI({
      apiKey: process.env.OPENAI_API_KEY,
  });
  const completion = await openai.chat.completions.create({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: "What is LLM Observability?" }],
  });

  console.log(completion?.choices?.[0]);
}

main();

The same pattern works for anthropic, cohere, and ollama. LangChain always uses the dynamic-import pattern above, since it patches via a different hook.

Via environment variables

Add the following two lines to your application code:

import shield360 from "shield360"

shield360.init()

Run the following command to configure the OTEL export endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT = "http://127.0.0.1:4318"

Examples:

import shield360 from "shield360"

shield360.init()

async function main() {
  const OpenAI = await import("openai").then((e) => e.default);
  const openai = new OpenAI({
      apiKey: process.env.OPENAI_API_KEY,
  });
  const completion = await openai.chat.completions.create({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: "What is LLM Observability?" }],
  });

  console.log(completion?.choices?.[0]);
}

main();

import shield360 from "shield360"

shield360.init()

async function main() {
  const Anthropic = await import("@anthropic-ai/sdk").then((e) => e.default);
  const anthropic = new Anthropic({
      apiKey: process.env.ANTHROPIC_API_KEY,
  });
  const message = await anthropic.messages.create({
      max_tokens: 1024,
      messages: [{ role: "user", content: "Hello, What is LLM Observability?" }],
      model: "claude-3-opus-20240229",
  });

  console.log(message);
}

main();

import shield360 from "shield360"

shield360.init()

async function main() {
  const { CohereClient } = await import("cohere-ai");
  const cohere = new CohereClient({
      token: process.env.COHERE_API_KEY,
  });
  const chat = await cohere.chat({
      message: "hello world!",
      model: "command",
  });

  console.log(chat);
}

main();

import shield360 from "shield360"

shield360.init()

async function main() {
  const { ChatOpenAI } = await import("@langchain/openai");
  const { HumanMessage, SystemMessage } = await import("@langchain/core/messages");

  const model = new ChatOpenAI({ model: "gpt-4o-mini" });

  const messages = [
      new SystemMessage("Translate the following from English into Italian"),
      new HumanMessage("hi!"),
  ];

  await model.invoke(messages);
}

main();

import shield360 from "shield360"

shield360.init()

async function main() {
  const ollama = await import("ollama").then((e) => e.default);
  const response = await ollama.chat({
      model: "llama3.1",
      messages: [{ role: "user", content: "Why is the sky blue?" }],
  });

  console.log(response);
}

main();

Tip

Using static top-level imports instead (e.g. import OpenAI from "openai")? Static imports are hoisted and run before shield360.init(), so the automatic patch can miss the module. Pass it into instrumentations so Shield360 patches it directly:

import shield360 from "shield360"
import OpenAI from "openai"

shield360.init({
  instrumentations: {
    openai: OpenAI,
  }
})

async function main() {
  const openai = new OpenAI({
      apiKey: process.env.OPENAI_API_KEY,
  });
  const completion = await openai.chat.completions.create({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: "What is LLM Observability?" }],
  });

  console.log(completion?.choices?.[0]);
}

main();

The same pattern works for anthropic, cohere, and ollama. LangChain always uses the dynamic-import pattern above, since it patches via a different hook.

Refer to Shield360 SDK configuration reference or TypeScript SDK reference for more advanced configurations and use cases.

Monitor, debug and test the quality of your AI applications

With real-time LLM observability data now flowing to Shield360, visualize comprehensive AI performance metrics including token costs, latency patterns, hallucination rates, and model accuracy to optimize your production AI applications.

Just head over to Shield360 at 127.0.0.1:3000 on your browser to start exploring. You can login using the default credentials

• Email: user@shield360.ai
• Password: shield360user

You’re all set! Your AI applications now have observability with real-time performance monitoring, cost tracking, and AI safety evaluations.

Send Observability telemetry to other OpenTelemetry backends

flowchart TB;
    subgraph " "
        direction LR;
        ApplicationCode[Application Code] -->|Instrumented with| Shield360_SDK[Shield360 SDK];
        Shield360_SDK -->|Sends Traces & Metrics| OT_Backend[OpenTelemetry Backend];
    end

If you wish to send telemetry directly from the SDK to another backend, you can stop the current Docker services by using the command below. For more details on sending the data to your existing OpenTelemetry backends, checkout our Supported Destinations guide.

docker compose down

If you have any questions or need support, reach out to our community.

---

Quickstart: LLM Evaluations

Get started with evaluating your LLM responses in 2 simple steps

Integrations

60+ AI integrations with automatic instrumentation and performance tracking

Destinations

Send telemetry to Datadog, Grafana, New Relic, and other observability stacks