Alerts and notifications

Alerts (/alerts, under Detect & respond → Alerts) watches your project and tells people when something needs them. It has these tabs:

Inbox — every alert that fired in this project, newest first. Filter by status and severity, open the page the alert is about, and acknowledge, snooze or silence it.
Incidents and On-call — alerts that match an escalation policy open incidents that page whoever is on call until someone acknowledges. See Incidents and on-call.
Rules (Alert rules in the menu) — what the project watches and where each rule sends.
Channels (Notification channels) — where alerts go: email, Slack, Microsoft Teams, a webhook or PagerDuty.
Maintenance (Maintenance windows) — times when alerts are recorded but not sent. See Maintenance windows.
Routing test (Alert routing test) — why an alert was sent, or not. See Routing test.
Provider findings — what your monitoring provider reported, under a managed service. See Provider findings.

Rules and the inbox belong to the project (and data source) you are in. Channels belong to the organisation, so a rule in any project can use them. The Home page shows open alerts and the latest three.

Permissions

Permission	Allows
View alerts (`alerts.read`)	The inbox, and seeing rules and channels. Every shipped role has it.
Manage alerts (`alerts.manage`)	Creating, changing and deleting rules and channels, sending test alerts, and acknowledging alerts. Owners and administrators have it.

Give Manage alerts to a custom role (the Roles section of Settings → Organisation → Organisation) for on-call engineers who should acknowledge alerts without being administrators.

Rules

Add a channel

On Channels, choose Add channel, pick the type, and paste what it needs (see Channels). Use Send test to check it arrives.

Add a rule

On Rules, choose Add rule, pick what to watch, set the threshold, a severity and the channels to send to.

Wait for the next check

Most rules are checked every few minutes; span rules within seconds. When one fires it appears in the inbox and is sent to its channels at once.

Rule	Fires when	Checked every
Spend over an amount	AI spend in the period (from the Costs report) goes over the amount	15 min
Spend spike	Spend in the period is up by at least the percentage on the period before, and at least the minimum amount	15 min
Blocked or redacted prompts	The guard blocks, redacts, or either, at least this many prompts in the period	5 min
Collector stopped reporting	Any enrolled computer has not checked in for longer than this many hours	15 min
New AI tool or model	A tool or model is used for the first time. The first check only learns what is already in use. One alert per new tool or model.	15 min
Failed sign-ins	At least this many sign-ins to Shield360 fail in the period (from the audit log)	5 min
Gateway calls denied	The agent gateway denies at least this many calls in the period	5 min
Evaluation failures	At least this many evaluations fail in the period	15 min
Alerts raised by features	A feature (budgets, the AI inventory, prompt-injection detection) raises an alert of the chosen kinds and at least the chosen severity	as they happen
Error rate, latency, tokens or cost	Error rate, errors, requests, p95 or average latency, tokens or cost over the last 30 seconds to an hour crosses the threshold, optionally per service, model or attribute. See Span health.	15 s
Application logs: count or error rate	Your application’s logs — all of them, only errors, or the share that are errors — over the last 30 seconds to an hour cross the threshold, optionally per service, severity or environment. See Log health.	15 s
A single span matches	One span matches every condition: one call over a cost, an error on chosen models or services, a token count over a number, an attribute with a value. See Span rules.	every span, as it arrives

Every rule reads the same numbers as the page it links to, so an alert never disagrees with the screen.

Firing, repeating and resolving

A threshold rule holds one open alert at a time. While the condition stays true, each check is counted on it (occurrences) rather than raising a new one.
Repeat at most every (default 60 minutes) is how often a still-firing alert is sent again. It also stops a metric that flaps around its threshold from sending on every check.
When the condition clears, the alert is resolved, and every channel that was told it fired is told it resolved (PagerDuty incidents are resolved).
Turning a rule off, or deleting it, closes its open alert without sending anything.

Quiet hours

A rule can have quiet hours, such as 22:00 to 07:00 in your time zone. Alerts below Critical raised in that span are recorded at once and sent when it ends. Critical alerts are always sent at once.

Delivery

A notification is sent the moment its alert is raised (or resolved), not on the next scheduled tick. The once-a-minute delivery job is the retry path: a send that fails with a timeout, a 5xx or a 429 is tried again by it after 30 s, 1 min, 2 min and so on, up to an hour apart. Each notification is claimed before it is sent, so the immediate send and the job never both send one. Maintenance windows, snoozes, flapping and quiet hours apply exactly as before: they are checked when the notification is queued and again when it is sent.

Variable	Default	Meaning
`ALERTS_IMMEDIATE_DELIVERY`	on	`false` sends everything from the delivery job instead (up to a minute later).
`ALERTS_DELIVER_SCHEDULE`	`* * * * *`	The delivery job: retries, and notifications that waited out quiet hours.

Span rules

A single span matches checks every span as it arrives, rather than reading a number on a schedule, and fires within seconds.

Each rule has up to 10 conditions, all of which must hold for one span:

Field	Meaning
`service`, `model`, `provider`, `operation`, `span_name`, `environment`	Text fields. `model` is the requested model (or the response’s), `provider` the GenAI provider.
`status`	`ok`, `error` or `unset`.
`cost`, `input_tokens`, `output_tokens`, `total_tokens`, `duration_ms`, `http_status`	Numbers. `cost` is the span’s cost in USD after pricing at ingest.
any span or resource attribute	For example `gen_ai.tool.name` or `http.route`.

Comparisons are is, is not, is one of, is none of, contains, is set for text (case does not matter), and >, ≥, <, ≤ for numbers.

Some attributes can never be used, whatever is configured: prompts, completions, messages, content, instructions, and anything that looks like a secret (passwords, tokens, API keys, authorization headers, cookies). Attributes that identify people or private places (user, session and conversation ids, email, phone, IP addresses, URL paths) can be used only when listed in SHIELD360_SIGNAL_ATTRIBUTES.

Separate alerts per (up to three fields, such as model) gives each value its own alert; empty means one alert for the rule. Matches within the rule’s repeat at most every interval (from 1 minute) are counted on the open alert rather than sent again.

How it works: the bundled collector’s shield360enrich processor fetches the enabled span rules for the database it writes to every 10 seconds, checks each span after it has been mapped and priced, and posts matches to the platform once a second. Telemetry the platform writes itself (every other tenant’s) is checked by the platform as it is written. Either way the alert is raised and sent at once.

Step	Typical time
SDK export to collector	up to the SDK’s batch delay (often 1–5 s)
Collector match to platform	under 1 s (measured 0.06–0.96 s)
Platform to channel	the channel’s own response time (immediate delivery)

Limits, so a noisy rule cannot flood the platform: 50 enabled span rules per data source; at most 100 distinct groups per rule per second in the collector and per batch in the platform; and 120 alerts per minute per data source (SHIELD360_SPAN_ALERTS_PER_MINUTE). Matches past a limit are dropped and counted, never queued. Checking rules never delays or drops telemetry: if the platform cannot be reached, the collector keeps its last rules and carries on; if the platform refuses the secret, the collector stops checking until it is accepted.

Variable	Where	Meaning
`SHIELD360_SIGNAL_SECRET`	app and collector	Shared secret, at least 32 characters, for `/api/internal/alerts/span-rules` and `/api/internal/alerts/span-matches`. Generated on first start in a single container (`SHIELD360_ROLE=all`). Without it, those routes answer `403` and the collector does not check spans; the platform still checks the telemetry it writes.
`SHIELD360_SIGNAL_URL`	collector	The app’s URL. Defaults to the container’s own app in `SHIELD360_ROLE=all`; set it to the web service on `ingest` pods.
`SHIELD360_SIGNAL_ATTRIBUTES`	app	Comma-separated personal attributes rules may use, such as `user.id`. At most 32.
`SHIELD360_SPAN_ALERTS_PER_MINUTE`	app	Span-rule alerts per minute per data source. Default 120.

Span health

Error rate, latency, tokens or cost reads the data source’s own spans over a short window on a fast cadence, every 15 seconds by default, and compares one number with the threshold:

Measure	Value
Error rate	Failed spans ÷ all spans, from 0 to 1 (0.25 is 25%)
Errors, Requests	Failed spans, all spans
p95 latency, Average latency	Span duration in milliseconds
Tokens	Input plus output tokens (or the reported total)
Cost	Cost in USD, priced at ingest

Window: 30 seconds to an hour.
Fire when: above, at least, below or at most the threshold.
Only spans where: optional filters, written like span-rule conditions.
Separate alerts per: up to three fields; each group holds its own alert, at most 20 per reading. Without it, the rule is one number and fires, repeats, resolves, and uses hysteresis and flap detection like any threshold rule.
Only with at least this many spans: a group with fewer spans in the window is not judged (and an open alert for it resolves). 0 judges an empty window too, so “Requests below 10” can catch traffic stopping.
Repeat at most every: from 1 minute.

Start from a template in the rule form fills in a common rule: error rate per service, p95 latency per model, a token or cost spike per model, an expensive single call, errors on a model.

The fast cadence runs inside every process that runs scheduled jobs (SHIELD360_ROLE all or worker), with no new service. Each rule is claimed before it is read, so several workers never read one twice; a run that is still going when the next tick comes skips that tick; and a run stops claiming rules after 12 seconds, leaving the rest for the next tick, oldest first. Each query reads only the rule’s window and is stopped by ClickHouse after 10 seconds. Every other rule type keeps its own schedule.

Expected time from a span leaving the SDK to the alert reaching a channel: the SDK’s batch delay, plus up to 5 seconds in the collector’s batch before the span is in ClickHouse, plus up to 15 seconds to the next tick, plus the query: typically 10–25 seconds.

Variable	Default	Meaning
`ALERTS_FAST_EVALUATE_SCHEDULE`	`/15 * * * *`	When span-health rules are read: a cron expression with a seconds field. `off` turns them off.

Log health

Application logs: count or error rate reads the data source’s own application logs (Telemetry → Logs) over a short window, on the same fast cadence as span health, and compares one number with the threshold:

Measure	Value
Logs	Log records that match the filters
Error logs	Of those, the ones at error or fatal
Error log rate	Error logs ÷ logs, from 0 to 1 (0.1 is 10%)

A log’s level is its OpenTelemetry severity number when it has one, and otherwise its severity text (WARNING, Error, CRITICAL and the like), read as one of trace, debug, info, warn, error or fatal.

Only logs where: optional filters on service, severity (is, is one of, or is at least a level), environment, scope (the logger name), body (the message contains a piece of text), or a log or resource attribute such as exception.type. Sensitive attributes are refused, and personal ones (user ids, email, IP) need SHIELD360_SIGNAL_ATTRIBUTES, as in span rules.
Separate alerts per: service, severity, environment, scope or an attribute — never the message itself, so an alert carries counts and names, not log text.
Window, Fire when, Only with at least this many logs and Repeat at most every work as in Span health.

Start from a template fills in a log error rate per service, error logs per service, or a message that contains a piece of text at warn or above. Log rules count toward the 50 span rules a data source may have enabled.

Inbox

Each alert shows its severity, what happened, when it fired (and resolved), how often it repeated, and whether it reached its channels. Open goes to the page it is about.

Acknowledge records that someone has seen to it; it moves from Open to Acknowledged. Acknowledging does not resolve an alert — a rule still resolves it when the condition clears.

Alerts are kept for 90 days after they close (ALERTS_RETENTION_DAYS on self-hosted installs).

Silence an alert

With Manage alerts, an open alert in the inbox has a Silence menu:

Snooze for 1 hour, 4 hours, 1 day or 7 days: nothing more is sent about the alert (no repeats, no escalation) until then. The alert shows Snoozed until …; End the snooze sends again. Through the API a snooze can last from 5 minutes to 30 days.
Acknowledge, even if it gets worse: a sticky acknowledgement. A normal acknowledgement is cleared when the alert’s severity rises, so the alert is open again.
Acknowledge for 24 hours: the acknowledgement lapses after a day and the alert is open again.
Unacknowledge takes an acknowledgement back.

Select several alerts to acknowledge, unacknowledge, snooze or end the snooze of up to 100 at once. Where was it sent? beside an alert opens it in the routing test.

Maintenance windows

A maintenance window holds back notifications during planned work. While it is on, the alerts it covers are still recorded and shown in the inbox, marked In maintenance, but nothing is sent for them. Windows belong to the project. You need Manage alerts to add or change one.

Open Maintenance windows

Go to Detect & respond → Alerts → Maintenance windows (the Maintenance tab) and choose Add maintenance window.

Say what it is for

Give it a Name (up to 120 characters) and a Reason (up to 500), such as database upgrade, change CHG-1234. Both are required; the list shows the reason and who set the window up.

Set the schedule

Under Schedule, choose:

Once: the Starts and Ends times. It may last up to 90 days, and must not have ended already.
Every week: the days it Starts on, the times From and Until, and the Time zone (your browser’s by default). An end before the start runs past midnight into the next day: 22:00 to 06:00 on Friday covers Friday night into Saturday morning. Times follow daylight saving in that zone.

Choose what it covers

Under Covers, choose Every alert in this project, Some rules (tick them; up to 50) or Some kinds of alert (tick them; up to 50). Turn on Still send critical alerts to let critical alerts through.

Save it

Select Save. The list shows whether the window is On until …, its Next start, or Ended. Use the switch to turn a window off without removing it, the pencil to edit it, and the bin to remove it. When it is removed, alerts it covered are sent again.

A project holds up to 100 maintenance windows. The same windows can be managed through GET and POST /api/alerting/maintenance and PATCH and DELETE /api/alerting/maintenance/<id>; reading needs View alerts, changes need Manage alerts and are recorded in the audit log.

Routing test

The routing test answers “why was an alert sent, or not?” without sending anything. It checks every alert rule in the project the way routing does.

Open the routing test

Go to Detect & respond → Alerts → Alert routing test (the Routing test tab), or choose Where was it sent? on an alert in the inbox, which fills in its id and explains it at once.

Pick or describe an alert

Enter an Alert id from the inbox, or leave it empty and choose a Kind and a Severity to describe an alert that has not happened.

Explain

Select Explain. A summary says how many channels the alert would reach. Below it, each rule shows an outcome and the reasons for it:

Outcome	Means
Sent now	The rule matches and would send at once
Sent later	It matches, but quiet hours delay it until the time shown
Held back	It matches, but a snooze, a flapping rule, a maintenance window or the rule’s repeat interval holds it
Not sent	The rule does not handle that kind, the severity is below its minimum, the alert came from another rule, the rule is off, or it has no enabled channels

A matching rule lists its channels and, if it escalates, where and after how long.

Send a real test, if you need to

With Manage alerts, the send button beside a channel sends a test of this kind and severity to it for real, after you confirm. A PagerDuty channel opens an incident. Each send is recorded in the audit log.

The routing test only covers the project you are in, and reading it needs View alerts. Through the API it is POST /api/alerting/explain with { "alertId": "…" } or { "kind": "…", "severity": "warning" }, and POST /api/alerting/explain/send with { kind, severity, channelId }.

Provider findings

When your organisation uses a managed service, its provider’s analysts report what they saw on Detect & respond → Alerts → Provider findings. Each new finding also raises a managed.finding alert, which you can route like any other: a critical finding raises a critical alert, a high or medium one a warning, and a low or info one an info alert.

Each finding shows its severity (Info, Low, Medium, High or Critical), what it is about (AI spend, Shadow AI, Data protection, AI agents, Coverage, Compliance or Other), its status, who reported it and when, what was seen and the provider’s Recommendation, with links to related alerts, a related case and analyst findings. Filter the list by Status.

To act on one, you need Act on provider findings:

Read it

Open Provider findings and find the finding, filtered by Status if you like (Open, Acknowledged, Resolved, Dismissed or All).

Add a note

Optionally type a note of up to 1,000 characters in the Add a note (optional) box beside the buttons; it is saved with the status change and shown on the finding.

Change its status

Acknowledge it once you have seen it, Resolve it when it is dealt with, or Dismiss it if it does not apply. A resolved or dismissed finding can be Reopened.

Only the provider’s staff report findings (Report a finding, shown to them alone); your own people open cases instead, and the provider cannot change a finding’s status. Seeing findings needs View provider findings. The API is GET /api/managed/findings?status=open and PATCH /api/managed/findings/<id> with { "status": "acknowledged", "note": "…" }.

Channels

What a channel stores — a webhook URL, an integration key, a signing secret — is encrypted and never shown again after you save it. The list shows a masked hint with none of the secret’s characters: a webhook’s host (for example hooks.slack.com/…) or •••• for an integration key. When you edit a channel, leave the field blank to keep what is stored.

Alerts are never sent to private, local or cloud metadata addresses: the destination is checked when you save it and again, including every redirect, each time it is sent.

With the Shield360 app (recommended). Under Channels, click Add to Slack and approve the app for your workspace. Then add a Slack channel, choose Shield360 app, and pick a channel from the list. Alerts are posted by the Shield360 app. It can post to any public channel. To use a private channel, invite the app there first with /invite @Shield360. Disconnect removes the app from the workspace and switches off the channels that used it.

With a webhook URL. In Slack, create an app with Incoming Webhooks, add a webhook to the channel, and paste its URL (https://hooks.slack.com/services/…). Use this when the Shield360 app is not available on your server. A self-hosted administrator enables the app by setting SLACK_CLIENT_ID and SLACK_CLIENT_SECRET.

With the Shield360 bot (recommended).

Under Channels, click Download Teams app. In Teams, upload it (Apps → Manage your apps → Upload an app) and add it to a team.
Back in Shield360, click Link a team, and post the message it shows, for example @Shield360 link ABCD-EFGH, in any channel of that team. The code works once, for 15 minutes.
Add a Teams channel, choose Shield360 bot, and pick the team and a channel.

Alerts arrive as Adaptive Cards from Shield360. Unlink, or removing the app from the team, switches off the channels that used it.

With a webhook URL. In the Teams channel, add the Workflows template Post to a channel when a webhook request is received and paste the URL it gives (…logic.azure.com/… or …api.powerplatform.com/…). Older Incoming Webhook connector URLs (…webhook.office.com/…) are still accepted, but Microsoft is retiring Office 365 connectors in Teams, so use Workflows for new channels and move existing ones. Use this when the Shield360 bot is not available on your server. A self-hosted administrator enables the bot by setting TEAMS_BOT_APP_ID and TEAMS_BOT_APP_PASSWORD.

Paste an https:// URL and a signing secret of at least 16 characters (Generate makes one). Copy the secret before saving — it is not shown again. See Receiving webhooks.

Receiving webhooks

Shield360 sends a POST with a JSON body:

{
  "id": "cm1delivery0000000000000",
  "type": "alert.fired",
  "createdAt": "2026-09-27T12:00:05.000Z",
  "alert": {
    "id": "cm1alert000000000000000",
    "kind": "spend.threshold",
    "severity": "warning",
    "status": "firing",
    "title": "AI spend is $1,250.00 in the last 1 day, over $1,000.00",
    "body": null,
    "url": "https://app.shield360.ai/costs",
    "firedAt": "2026-09-27T12:00:00.000Z",
    "resolvedAt": null,
    "occurrences": 1,
    "value": 1250,
    "projectId": "cm1project000000000000000",
    "projectName": "Production",
    "attributes": { "spent": 1250, "threshold": 1000, "windowHours": 24 }
  }
}

type is alert.fired, alert.resolved or alert.test. A delivery that fails (a timeout, a 5xx or a 429) is retried with backoff for about an hour; id stays the same, so drop a delivery you have already processed. A 4xx other than 408 and 429 is not retried.

Verifying the signature

Every delivery has these headers:

Header	Value
`X-Shield360-Timestamp`	Unix time in seconds when it was sent
`X-Shield360-Signature`	`v1=` and the hex HMAC-SHA256 of `{timestamp}.{raw body}` with your signing secret
`X-Shield360-Event`	`alert.fired`, `alert.resolved` or `alert.test`
`X-Shield360-Delivery`	The delivery id (the body’s `id`)

To verify: take the raw request body before parsing it, compute HMAC-SHA256(secret, timestamp + "." + body), compare it with the signature in constant time, and reject a timestamp more than five minutes from your clock.

import { createHmac, timingSafeEqual } from "node:crypto";

export function verifyShield360(rawBody: string, headers: Headers, secret: string): boolean {
  const timestamp = headers.get("x-shield360-timestamp") ?? "";
  const signature = headers.get("x-shield360-signature") ?? "";
  const age = Math.abs(Date.now() / 1000 - Number(timestamp));
  if (!/^\d+$/.test(timestamp) || age > 300) return false;
  const expected = "v1=" + createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex");
  const a = Buffer.from(expected);
  const b = Buffer.from(signature);
  return a.length === b.length && timingSafeEqual(a, b);
}

import hashlib, hmac, time

def verify_shield360(raw_body: bytes, headers, secret: str) -> bool:
    timestamp = headers.get("X-Shield360-Timestamp", "")
    signature = headers.get("X-Shield360-Signature", "")
    if not timestamp.isdigit() or abs(time.time() - int(timestamp)) > 300:
        return False
    message = timestamp.encode() + b"." + raw_body
    expected = "v1=" + hmac.new(secret.encode(), message, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

Email setup

Self-hosted installs set these environment variables; hosted deployments use Amazon SES.

Variable	Meaning
`ALERTS_EMAIL_FROM`	The sender, e.g. `Shield360 Alerts <alerts@example.com>`. Required.
`ALERTS_EMAIL_TRANSPORT`	`ses` or `smtp`. Optional: an SMTP host means SMTP, an AWS region means SES.
`ALERTS_SES_REGION`	SES region (falls back to `AWS_REGION`). The server’s AWS identity needs `ses:SendEmail`, and the sender’s domain must be verified in SES.
`ALERTS_SMTP_HOST`, `ALERTS_SMTP_PORT`, `ALERTS_SMTP_SECURE`, `ALERTS_SMTP_USER`, `ALERTS_SMTP_PASSWORD`	Your SMTP server.
`SHIELD360_PUBLIC_URL`	The address links in alerts point to (falls back to `NEXTAUTH_URL`).

For developers: raising alerts from a feature

Features raise alerts with emitAlertEvent from src/client/src/lib/alerts/emit.ts. It never throws, so it is safe in routes, scheduled jobs and ingest; repeats with the same dedupeKey within the kind’s window are counted on one alert rather than raised again.

import { emitAlertEvent } from "@/lib/alerts/emit";

await emitAlertEvent({
  organisationId,
  projectId,                         // optional
  databaseConfigId,                  // optional
  kind: "budget.exceeded",           // registered in lib/alerts/kinds.ts
  severity: "critical",              // "info" | "warning" | "critical"
  title: "Engineering is over its monthly AI budget",
  body: "Spent $5,210 of $5,000.",
  link: "/costs?tab=budgets",
  dedupeKey: `budget:${budgetId}:${month}`,
  attributes: { budget: "Engineering", spent: 5210 },
});

The alert appears in the inbox at once, and is sent to channels by any Alerts raised by features rule in the project that matches its kind and severity. Never put credentials, prompts or personal data in an alert: it is shown to everyone who can view alerts and sent to external services.

Costs

Where spend alerts come from

Visibility

Collectors, and the computers that stop reporting