Incidents and on-call

Incidents (/alerts/incidents) and On-call (/alerts/on-call) are tabs of Detect & respond → Alerts. Alerts tell you something happened; an incident is the thing a person owns until it is over.

How an alert becomes an incident

1. An alert is raised in a project: an uptime check failed, an error-rate or log rule fired, a server crossed a CPU threshold, or any other alert.
2. The project’s escalation policies are checked, oldest first. The first enabled policy whose alert kinds, least severity and (optional) services match it decides.
3. If an incident of the same group is already open, the alert joins it. Otherwise a new incident is triggered and the policy’s first level is paged.

An alert raised inside one of the project’s maintenance windows opens no incident. Without a matching policy, alerts are only sent to their rules’ own channels, as before.

Grouping

A policy groups alerts into one incident per:

Option	One incident for
Service (default)	every matching alert about the same app or service (its service attribute), or the same kind when it names none
Kind of alert	every matching alert of the same kind
Alert	each alert on its own

The incident’s severity is the worst of its alerts.

States

State	Meaning
Triggered	Nobody has acknowledged it. Escalation is running.
Acknowledged	Someone is on it. Escalation stops; everyone paged is told.
Resolved	Over. Everyone paged is told. A later alert of the same group opens a new incident.

An incident resolves automatically once every alert in it has resolved (an uptime check recovered, a rule’s condition cleared), unless its policy turns that off. Alerts features raise once (an injection detected, a budget exceeded) never resolve on their own, so their incidents wait for a person.

Every change is on the incident’s timeline, with who did it and when: triggered, alerts added, people and channels paged, escalations, acknowledgement, resolution and notes. Every change is also in the audit log.

Escalation policies

A policy has up to five levels. Each level pages any mix of:

• an on-call schedule (whoever it has on call at that moment),
• people in your organization (by email),
• alert channels: email, Slack, Microsoft Teams, PagerDuty or a signed webhook.

If nobody acknowledges within a level’s escalate after minutes, the next level is paged. After the last level, the policy can repeat from level 1 up to three times. Escalations are checked every 15 seconds.

On-call schedules

A schedule is a simple rotation: the people in its list take turns of a fixed length (7 days for weekly), starting at the time you choose, and the order wraps. An override puts someone else on call for a while (up to 90 days); when overrides overlap, the newest wins. The On-call tab shows who is on call now and until when.

Acknowledge and resolve from the notification

Email, Slack and Teams notifications about an incident carry Acknowledge and Resolve links, so whoever is paged can act without signing in. Each link:

• is signed by your Shield360 deployment and works once,
• expires after 24 hours,
• asks you to confirm on a page first, so a mail scanner that opens links acknowledges nothing,
• acts as the person it was sent to (or, for a channel, as “someone with the link”), and stops working for anyone who has left the organization.

You can always acknowledge, resolve and add notes on the incident in Shield360 too (this needs the Manage alerts permission).

Note

Links need the deployment’s public address (SHIELD360_PUBLIC_URL or NEXTAUTH_URL) and a signing secret: SHIELD360_INCIDENT_LINK_SECRET, or NEXTAUTH_SECRET when that is not set. Without them notifications are sent without links.

Keep PagerDuty as your pager

If your team already pages through PagerDuty, add a PagerDuty channel (an Events API v2 integration key) to a policy level. Shield360 then sends PagerDuty:

Shield360	PagerDuty Events API v2
Incident triggered or escalated	trigger
Incident acknowledged	acknowledge
Incident resolved	resolve

All of them carry one dedup_key per Shield360 incident (shield360-incident-<id>), so PagerDuty keeps one incident for it, however many alerts join and however often it escalates. PagerDuty’s own escalation and schedules then take over. Acknowledging in PagerDuty does not acknowledge the incident in Shield360.

Tip

If a PagerDuty channel is also on an alert rule, PagerDuty receives that rule’s alerts under their own dedup keys as well. Put PagerDuty on the escalation policy or on the rules, not both.

Webhooks

A signed webhook channel on a policy level receives incident.triggered, incident.acknowledged and incident.resolved events (header X-Shield360-Event) with the alert and an incident object: id, status, service, alertCount, level and actorEmail. Webhooks never receive the acknowledge and resolve links.

Limits

	Per project
Escalation policies	50
Levels per policy	5, each with up to 10 targets
Schedules	50, each with up to 50 people and 100 upcoming overrides

Resolved incidents are kept as long as alerts (ALERTS_RETENTION_DAYS, 90 days by default).