AI analyst checks: plausibility, cost waste, policy tuning and unknown AI

Overview

Besides investigating alerts, the AI analyst runs four sets of checks over each environment’s data once a day:

• The plausibility auditor flags figures that cannot be right, such as more AI time than a day has, or one machine counted three times. These are usually collection or attribution problems. Fix them before you rely on the numbers built on them.

• The cost waste finder looks for money the data says could be saved, such as idle seats, a premium model doing light work, or prompt caching left off. Each finding gives a monthly estimate as a range, and a concrete recommendation.

• Policy tuning reads how the environment’s enforcement policy, approved list and detection rules actually behaved over the last 30 days, and suggests changes with the evidence. See Policy tuning.

• Unknown AI classifies website domains, desktop apps and MCP servers first seen this week that are not in Shield360’s AI catalogue. See Unknown AI.

All four add findings to Detect & respond → Detections → Analyst findings. Filter by Plausibility check, Cost waste, Policy tuning or Unknown AI. With the inbox filtered to Cost waste, the top of the page shows the possible savings: the sum of the open cost findings’ ranges. The executive page can show the same total.

The same daily scan also runs the story pass and behaviour drift.

The checks are deterministic. They are SQL over your data and fixed rules, and they work with the analyst off and with no model configured. The one exception is a name the unknown-AI hints cannot decide, which only the model is asked about (see Unknown AI). When the analyst is on and a model is configured, the analyst also writes a short explanation for up to three new or changed plausibility, cost or policy findings per scan. The explanation is pseudonymised like every other analyst run (see Privacy) and counts against the daily budget.

How findings behave

• One finding per check and subject. A subject is, for example, a person on a machine, a model, or a subscription. A scan that runs again on the same day, or the next day, refreshes the same finding. It does not raise another. Status and feedback are kept.
• Resolved automatically. When a check runs cleanly and no longer finds something, its open finding is resolved and marked Resolved by a later scan. If a check could not run, for example because a query failed, its findings are left as they were.
• Reopened when it comes back. A resolved finding whose condition returns becomes new again.
• Dismissed stays dismissed. A dismissed finding is not raised again unless its evidence changes materially. For example, more people or machines are affected, or the estimate roughly doubles.
• At most 25 findings per check per scan, the most important first.

Findings name people and machines through aliases, such as person-1, unless your organisation turned pseudonymisation off. Real names are shown only to roles that may see people.

Running the checks

The checks run once a day for every environment. The job runs every hour and scans the environments whose last scan started more than 23 hours ago, up to 20 per run. Self-hosted installations can change the schedule with AI_ANALYST_CHECKS_SCHEDULE. Each scan holds a lease on its environment, so two scans of one environment never overlap.

A role that may triage findings can select Run checks now in Findings. This runs all four sets, and the story pass and behaviour drift. It is limited to once every ten minutes per environment, and not while a scan is running. The run is recorded in the audit log.

The same hourly job also writes the weekly briefing for each environment once its Monday has come. There is no second scheduler.

Plausibility checks

Days are UTC days.

Identical durations

What it flags: Two or more AI websites or apps credited with the same time, to within a minute (60 seconds) of each other and at least 15 minutes each, for the same person on the same machine on the same day. The tolerance catches times the collector’s rounding split by a few seconds, such as 1,050 and 1,025 seconds.

Why it cannot be right: Two services almost never take the same minutes. The collector most likely saw one connection and attributed it to both, because the services share addresses. Gemini and Google AI Studio are a common example.

Evidence: Discover → AI usage. A finding seen on two or more days has high confidence.

Usage without seat

What it flags: In the last 30 days, a person spent at least an hour on the websites or apps of a vendor that sells seats, but has:

• no seat with that vendor under Costs → Subscriptions
• no usage from that vendor’s tools
• no seat in a connected spend connector

There is one finding per vendor.

Verdict: The finding says which explanation is more likely.

• Likely misattribution: The vendor serves its AI from addresses shared with its other products. Today that is Google. The time is probably not AI use.
• Likely shadow AI: For any other vendor, the time is probably real use on personal or free accounts.

Plan usage priced

What it flags: At least 20 responses in the last 30 days where:

• a tool reported a plan, such as Codex signed in with ChatGPT, but its sign-in was not recognised as a plan, so the usage is billed by the token or listed as unpriced; or
• metered usage with tokens was priced at $0, because the model’s price does not cover the kind of token used.

What to do: Add the plan’s seats under Costs → Subscriptions, or fix the price under Costs → Models.

Duplicate enrolment

What it flags: One machine and person reporting under three or more collector ids in the last 7 days, or two ids checking in on the same day on two or more days, or three or more active enrolments.

Why it matters: Every report from the machine is counted more than once. A single reinstall, with two ids overlapping on one day, is not flagged.

Time exceeds wall clock

What it flags: For a person on a machine in the last 7 days, a day with either:

• more than 24 hours of AI website time or application time; or
• more AI website time than the machine’s own active application time.

A tolerance of 10 minutes, and 5% for the second case, allows for reports that straddle midnight. More than 24 hours is critical.

Collector clock skew

What it flags: In the last 2 days, a collector’s heartbeats stamped more than 15 minutes after they arrived (a fast clock), or more than 24 hours before they arrived (a slow clock, or reports held back).

Why it matters: The machine’s activity lands on the wrong days. Turn on automatic time (NTP) on the machine.

Heartbeats without data

What it flags: In the last 7 days, an endpoint collector that checked in on 3 or more days but sent no application time, AI website time, usage or machine state. Proxies are not checked.

What to check: The collector’s collection level, its permissions on the machine (screen and accessibility access on macOS), and its log.

Connector vs collector

What it flags: Over the last 30 days, the collector’s metered estimate for OpenAI, Anthropic or Google is more than 25% and more than $25 above what the vendor’s spend connector billed for API use.

Why it cannot be right: The bill covers every key on the account. Only this direction is flagged. A bill higher than the estimate is normal, because SDK apps and console use also appear on the bill. Compare the two on Costs → Reconcile.

Missing prices

What it flags: A model with no price whose metered usage in the last 30 days reached 1,000,000 tokens or 200 responses. Usage a seat covers is not counted, because its price does not change the bill. Add a price links to Costs → Models.

Cost waste

Estimates are monthly and come from the last 30 days. They are shown as a range. The low end is what the data makes near certain, and the high end is what is possible. Token prices come from your price table (Costs → Models), and the coverage rules are the same as on the Costs page. Usage a seat covers is never counted as metered spend. Each finding lists its own assumptions. A finding is raised only when its high end is at least $10 a month. A high end of $500 or more makes it a warning.

Idle seats

What it flags: For each subscription with a seat price:

• Idle seats: no use of the vendor’s tools, websites or apps on a monitored machine in 30 days.
• Underused seats: use on fewer than 4 days.
• Unassigned seats: seats purchased beyond those assigned.

Seats assigned in the last 14 days are left out. If a spend connector reports the seat active in the window, the seat counts as used.

Estimate: Low is (idle + unassigned) × seat price. High adds the underused seats.

Assumption: Use on unmonitored devices, such as a phone or personal laptop, is not seen. Check with the holder before you reclaim a seat.

Premium model, light work

What it flags: A model on metered usage with at least 200 requests where 40% or more are small: at most 4,000 tokens in (cached or not) and 500 out. The same provider must have a lighter model in your price table. The lighter model is the provider’s most expensive chat model that costs at most 40% of the current model’s output price. That is usually the next tier down, not the cheapest model listed.

Estimate: High is the small requests’ cost now minus their cost on the lighter model. Low is half of that, because not every small request can move.

Missed prompt caching

What it flags: A model on metered usage that receives at least 5 million uncached input tokens a month, with less than 5% of its input read from the cache. The provider must price cache reads below fresh input.

Estimate: 20% (low) to 50% (high) of the uncached input, billed at the cache-read price instead of the input price. The cost of writing to the cache is not counted.

Retry waste

What it flags: Repeated requests: the same tool and model, from the same machine, in the same minute, with exactly the same tokens in and out. Every repeat after the first is counted. A finding needs at least 50 repeats, making up at least 2% of the model’s metered requests.

Estimate: High is the repeats’ cost. Low is half of that, because some identical requests are deliberate.

Plan fit

Two recommendations:

• A seat would cost less. A person’s metered use of a vendor’s tools costs at least 1.5 times the vendor’s largest plan at list price, and they hold no seat with that vendor. High is the metered cost minus the plan price. Low is half of that, because plans have usage limits. The calculation uses calculateSubscriptionSavings, the same as the Costs page.
• Paying by the token would cost less. A seat holder’s covered use, at API prices, is worth no more than 25% of the seat, and they used none of the vendor’s websites or apps. High is the seat price minus that value. Low is nothing, because the seat may be used where Shield360 cannot see. Confidence is low. A seat with no use at all is reported under Idle seats instead.

Reworked AI code

What it flags: A repository where people reworked at least 200 AI-written lines, and at least 20% of the AI-written lines, in 30 days. The same applies to a person whose sessions’ lines were reworked at that volume, at a rate of 30% or more. This uses the AI code figures.

Estimate: The rework’s cost, using the AI code page’s hourly rate and minutes per line, is the high end. How much of it was avoidable is not known, so the low end is nothing. Without an hourly rate, the finding shows hours and no money.

Policy tuning

Policy tuning looks at how policies behaved, not at how they are written. Each finding names what it saw and offers one or more changes. The analyst never applies a change. Selecting an option, for example Mask it · Apply…, opens the screen that makes that change with it filled in:

Change	Opens
A sensitive-data setting, or blocking unapproved models	Govern → Policies & rules, on the current environment’s overrides
Adding to or removing from the approved list	Settings → Collection & deployment → Approved AI providers and models
A detection rule’s threshold or cooldown	The rule in Rule engine → Detections

Check the change and select Save. Saving goes through that screen’s own route. That route checks its own permission (for example Manage agent policies and Change what is collected) and records the change in the audit log. A role that may read findings but not save the change sees the suggestion with a note, and nothing is filled in. When the behaviour clears after the change, the next scan resolves the finding.

Warnings sent anyway

What it flags: A kind of sensitive data that is set to warn in a channel, where at least 60% of at least 20 warnings in 30 days ended with the prompt sent anyway.

Suggests: Mask it or block it in that channel. Coding agents cannot mask a value, so only block is offered there. A kind that is already blocked is left alone. People who retry with the value removed show the block working, not a problem.

Dormant warnings

What it flags: Kinds set to warn on purpose (a per-kind setting, not just the channel’s default) that never fired in 30 days in a channel that was in use.

Suggests: Nothing to apply. It is a low-severity note. If the data should have appeared, check the channel’s guard is running.

Unprotected channel

What it flags: The browser extension, the AI proxy or coding agents in use in the last 30 days (enrolled devices seen, guard decisions, or coding-agent usage) with every kind of sensitive data set to off.

Suggests: Turn on warnings as the channel’s default.

Unapproved model in daily use

What it flags: With the approved list on and the Approved models mode off or warn, a model used on at least 10 of the last 14 days that the list does not cover.

Suggests: Both options: approve it (adds the provider, and the model when the list names models), or block models outside the approved list.

Unused approved entries

What it flags: Approved providers and model patterns that nothing used in the last 90 days, from coding-tool usage and direct connections. It says nothing when nothing at all was reported, and it never offers to remove every provider.

Suggests: Remove them from the list.

Noisy detection rule

What it flags: An enabled detection rule whose alerts’ findings were marked Noise at least three times and at least twice as often as Useful, or that raised 90 or more alerts in 30 days.

Suggests: A threshold rule’s value raised to the median matched value, a baseline rule’s score raised the same way, or, for other kinds, a cooldown four times longer (at least four hours).

Backtest: The finding shows how many alerts the rule raised over the last 30 days as it is, and how many it would have raised with the change. The backtest uses the rule’s own preview, with the cooldown applied per group. At most three rules are backtested per scan.

Unknown AI

Every day the analyst looks for names first seen in the last 7 days that Shield360’s AI catalogue does not know:

• Website domains, from browsing history. Only the domain is used. The history’s paths never leave ClickHouse; the query only counts how many visits went to paths that a model API serves.
• Desktop apps, from focused-application time.
• MCP servers, from coding agents’ tool calls.

Names already in the catalogue are skipped. So are names this environment already approved or rejected, and shared front-end infrastructure. Shared infrastructure means hosts on the same front ends as AI services that carry ordinary traffic, such as Google’s and Microsoft’s shared hosts (the collector’s sharedhosts list), CDNs and cloud front ends. These are never suggested.

Shield360’s own hosts are never suggested either: any shield360.ai host, this deployment’s configured address (SHIELD360_PUBLIC_URL, NEXTAUTH_URL, APP_URL, NEXT_PUBLIC_APP_URL), the hosted proxy (SHIELD360_HOSTED_PROXY_URL), the OTLP endpoint (OTEL_EXPORTER_OTLP_ENDPOINT) and the environment’s own ClickHouse host.

Neither are the organisation’s own sites. A domain is skipped when its registered domain (acme.com for app.acme.com) is one that the organisation’s members’ email addresses use, or that its enrolled machines’ user names use. Public mailbox providers such as gmail.com and outlook.com are ignored for this.

Each name is classified by deterministic hints first:

Hint	Confidence
The name of a known AI company or product	High
Visits to paths a model API serves, such as /v1/chat/completions	High
An AI word in the name (gpt, llm, ai)	Medium, or high with an .ai domain or another hint
An .ai domain, and nothing else	None by itself. The name is ambiguous.
An MCP server	High. It is a tool for AI agents by definition.
A very common app, such as Slack or Zoom	Not AI

Names with no hint, or only an .ai domain, are ambiguous. With the analyst on and a model configured, up to 25 of them a day (the most used first) are sent to the model in one call. The model gets only each name and its counts: visits, minutes or calls, and the number of people and computers. It never gets a URL, a path or a query string, and never a person. Names are passed through pseudonymisation too. A name the model judges not AI is not asked about again for 30 days. A name with only an .ai domain is raised only when the model judges it AI with medium or high confidence. With the analyst off, it is not raised at all.

A finding is raised only for what is likely AI. It shows the likely vendor and category (chat, model API, coding, image, audio, video, agent tool or other), the confidence and the hints. It stays open until someone decides it:

• Approve as AI adds the name to this environment’s AI catalogue, with the vendor and category you choose, and resolves the finding. Shadow AI and the AI inventory then count it for this environment. Approved apps are counted by time in use. Approved websites are counted by visits, because the collector does not time them.
• Not AI remembers the name, so it is not suggested again, and dismisses the finding.

Approved additions change what Shield360 reports. They do not yet change what the collector or the browser extension match on the machine, so an approved website is not timed or guarded there. Both decisions are recorded in the audit log.

Unknown AI domain

A website domain first seen this week that looks like an AI service.

Unknown AI app

A desktop application first seen this week that looks like AI.

Unknown MCP server

An MCP server that coding agents started calling this week and that is not one of the common servers. Check what it can reach and who added it.

Savings total

The possible savings at the top of the cost inbox is the sum of the low ends and the sum of the high ends of the open cost findings (new or acknowledged) in the current environment. Dismissed and resolved findings are not counted. If findings are in more than one currency, the total uses the most common currency and leaves the others out.

Permissions

Permission	What it allows
View AI analyst findings	Read the checks’ findings, the last run and the savings total
Triage AI analyst findings	Change a finding’s status, and Run checks now
Triage AI analyst findings and Maintain the AI inventory	Approve or reject an unknown-AI classification
The target screen’s own permission	Save a policy suggestion’s change. Viewing the suggestion needs only View AI analyst findings.

Approving or rejecting a classification needs both permissions because it changes what the AI inventory and shadow AI report. Administrators and compliance officers hold both by default; analysts do not.

Limits

• The checks see monitored machines and connected accounts only. Each finding’s assumptions say what that leaves out.
• Thresholds are fixed. Each check’s evidence links show the underlying figures, so you can judge borderline cases.
• An AI explanation is the model’s reading of the finding. It does not add evidence.
• Policy tuning reads 30 days (90 days for unused approved entries). A policy changed during that time is judged partly on how it behaved before the change.
• The unknown-AI classification judges names. A model’s view of an unfamiliar name can be wrong, so check before you approve. Provider network ranges (ASNs) are not used yet.