AI providers
Overview
Section titled “Overview”Shield360 calls a model for some of its own work: scoring traces, answering questions in Rex, and investigating alerts. Settings → Integrations → AI providers (/settings/ai-providers) puts those choices side by side, so you can see and change them in one place.
Each feature still keeps its own provider, model and key. This page shows the same forms the features use, so a change made here is the same as a change made on the feature’s own page. The API key is never typed in here. You pick a secret you stored in the Vault.
| Section | Used by | Also set on |
|---|---|---|
| Evaluations and content checks | LLM-judge evaluations of traces, the Office add-in’s content checks, the model check in prompt-injection scoring, and generated and multi-turn attacks in agent tests | Developer tools → Testing → Evaluations, Configuration tab |
| Rex assistant | Answers in the Rex chat | Rex’s chat settings |
| AI analyst | AI analyst investigations | The analyst runs on the model Rex uses in the environment. Open analyst settings opens Settings → Integrations → AI analyst, where its budgets and caps are |
Openground has no setting here, because it picks providers for each comparison.
Who can change it
Section titled “Who can change it”| Section | Permission to read | Permission to change |
|---|---|---|
| Evaluations and content checks | evaluation.read (View evaluations): owners, administrators, analysts and members | evaluation.configure (Configure evaluations): owners and administrators |
| AI analyst settings | analyst.read | analyst.settings (Change AI analyst settings): owners and administrators. The settings apply to the whole organisation |
| Rex model | Anyone who can use Rex in the environment. The key is never shown | analyst.settings, because the AI analyst runs on the same model and key |
Every save of these settings is recorded in the audit log.
The models are set per environment, that is, per project database. Switch project or environment in the header to change another environment’s models.
Set the evaluation model
Section titled “Set the evaluation model”Add the key to the Vault, or use Create new next to the key field.
Under Evaluations and content checks, choose a Provider and a Model. You can type a model name that is not in the list. Then choose the secret under API Key (Vault).
Turn on Enable auto evaluation and give a Cron schedule to score new traces automatically, and set a Sample rate (%) to score only part of them. Without auto evaluation, traces are scored when someone selects Run Evaluation on a trace, or through the SDK.
When content is not stored and spans are evaluated as they arrive, three caps also apply: Evaluations per minute (as spans arrive), Evaluations per day (as spans arrive) and Daily evaluation cost cap (USD, 0 for none). Spans over the per-minute cap wait their turn. Spans over a daily cap are not evaluated.
Select Create Config the first time, or Save Changes after that.
Which evaluators run, such as Hallucination, Toxicity or Prompt Injection, is set in the Evaluators tab of Developer tools → Testing → Evaluations. See Evaluators. For the full set of fields, see Evaluation configuration.
Set the Rex model
Section titled “Set the Rex model”Add it to the Vault first.
Under Rex assistant, choose the provider and model, and the secret under API Key (from Vault).
Select Save Configuration. Shield360 sends the model a test request first and saves it only if it answers. A retired model, a wrong key or an unreachable gateway is refused with the provider’s own error, rather than discovered by the next person who asks Rex something.
In the cloud, an environment with no Rex model of its own uses the model Shield360 provides. The AI analyst follows Rex: whatever you set here is the model the analyst uses in that environment. The Rex section is not shown in the public demo.