Model testing and oversight

Overview

Insurance regulators ask insurers to show how each high-risk AI model is tested and overseen. The NAIC AI Risk Evaluation Supplement (Exhibit C) asks, per model, about testing and validation for drift, accuracy, unfair trade practices, unfair discrimination and performance degradation (item 10), proxy analysis of input and derived variables (10a), the last test with its results and corrective actions (11), the review for compliance with unfair trade practices and unfair claims settlement laws (13), and regulatory actions taken over the model (14). Its AIS program checklist (Exhibit B 3j) asks how consumer complaints from AI systems are identified, tracked and addressed.

Govern → Model testing keeps these records for each AI system in the AI inventory, by the system’s inventory key:

• Validation & testing: each test, why it was run, who ran it and whether they are independent of development, what it covered, what it found, the decision, corrective actions and the documents behind it. Somebody else signs it off.
• Drift monitoring: the population stability index of a score or variable against its baseline.
• Material changes: changes to the model, its data or its use, each of which sets revalidation due.
• Compliance reviews, regulatory actions and consumer complaints.

Each system shows when it was last validated and when it is next due.

Caution

Shield360 records and exports what your organisation tested and decided. It does not certify a model, determine whether a practice is unfairly discriminatory, or give a legal opinion. The four-fifths rule is a screening heuristic, not a legal test.

Prerequisites

• To read the records, you need the View the compliance overview permission (compliance.read). Owners, administrators, the Compliance officer, Analyst and Viewer roles, and Auditors have it.
• To add, change, submit and sign off records, you need the Manage AI impact assessments permission (compliance.impact.manage). Owners, administrators and the Compliance officer role have it. Auditors never do.
• Records are kept per environment (database configuration), as the rest of governance is.

Which systems are listed

The list shows every system with a record, and every in-use AI application, agent and high-risk AI tool from the inventory (the same systems impact assessments cover). A system without a signed-off validation shows as Never validated. To start a record for any other model or tool, choose it from Another system from the inventory.

Record a test or validation

Open the system

Select its row in Govern → Model testing, then Validation & testing → Record a test.

Describe the test

Record the date, the type (pre-deployment, ongoing monitoring, after a material change, or periodic revalidation), the validation method (internal, external or both: the source of the reference data, as the NAIC defines it), who performed it and whether they are independent of development, and the scope: accuracy, model drift, performance degradation, unfair trade practices, unfair discrimination, proxy analysis or other.

Record what it found and what was decided

Add the methodology and a results summary, whether material differences were found and what contributed to them, and the decision: no action, change the model, change the data, change the controls, or retire the system. Link the findings that hold the corrective actions, and list the documents (validation report, model card) by name and where they are kept. Shield360 stores the reference, never the document.

Submit for sign-off

Submit for sign-off is available once the date, performer, scope, results and decision are recorded, plus the proxy analysis when it is in scope and the material differences when you found any.

Sign off

Someone else who may manage these records adds any notes and selects Sign off. Whoever recorded or submitted the test cannot sign it off. A signed-off test is final; a later test is a new record.

Unfair discrimination test

When Unfair discrimination is in the scope, open a CSV file of decisions with one row per decision. Choose the group column, the outcome column and which outcome values are favourable, and optionally a model score column. For each group Shield360 shows:

Measure	How it is worked out
Selection rate	Favourable outcomes divided by decisions
Adverse impact ratio	The group’s selection rate divided by the most favoured group’s (the highest rate). Below 0.8 is flagged under the four-fifths rule of thumb
Standardized mean difference	The group’s mean score minus the most favoured group’s, divided by the two groups’ pooled standard deviation

Groups with fewer than 30 decisions are marked as a small sample.

Note

The file is read in your browser. Only the per-group counts and score statistics are saved with the test, and the server works the measures out again from them. No row of the file, and no consumer data, is stored.

Proxy analysis

When Proxy analysis is in the scope, record the input variables, derived variables or outputs examined for proxy relationships, the method, what was found and the action taken (Exhibit C 10a).

Drift monitoring

Under Drift monitoring → Add, give the date, the score or variable, and two distributions over the same bins: either the baseline and current counts per bin, or a CSV file with a baseline column and a current column, which Shield360 bins on the baseline’s deciles. The population stability index is

PSI = Σ (current_i − baseline_i) × ln(current_i / baseline_i)

over the bins’ proportions. Below the watch threshold (0.1 by default) the level is Stable, from it Watch, and from the act threshold (0.25 by default) Act. Only bin proportions are saved.

Material changes and the revalidation schedule

Record a material change when the model, its data or its use changes. Revalidation falls due 30 days after the change (set Revalidate within (days) of a material change, or a date on the change itself). The change is resolved when a validation of type After a material change, dated on or after it, is signed off.

A system’s next due date is its last signed-off validation plus its revalidation cadence (12 months by default; set it for the environment on the list, or for one system on its page), or the revalidate-by date of its earliest open material change, whichever is sooner. A system shows Due soon within 30 days of that date and Overdue after it.

Compliance reviews, regulatory actions and complaints

Register	What it records
Compliance reviews	Reviewer and function, date, laws considered (unfair trade practices, unfair claims settlement, unfair discrimination, other), conclusion, notes and linked findings
Regulatory actions	Informal agreement, voluntary compliance plan, administrative complaint, ongoing third-party monitoring, cease and desist, remediation, restitution, fine or penalty, investigation, consent order, or other; jurisdiction, opened and closed dates, status, the regulator’s reference and a summary
Consumer complaints	Your complaint system’s reference, received date, category, outcome, remediation and resolved date

A complaint reference may hold only letters, digits and . _ : / # -, so a name or an email address cannot be recorded in its place. To bring in many, select Import CSV with the columns external_ref, received_on (yyyy-mm-dd), category, and optionally outcome, remediation and resolved_on; a reference already recorded for the system is skipped.

Export

Every register has CSV and JSON downloads, for one system from its page. CSV has one row per record; JSON carries every record in full, including the discrimination test’s per-group measures and the drift bins. Each download is recorded in the audit log.

Audit

Every change is recorded in the audit log with the route and outcome. Submitting, returning and signing off a test are recorded with the system, the statuses, the decision and the material changes resolved, never the text recorded. A refused self sign-off is recorded too.

API

Every route acts on your current environment; a configuration id in the request is ignored. <register> is validations, monitoring, material-changes, compliance-reviews, regulatory-actions or complaints.

Method	Path	Parameters
GET	/api/governance/model-testing	Optional asset (an inventory asset key) for everything recorded about one system. One row per system otherwise.
GET	/api/governance/model-testing/<register>	Optional asset. The records, newest first.
POST	/api/governance/model-testing/<register>	JSON body: assetKey and the register’s fields. Drift: expected and actual (counts or proportions per bin), optional labels, warnAt, actAt. Complaints import: assetKey and rows (up to 500).
GET	/api/governance/model-testing/<register>/<id>	One record; for a validation also its next steps and whether you may sign it off.
PATCH	/api/governance/model-testing/<register>/<id>	JSON body: the fields to change, or for a validation action: submit, return or sign_off, with optional notes.
DELETE	/api/governance/model-testing/<register>/<id>	A draft validation, an unresolved material change, or any other record made in error.
GET	/api/governance/model-testing/<register>/export	format: csv (default) or json; optional asset.
PUT	/api/governance/model-testing/cadence	JSON body: months (1–60) and optional revalidateDays (1–365) for the default, with assetKey for one system, or assetKey and clear: true to return it to the default.

Malformed JSON, an invalid id, asset key or field, or a finding from another environment returns 400. Signing off a test you recorded or submitted returns 403. A record in another environment, or an unknown register, returns 404. A step the test cannot take from its status, changing a test that is not a draft, or a complaint reference already recorded for the system returns 409.

curl -X POST -H "Cookie: <session>" -H "Content-Type: application/json" \
  -d '{"assetKey":"model:pricing-glm-v3","measuredOn":"2026-10-01","variable":"model score","expected":[120,340,410,130],"actual":[100,300,450,150]}' \
  "https://<your-shield360>/api/governance/model-testing/monitoring"

Next steps

AI inventory

The systems these records belong to.

Findings

The corrective actions a test or review links to.

Impact assessments

Each system’s impact on people and society.