Dataset registry

Overview

Govern → Datasets is the register of the data assets behind your AI systems in the current environment: training and fine-tuning sets, evaluation and test sets, knowledge bases used for retrieval (RAG) and prompt libraries. For each dataset it records:

• Purpose and owner: what it is for, and the member accountable for it.
• Source and provenance: the source type (internal system, vendor, public source, generated by users, synthetic), the origin system, URL or vendor, how it was collected, and the period it covers.
• License and legal basis: the license or terms of use, the GDPR legal basis, and a short note on consent or where the assessment is kept.
• Classification: the data classes and kinds of sensitive data it holds, in the same vocabulary as the data dictionary and the classification catalog. Personal data is derived from the classes and kinds.
• Quality and bias: the EU AI Act Art. 10(2) and 10(3) examination: relevance, representativeness, errors, completeness, bias examination, bias mitigation and data gaps, the data preparation (labeling, cleaning, enrichment), and the day it was examined. Whether special categories of personal data are processed to detect bias (Art. 10(5)).
• Retention and reference: how long it is kept, its version, the SHA-256 of the released dataset, and a location reference.
• Links: the AI systems it feeds (from the AI inventory), and the data dictionary tables and lineage sources it corresponds to.
• Consumer data and privacy: whether it holds third-party data about consumers, the protected-class and medical attributes it holds, the privacy basis or notice that covers it, and whether it holds nonpublic personal information. See Consumer data and privacy.
• NAIC data elements: the NAIC Exhibit D data element types it holds, how each AI system uses them, where they come from and the lines of business they serve.
• Data dictionary: its fields, with a description, type, source, NAIC data element type and a sensitive flag. See Data dictionary.

Note

The registry records what you declare about each dataset, never the data itself: no row, document, prompt or sample is uploaded or read. Keep notes short and leave out personal details. Credential-shaped strings are removed from free text before it is saved, and a location loses any user name, password, query string and fragment.

Register a dataset

Open the registry

Open Govern → Datasets and select Register a dataset, or select Register on one of the unregistered data sources.

Describe it

Fill in the purpose, owner, provenance, license, legal basis, classes and the quality and bias examination. Link the AI systems it feeds.

Save

The dataset is saved as Proposed, by you. The Governance column lists anything it still lacks.

Approve

Another member with the Manage the data dictionary permission opens it and selects Approve. Nobody approves a dataset they proposed, and a dataset that lacks anything cannot be approved.

Status

Status	Meaning
Proposed	Registered or changed, waiting for a second person’s approval.
Approved	A member other than the proposer approved it, with nothing missing.
Retired	No longer used. Retired datasets leave the lineage and the controls check, and are kept for the record. Reopen proposes one again.

Changing anything but the description of an approved dataset proposes it again, by you, and clears the approval: an approval covers the dataset as it was approved. A proposed dataset that was never approved can be deleted; retire any other.

What a governed dataset has

Item	Recorded when
Owner	An owner is set and is still an active member of the organization (not an Auditor, not deactivated by your identity provider)
Provenance	The origin and the collection method
License	The license or terms of use
Legal basis	A basis is chosen; “not applicable” only when the dataset holds no personal data
Classification	At least one data class
Quality and bias examination	Relevance, representativeness, errors, completeness, bias examination and bias mitigation, and the day it was examined
AI systems	At least one AI system linked

Consumer data and privacy

The Consumer data and privacy (NAIC) section of the form records:

Field	Records
Holds third-party data about consumers	External consumer data: credit-based insurance scores, telematics, geo-demographics and other data about consumers obtained from a third party.
Protected-class and medical attributes	Age, gender, ethnicity/race, national origin, religion, disability, marital status, sexual orientation, medical, genetic information.
Privacy basis or notice	The privacy notice or basis that covers this use of the data, by name and version. A reference, not the notice.
Nonpublic personal information (NPI)	Present, not present, or not recorded.

With the retention and the legal basis, these answer item 3i of the NAIC AIS Program checklist (Exhibit B: data privacy and protection of consumer data used in AI systems) for each dataset. Changing them on an approved dataset proposes it again, like any other governed field.

NAIC data elements

NAIC data elements on a dataset’s detail records which of the 25 data element types of Exhibit D of the NAIC AI Systems Evaluation Supplement the dataset holds. For each one:

Field	Records
Data element type	One of the 25 types, in the Supplement’s wording. Other: Non-Traditional Data Elements takes your own name for the element.
AI system	One AI system the dataset feeds, or every system it feeds. Record the same type once per system when systems use it differently.
The AI model uses it as	Operational input, training input, testing input, or several.
How the AI model uses this data and how the output is used within operations	Short free text, column (2) of Exhibit D.
Internal data source	Column (3).
Third-party data source / vendor name	Column (4). Link the vendor to its vendor: key when the AI inventory knows it; the name is filled from the inventory.
Lines of business	The lines it serves. Leave it empty when it serves every line.

Every element needs an internal source, a third-party vendor or both. Select Add data element, fill it in, select Done, then Save: saving replaces the dataset’s list. Changing data elements records a step on the dataset (Changed: dataElements) and does not propose an approved dataset again.

What the data elements flag

For each AI system, Shield360 combines its datasets and their data elements into a data profile with these flags:

Flag	Set when
External consumer data	A dataset says it holds third-party data about consumers, or a consumer data element (scores, telematics, geo-demographics, income, medical and the like) comes from a third party
Protected class	A dataset holds a protected-class attribute, or an element or dictionary field is Age, Gender, Ethnicity/Race
Medical	A dataset holds medical or genetic attributes, or an element or dictionary field is Medical
NPI	A dataset records nonpublic personal information
Third-party training data	A third-party element is a training or testing input

Testing and risk tiering read these flags. Retired datasets are left out.

Data dictionary

Data dictionary on a dataset’s detail lists its fields: name, description, type, source, the NAIC data element type it is (optional) and whether it is sensitive. Never a value of a field.

• Add field adds one; change a field’s NAIC element or sensitive flag in place; Save replaces the dictionary.
• Import CSV reads a file with a header row. Columns, in any order and case: name (or field, column), description, type (or data type), source, NAIC element (a type’s key, its Ref 1–25, or its NAIC wording) and sensitive (yes or no). Only name is required. Merge by field name replaces fields with the same name and adds the rest; Replace the dictionary swaps it. One bad row refuses the whole import, naming the row.
• Export CSV writes the same columns, so an export imports back. Export JSON is also available.

A dictionary holds at most 500 fields.

name,description,type,source,NAIC element,sensitive
dob,Date of birth,date,Application,2,yes
garaging_zip,Garaging ZIP code,string,Policy admin,"Geocoding (including address, city, county, state, ZIP code, lat/long, MSA/CSA, etc.)",no
credit_tier,Credit-based insurance score tier,string,Bureau,insurance_risk_score,yes

A cell that contains a comma is quoted, as a spreadsheet does when it saves CSV.

NAIC Exhibit D

Govern → Datasets → NAIC Exhibit D shows Exhibit D (AI Model Data) of the NAIC AI Systems Evaluation Supplement for one line of business, filled from the data elements recorded on this environment’s datasets that are not retired:

Column	Filled from
Ref and (1) Type of Data Element Used in AI Model	The 25 types, in the Supplement’s order and wording.
(2) How the AI model uses this data	Each element’s uses, how the model uses it and how the output is used.
(3) Internal Data Source	The elements’ internal sources.
(4) Third Party Data Source / Vendor Name	The elements’ vendors.

Above the grid, AI systems or AI models relying on this data lists every system an answered row names. Choose the line of business at the top; All lines of business combines them. An element that names no line of business appears under every line. A type no dataset records is left blank, as the Supplement asks.

Export CSV writes the form’s header (line of business, the AI systems relying on the data, the date) followed by the 25 rows in the Supplement’s columns, plus the datasets and systems behind each row. Export JSON has the same content. Each export is recorded in the audit log with the format, the line of business and the number of rows answered.

Note

Shield360 fills Exhibit D from what your team records. It does not inspect datasets to find data elements, and the export is a starting point for your submission, not a filing.

Datasets in the lineage

Every dataset that is not retired appears in data lineage as a source, with a line to each AI system it feeds, labeled with how it feeds it: trains, fine-tunes, evaluates, grounds or prompts. These lines are declared, not observed: they are drawn dashed, recorded by Dataset registry (declared), and count no events. A dataset carries its classes and kinds, so a line from a dataset holding personal data is amber like any other line that carried classified data.

Show in lineage on a dataset opens the lineage filtered to it.

Unregistered data sources

Below the list, Shield360 suggests data sources it saw feeding agents and apps in the last 30 days that no registered dataset corresponds to:

Suggestion	Seen in
Dictionary table read by agents	The data gateway’s access records
Connected system reached by agents	MCP gateway tool calls, including Shield360’s documents and contexts
Vector store seen in traces	Spans naming a vector database (Pinecone, Chroma, Qdrant, Weaviate, Milvus, pgvector and similar)

Register opens the form with the source linked. A suggestion disappears once a registered dataset names it as a dictionary table or a lineage source. Suggestions come only from records Shield360 already keeps: training data that never passes through a path Shield360 observes is not suggested.

Controls and evidence

The check Datasets feeding AI systems are owned, sourced, licensed, classified, examined for quality and bias, and approved (datasets.governed) on Controls looks at every dataset that is not retired and feeds an in-scope AI system: an application or agent in use, a tool rated high risk, or a system classified high-risk under the EU AI Act.

• It is not applicable while no registered dataset feeds an in-scope system.
• It is a gap while any of those datasets lacks an item above or is not approved. The reason names each dataset and what it lacks.
• It is met otherwise.

It backs:

Framework	Controls
EU AI Act	Art. 10 (data and data governance); through the technical documentation pack, Annex IV 2(d)
ISO/IEC 42001	A.7.2 data for AI systems, A.7.3 acquiring data, A.7.4 data quality, A.7.5 data provenance, A.7.6 preparing data
NIST AI RMF	MAP-2.3 (data collection and selection documented), MAP-4.1 (legal risks of components, including third-party data), MAP-4.2 (internal risk controls for components), MEASURE-2.1 (test sets documented)

In the EU AI Act technical documentation pack, item 2(d) Data requirements and datasheets is observed when at least one approved dataset is linked to the system and every dataset linked to it is approved and complete. The pack lists those datasets.

Evidence packages include dataset-register.json: each dataset’s purpose, status, approval, which governed items are recorded, its classes, kinds, legal basis, version, SHA-256 and linked systems. Free text (origin, notes, quality findings, preparation, retention, location) is left out.

Export

Export CSV and Export JSON download the datasets the filters match, with every field. Each export is recorded in the audit log with the format, the filters and the number of datasets.

Access

Action	Permission
See the registry and the suggestions	View data protection events
See data elements, the data dictionary, data profiles and Exhibit D	View data protection events
Register, change, approve, retire, reopen and delete; change data elements and the data dictionary, and import a dictionary	Manage the data dictionary
Export the registry, a data dictionary or Exhibit D	Export data protection evidence

Administrators and Compliance officers have all three. Every change is recorded in the audit log with the dataset’s id, the statuses and the names of the fields changed, never their values.

Datasets belong to one environment: switching project or environment shows that environment’s registry. An organization export includes every dataset with its links and steps; a person export includes datasets whose free text names the person, and erasure clears that text and keeps the dataset.

API

# Datasets that still lack something
curl -H "Authorization: Bearer <api key>" "https://<your-shield360>/api/data-gateway/datasets?incomplete=1"

# Register a dataset
curl -X POST -H "Authorization: Bearer <api key>" -H "Content-Type: application/json" \
  -d '{"name": "Support tickets 2025", "purpose": "rag", "systemKeys": ["otel:support-bot"], "dataClasses": ["customer", "pii"], "license": "Internal use only"}' \
  "https://<your-shield360>/api/data-gateway/datasets"

# Approve it (another member)
curl -X POST -H "Authorization: Bearer <api key>" -H "Content-Type: application/json" \
  -d '{"action": "approve"}' "https://<your-shield360>/api/data-gateway/datasets/<id>"

# Export as CSV
curl -H "Authorization: Bearer <api key>" -o datasets.csv "https://<your-shield360>/api/data-gateway/datasets/export?format=csv"

# Tag a dataset's NAIC data elements (replaces the list)
curl -X PUT -H "Authorization: Bearer <api key>" -H "Content-Type: application/json" \
  -d '{"elements": [{"element": "insurance_risk_score", "uses": ["operational"], "usage": "Rating factor", "outputUse": "Sets the premium tier", "vendorName": "Credit bureau", "linesOfBusiness": ["Personal Auto"]}]}' \
  "https://<your-shield360>/api/data-gateway/datasets/<id>/elements"

# Import a data dictionary, merging by field name
curl -X POST -H "Authorization: Bearer <api key>" -H "Content-Type: application/json" \
  -d '{"csv": "name,description,type\ndob,Date of birth,date\n", "mode": "merge"}' \
  "https://<your-shield360>/api/data-gateway/datasets/<id>/fields"

# One AI system's data profile
curl -H "Authorization: Bearer <api key>" "https://<your-shield360>/api/data-gateway/datasets/profile?asset=model:pricing-glm-v3"

# Exhibit D for a line of business, as CSV
curl -H "Authorization: Bearer <api key>" -o exhibit-d.csv "https://<your-shield360>/api/data-gateway/datasets/exhibit-d/export?format=csv&line=Personal%20Auto"

Route	Method	Does
/api/data-gateway/datasets	GET	The datasets, filtered by status, purpose, system, owner (unassigned), incomplete=1 and q
/api/data-gateway/datasets	POST	Registers a dataset, proposed by the caller
/api/data-gateway/datasets/<id>	GET, PATCH, DELETE	Reads, changes or deletes one dataset
/api/data-gateway/datasets/<id>	POST	{"action": "approve" | "retire" | "reopen"}
/api/data-gateway/datasets/export	GET	format=csv or json, with the same filters
/api/data-gateway/datasets/suggestions	GET	Unregistered data sources over period (default 30d)
/api/data-gateway/datasets/<id>/elements	GET, PUT	A dataset’s NAIC data elements; PUT {"elements": [...]} replaces them
/api/data-gateway/datasets/<id>/fields	GET, PUT	A dataset’s data dictionary; PUT {"fields": [...]} replaces it
/api/data-gateway/datasets/<id>/fields	POST	{"csv": "...", "mode": "merge" | "replace"} imports a CSV
/api/data-gateway/datasets/<id>/fields/export	GET	format=csv or json
/api/data-gateway/datasets/profile	GET	asset=<AI inventory key>: one system’s data profile; without it, every system’s
/api/data-gateway/datasets/exhibit-d	GET	The Exhibit D grid; line=<line of business>
/api/data-gateway/datasets/exhibit-d/export	GET	format=csv or json, with line

Malformed JSON, an unknown field value, an unknown step, an unknown import mode or export format, a malformed asset key or a line of business over 80 characters is refused with 400. The environment is always the caller’s own; a dataset of another environment answers 404.

Limits

• The registry covers the datasets people register. Shield360 does not see training or fine-tuning that happens outside the paths it observes, and does not inspect dataset contents: the quality and bias examination is what your team recorded.
• Suggestions come from the data gateway, the MCP gateway and traces; a vector store is recognized by the database name its spans report.
• The checksum is recorded as given; Shield360 does not recompute it.
• NAIC data elements, the data dictionary and the consumer data flags are what your team records. Shield360 does not detect data element types in dataset contents.

Next steps

Data lineage

See registered datasets next to the data flows Shield360 observed.

EU AI Act

Art. 10 and the Annex IV technical documentation pack.