- 01
Inventory first — most enterprises have twice the AI systems they think
- 02
Three risk tiers with pre-agreed controls remove per-project negotiation
- 03
Evaluation gates in CI turn governance from policy into engineering
Start with an inventory you can trust
You cannot govern what you cannot enumerate. Every AI system, its owner, the data it touches, the systems it writes to, the model and version behind it, and its risk tier. Most enterprises discover twice as many live AI systems as they expected, mostly embedded in SaaS products they already bought.
Risk tiering that maps to real controls
Three tiers work in practice. Tier one: read-only, internal, human-reviewed output — light controls, fast approval. Tier two: customer-facing or writes to a system of record — evaluation gates, audit trail, human escalation path. Tier three: consequential decisions affecting credit, coverage, employment or clinical care — full model documentation, bias testing, human decision authority and regulator-ready evidence. Tie the tier to the controls in writing so teams know the cost of the path they choose.
Evaluation gates and change control
The operational core: a golden evaluation set per workload, run in CI, with a pass threshold that blocks deploy; documented approval for model, prompt or index changes; and rollback capability. This is where governance stops being policy and becomes engineering — and it is the part most frameworks omit entirely.
Audit trails and evidence
Log the input, the retrieved sources, every tool call and its authorisation, the output, and the human decision if any — retained per your records policy. When a regulator, a client or your own risk committee asks how a specific decision was reached, the answer must be a query, not an investigation.
Mapping EU AI Act and NIST AI RMF onto delivery
The EU AI Act's high-risk obligations — risk management, data governance, technical documentation, logging, human oversight, accuracy and robustness — map cleanly onto tier three above. NIST AI RMF's Govern, Map, Measure and Manage functions map onto inventory, risk tiering, evaluation and change control respectively. Implement the controls once, and produce the frameworks as reports rather than running parallel programs.
Risk tiering is the whole design
A governance framework that treats every AI use case identically will either block the safe ones or wave through the dangerous ones. Tier by consequence: informational and internal-only assistants; workflows that write to systems of record within a bounded, reversible blast radius; and decisions affecting customers, money, employment, credit, health or legal standing. Publish what evidence each tier requires and who can approve it. The goal is that a team building a tier-one assistant can self-certify in days while a tier-three use case gets genuine independent scrutiny — the same review burden for both is the signature of a framework designed for defensibility rather than for delivery.
The artefacts that constitute an AI system of record
Governable AI needs an inventory. For every deployed use case record: purpose and business owner, risk tier, model and version, prompt and configuration version, tool and data access scope, retrieval corpus and its provenance, evaluation set and last results, human oversight design, incident history, and re-certification date. Most enterprises already have a model inventory in another form; extend it rather than create a parallel register that nobody maintains.
Human oversight that means something
'Human in the loop' is frequently a checkbox that describes a person clicking approve on outputs they cannot realistically evaluate. Meaningful oversight requires the reviewer to see the evidence behind the recommendation, to have the authority and time to reject it, to have their rejections recorded and analysed, and to be measured on judgement rather than throughput. If approval rates sit near one hundred percent, the control is decorative and should be redesigned or removed in favour of a stronger automated guardrail.
Mapping to regulation without freezing delivery
Obligations differ by jurisdiction and sector, but the underlying evidence set is remarkably consistent: documented purpose, data provenance and lawful basis, risk assessment, testing including bias and robustness, human oversight design, transparency to affected people, logging and traceability, and post-deployment monitoring. Build the framework around producing that evidence as a by-product of delivery, and specific regulatory mappings become an annotation exercise rather than a re-engineering project.
Monitoring, incidents and re-certification
Governance does not end at approval. Production monitoring should cover quality drift against the golden set, input distribution shift, escalation and override rates, cost, and safety events. Define what constitutes an AI incident, how it is triaged, who can suspend a workflow, and how affected people are notified where required. Re-certify on a fixed cadence and on material change — a new model version, a new data source, a new jurisdiction or a new user population all reset the clock.
Making the framework survive contact with delivery
The best predictor of whether governance works is whether engineers can satisfy it without leaving their tooling. Ship templates, an intake form that takes minutes, automated evidence collection from CI, and a published service-level commitment for review turnaround. Governance that is slower than the delivery cycle gets routed around, and shadow AI is a worse outcome than an imperfect but usable framework.
Data governance underneath AI governance
AI governance fails without data governance beneath it. Retrieval corpora need classification, ownership and lifecycle. Training and tuning data need documented provenance and lawful basis. Derived artefacts — embeddings, transcripts, summaries, evaluation sets — need retention rules of their own, because they frequently contain the same sensitive content as the source and routinely escape existing policy. Extend the existing data catalogue to cover these assets rather than starting a parallel AI-specific inventory nobody will maintain.
Testing standards: bias, robustness and safety
Define what testing each risk tier requires. For high-tier use cases that typically means outcome analysis across protected characteristics where lawful and relevant, adversarial and prompt-injection testing, robustness under degraded or missing inputs, boundary tests confirming the system declines what it should, and a documented review of failure modes and their consequences. Store results with the version they tested, and re-run on material change. Testing that is not versioned and re-run is documentation, not assurance.
Transparency to customers and employees
Decide and publish when people are told they are interacting with an automated system, how they request human review, what data is used, and how to contest an outcome. For employee-facing systems, be equally explicit about monitoring, evaluation and how AI-derived signals feed performance processes — opacity here creates industrial-relations risk that dwarfs the efficiency gain. Consistency matters more than legal minimalism; inconsistent disclosure across channels is itself a finding.
Third-party and supply chain governance
Most enterprise AI risk arrives through vendors. Assess model providers, application vendors and integrators on data handling, training use, sub-processors, jurisdiction, security posture, incident notification, evaluation transparency and exit terms. Maintain an inventory of AI capability embedded inside purchased software — much of it arrives switched on by default and outside any AI governance process. Periodic supplier attestation should include AI feature changes, because vendors ship capability faster than procurement reviews it.
Making governance measurable
Report governance like any other operational function: number of use cases by tier, first-time pass rate, median review turnaround, overdue re-certifications, open incidents, monitoring coverage, and shadow AI discovered. These numbers tell you whether the framework is enabling or obstructing. A rising review turnaround with rising shadow AI is a governance failure even if no incident has occurred, and it is the earliest reliable warning available.
Implementing the framework in ninety days
Month one: publish the risk tiers with concrete examples drawn from use cases already in flight, define the evidence each tier requires, name the approvers, and commit publicly to a review turnaround. Build the intake form so it takes minutes rather than hours, and extend the existing model or system inventory to carry AI attributes instead of standing up a parallel register. Month two: automate evidence collection where the delivery pipeline already produces it — evaluation results, configuration versions, test coverage, dependency and provider lists — and ship templates that pass tier-one governance by default. Run the first tier-three review end to end with a real use case so the process is tested rather than theoretical, and fix what proves unworkable before mandating it. Month three: turn on monitoring requirements for anything in production — quality drift against golden sets, input distribution shift, escalation and override rates, cost and safety events — define what constitutes an AI incident and who can suspend a workflow, and rehearse one incident. Then publish the governance operating metrics: use cases by tier, first-time pass rate, median review turnaround, overdue re-certifications, monitoring coverage and shadow AI discovered. Review those numbers monthly with the same seriousness applied to the use cases themselves. A framework introduced this way is judged by delivery teams as infrastructure rather than obstruction, which is the only condition under which it is actually followed. Frameworks launched as policy documents with a mandate and no tooling produce compliance theatre in the visible programmes and shadow AI everywhere else.
- Inventory first — most enterprises have twice the AI systems they think
- Three risk tiers with pre-agreed controls remove per-project negotiation
- Evaluation gates in CI turn governance from policy into engineering
- Implement controls once; produce EU AI Act and NIST AI RMF evidence as reports
- Tier by consequence so safe use cases self-certify and high-consequence ones get real scrutiny.
- Maintain a single AI inventory covering owner, tier, versions, data scope, evaluation and re-certification date.
- Oversight with near-total approval rates is decorative; require evidence, authority, time and recorded rejections.
- Produce regulatory evidence as a by-product of delivery, and commit to a review turnaround engineers can plan around.
Questions leaders ask us
- What is an AI governance framework?
- A structured set of controls covering AI system inventory, risk tiering, evaluation gates, audit trails, model change control and human oversight — designed so a workload can reach production against pre-agreed criteria rather than a bespoke review each time.
- Does AI governance slow delivery down?
- Poorly designed governance does. Tiered controls agreed in advance speed delivery up, because teams know exactly which evidence a workload needs before it can ship.
- How do EU AI Act and NIST AI RMF fit together?
- EU AI Act high-risk obligations map onto the highest control tier — risk management, data governance, documentation, logging, human oversight and robustness. NIST AI RMF's Govern, Map, Measure and Manage map onto inventory, risk tiering, evaluation and change control. Implement once and report to both.
- What should an AI governance framework contain?
- Risk tiers with defined evidence per tier, a use-case inventory, human oversight design, testing and evaluation standards, logging and traceability, monitoring, incident handling, and re-certification rules.
- Who should own AI governance?
- A cross-functional body with risk, legal, security, data and delivery representation, chaired by someone accountable for outcomes rather than for approvals, with day-to-day evidence collection automated into delivery tooling.
- How do we govern AI without slowing delivery?
- Let low-risk tiers self-certify against published criteria, automate evidence collection from CI, provide templates, and commit to a review turnaround shorter than the delivery cycle.
- When should an AI use case be re-certified?
- On a fixed cadence and on any material change — new model version, new data source, new jurisdiction, new user population, or a sustained shift in monitoring signals.
Sources
- [1] Enterprise AI governance should map to a recognised risk framework rather than a bespoke checklist. AI Risk Management Framework (AI RMF 1.0) — NIST, 2023
- [2] High-risk AI systems in the EU carry statutory obligations for risk management, logging and human oversight. Regulation (EU) 2024/1689 — the EU AI Act — European Union, 2024
- [3] A certifiable AI management system standard now exists for enterprise programmes. ISO/IEC 42001:2023 — AI management systems — ISO/IEC, 2023