- Product squads (Outcomes): Own the KPI, prioritize agents by ROI, run experiments
- AI platform (Capability): Tools, memory, orchestration, evaluation harness
- Safety function (Policy): Guardrails, red-team, audit trail, model risk
- CoE / Chief AI Officer (Governance): Portfolio, funding model, cross-BU standards
The three design choices that decide the outcome
First, does the CoE build workloads or enable business units to build them — and when does it switch? Second, who funds it: central budget, chargeback, or business-unit co-funding? Third, does it own production run or hand off? Ambiguity on any of the three is what turns a CoE into a bottleneck within two quarters.
Roles you actually need
A small core beats a large one: a product owner accountable for the workload portfolio, two or three AI engineers, a data and retrieval engineer, an evaluation and quality lead, and a risk and governance partner with real authority. Architecture and change management are federated into business units, not hoarded centrally.
Build first, enable second
For the first two or three quarters the CoE should build — because reusable assets, patterns and guardrails only emerge from real delivery. After that, shift to enablement: templates, a paved road, the evaluation harness as a service, and reviews. CoEs that start as enablement functions produce standards nobody follows, because the standards were never tested against production.
Intake and prioritisation without politics
Publish the scoring criteria: business value, data readiness, write-path availability, evaluability and risk tier. Score in the open and publish the queue. The CoEs that lose credibility are the ones where prioritisation looks like influence rather than criteria — and once that perception sets in, business units route around them.
Failure modes to watch for
Four common ones: the CoE becomes a demo shop with no production ownership; it owns run for everything and saturates; it writes standards before it has shipped; or it is funded centrally with no business-unit skin in the game, so nobody adopts the output. Each is a design fault, and each is cheaper to prevent than to unwind.
Three models, three failure modes
A centralised centre of excellence delivers consistency and becomes a bottleneck. A fully federated model delivers speed and produces five incompatible platforms and no reusable governance. The hub-and-spoke model — a small central team owning platform, standards and evaluation, with embedded engineers in business units owning workflows — works for most enterprises, provided the centre is explicitly accountable for making the spokes faster rather than for approving their work. Write that accountability into the charter, because a centre measured on control will inevitably drift toward gatekeeping.
What the centre should own and what it must not
The centre owns the reference architecture, the shared tool and integration layer, evaluation infrastructure, model and vendor portfolio, cost attribution and reporting, governance tooling, and the standards library. It must not own business outcomes, workflow backlogs, or the right to approve every change. Business units own the workflow, the metric and the release decision within their tier. This division keeps the centre's work leveraged and stops it becoming a queue that every project waits in.
Staffing the centre realistically
A workable centre for a large enterprise is small: a lead, a handful of platform and integration engineers, one or two evaluation and quality specialists, a governance and risk partner shared with the second line, and a product or enablement role that runs standards, templates and internal education. Overstaffing the centre is a common error — it converts a leverage function into a delivery organisation that competes with the business units it is meant to serve.
Enablement, standards and reuse
The centre's real product is reuse. That means published reference implementations rather than reference diagrams, templates that pass governance by default, a shared tool registry so the same integration is not built five times, an internal catalogue of live use cases with their metrics, and regular clinics where teams bring real problems. Measure the centre on adoption of these assets: how many workflows use the shared tool layer, how many pass governance first time, and how long a new team takes to reach production.
Funding and the cost attribution question
Central funding for platform and standards, business-unit funding for workflows, with transparent chargeback for consumption is the model that survives budget season. Attribution matters more than the funding split: every workflow should carry its inference, platform and support cost so that value cases stay honest and unused capability is visible. Without attribution, the centre absorbs cost the business never sees and becomes the first line cut when scrutiny arrives.
The first ninety days of a new centre
A new centre earns its licence by removing a real obstacle quickly, not by publishing a strategy. Pick the most painful shared problem — usually entitled data access, an integration everyone rebuilds, or an unusable governance intake — and solve it in the first ninety days. Ship a reference implementation, not a standard document. Centres that open with a framework and a governance process are resisted; centres that open by unblocking a team acquire the credibility to introduce standards later.
Standards that teams adopt voluntarily
Adoption follows usefulness. A standard is adopted when following it is the fastest path: a template that passes governance automatically, a shared client library that handles auth and tracing, a golden-set harness that runs in the existing pipeline. Standards distributed as documents with mandates get partial, resentful compliance. Write standards as code and examples wherever the format allows, and measure adoption rather than issuing exceptions.
Relationship with existing functions
The centre overlaps with data platform, security, architecture, risk and the CIO's delivery organisation. Agree boundaries explicitly and early: who owns identity propagation, who owns the data catalogue, who approves architecture, who runs production. Unresolved overlaps surface later as duplicated tooling and stalled releases. In most enterprises the right answer is that the centre owns the AI-specific layer and consumes existing platform services rather than rebuilding them.
Knowledge sharing and internal community
Run a practical community: a catalogue of live use cases with their metrics and owners, an internal demo forum, office hours where teams bring real problems, a searchable record of decisions and failures, and short enablement paths for engineers, product managers and business owners. The failure record matters most — enterprises repeat each other's mistakes for years in the absence of a place where those mistakes are written down honestly.
When to wind the centre down
A successful centre eventually shrinks. Once the platform is stable, standards are embedded in tooling, business units ship independently and governance is routine, the coordinating function can reduce to a small platform and standards team. Plan for that trajectory openly. Centres that measure their success by growth in headcount and scope reliably become the bottleneck they were created to remove.
Charter, metrics and the first year
Write a charter of one page that states the centre's purpose as making business units faster and safer, lists what it owns and explicitly what it does not, names the decisions it can make alone and those it can only advise on, and defines the metrics it will be judged by. Those metrics should measure leverage rather than activity: number of workflows in production using the shared tool layer, median time from intake to production for a new team, first-time governance pass rate, reuse rate of shared components, and the cost curve of successive deployments. Deliberately exclude counts of pilots reviewed, standards published and workshops run, since optimising for those produces a bureaucracy. In the first quarter, solve one painful shared obstacle and ship a reference implementation. In the second, formalise the tool registry and the evaluation harness, and support two business-unit deployments as a partner rather than a gatekeeper. In the third, automate governance evidence collection and publish the use case catalogue with metrics and owners. In the fourth, review the charter honestly: which standards are adopted voluntarily, where teams route around the centre and why, and which responsibilities should now move permanently into business units or existing platform functions. Plan the centre's trajectory toward a smaller platform and standards team from the beginning and say so publicly, because a centre whose implicit goal is its own growth eventually becomes the queue every project waits in — which is precisely the condition it was created to eliminate.
Staffing the centre and where the roles come from
A functioning centre at enterprise scale is small: a lead with delivery credibility, two to four platform engineers owning the shared tool layer, evaluation harness and observability, one or two evaluation and quality specialists, a governance partner embedded from risk rather than seconded occasionally, and a product manager who maintains the use case catalogue and intake. Recruit platform engineers who have operated production systems rather than data scientists who have built notebooks, since the binding constraint is almost always engineering and operations rather than modelling. Embed centre staff into business-unit teams for the duration of a deployment and return them afterwards, which transfers capability and keeps the centre grounded in delivery reality. Resist growth beyond this shape: every additional reviewer adds queue time, and the centre's value is measured by how quickly business units ship, not by how many people the centre employs.
- Decide build-vs-enable, funding model and run ownership in the first ninety days
- Small core team; federate architecture and change management to business units
- Build for two to three quarters before writing standards
- Publish scoring criteria and the queue, or business units will route around you
- Hub and spoke works when the centre is accountable for making business units faster, not for approving their work.
- The centre owns platform, standards, evaluation and cost attribution; business units own workflows, metrics and releases.
- Keep the centre small; overstaffing turns leverage into a competing delivery organisation.
- Measure the centre on reuse and time-to-production, and attribute every workflow's true running cost.
Questions leaders ask us
- What is an AI center of excellence?
- A central team accountable for the enterprise AI workload portfolio, the reusable platform patterns, the evaluation harness and the governance guardrails — building the first workloads directly, then enabling business units to build against a paved road.
- Should the AI CoE own production run?
- Not for everything. A CoE that runs every workload saturates within two quarters. Own run for shared platform components and hand workload operations to the owning business unit with a supported paved road.
- How large should an AI center of excellence be?
- A small core: a portfolio product owner, two or three AI engineers, a data and retrieval engineer, an evaluation lead, and a governance partner with real authority. Federate architecture and change management rather than centralising them.
- Should AI capability be centralised or federated?
- Hub and spoke suits most enterprises: a small centre owning platform, standards and evaluation, with engineers embedded in business units owning the workflows and their metrics.
- How big should an AI centre of excellence be?
- Small — a lead, a few platform engineers, evaluation specialists, a shared governance partner and an enablement role. Growth beyond that usually signals it has started doing business-unit delivery work.
- How do we measure whether the centre is working?
- Adoption of shared assets, first-time governance pass rate, time for a new team to reach production, and reuse of the shared tool layer — not the number of pilots it has reviewed.
- How should AI capability be funded?
- Centrally fund platform and standards, let business units fund workflows, and attribute inference, platform and support cost per workflow so value cases and unused capacity stay visible.