- Product squads (Outcomes): Own the KPI, prioritize agents by ROI, run experiments
- AI platform (Capability): Tools, memory, orchestration, evaluation harness
- Safety function (Policy): Guardrails, red-team, audit trail, model risk
- CoE / Chief AI Officer (Governance): Portfolio, funding model, cross-BU standards
The three design choices that decide the outcome
First, does the CoE build workloads or enable business units to build them — and when does it switch? Second, who funds it: central budget, chargeback, or business-unit co-funding? Third, does it own production run or hand off? Ambiguity on any of the three is what turns a CoE into a bottleneck within two quarters.
Roles you actually need
A small core beats a large one: a product owner accountable for the workload portfolio, two or three AI engineers, a data and retrieval engineer, an evaluation and quality lead, and a risk and governance partner with real authority. Architecture and change management are federated into business units, not hoarded centrally.
Build first, enable second
For the first two or three quarters the CoE should build — because reusable assets, patterns and guardrails only emerge from real delivery. After that, shift to enablement: templates, a paved road, the evaluation harness as a service, and reviews. CoEs that start as enablement functions produce standards nobody follows, because the standards were never tested against production.
Intake and prioritisation without politics
Publish the scoring criteria: business value, data readiness, write-path availability, evaluability and risk tier. Score in the open and publish the queue. The CoEs that lose credibility are the ones where prioritisation looks like influence rather than criteria — and once that perception sets in, business units route around them.
Failure modes to watch for
Four common ones: the CoE becomes a demo shop with no production ownership; it owns run for everything and saturates; it writes standards before it has shipped; or it is funded centrally with no business-unit skin in the game, so nobody adopts the output. Each is a design fault, and each is cheaper to prevent than to unwind.
- Decide build-vs-enable, funding model and run ownership in the first ninety days
- Small core team; federate architecture and change management to business units
- Build for two to three quarters before writing standards
- Publish scoring criteria and the queue, or business units will route around you
Questions leaders ask us
- What is an AI center of excellence?
- A central team accountable for the enterprise AI workload portfolio, the reusable platform patterns, the evaluation harness and the governance guardrails — building the first workloads directly, then enabling business units to build against a paved road.
- Should the AI CoE own production run?
- Not for everything. A CoE that runs every workload saturates within two quarters. Own run for shared platform components and hand workload operations to the owning business unit with a supported paved road.
- How large should an AI center of excellence be?
- A small core: a portfolio product owner, two or three AI engineers, a data and retrieval engineer, an evaluation lead, and a governance partner with real authority. Federate architecture and change management rather than centralising them.