- 01
AI in production is an operated system; the run function is permanent and is the most consistently under-budgeted line in the programme.
- 02
Evaluation specialists and conversation designers are the scarce roles, and both can be developed from experienced operations staff.
- 03
Ratios scale with workflow count and change rate, not with transaction volume.
The run function nobody budgets for
Enterprises budget generously for building AI systems and sparsely for operating them, which is why so many deployments perform best in their first quarter and decline thereafter. A production AI workflow requires continuous attention that has no equivalent in traditional software. Content changes and retrieval quality shifts. Customer language moves with products, campaigns and seasons, so intent models drift. Model providers release new versions and deprecate old ones. Integrated systems change their responses. Every one of these can degrade behaviour without producing an error, which means the degradation is invisible unless somebody is measuring it. The run function exists to notice and correct. Its work is concrete: reviewing escalations and failures, maintaining and growing evaluation sets, investigating drift, triaging incidents, releasing changes safely, tuning cost, remediating knowledge, and reporting quality and economics to the business owner. None of this is optional, and none of it is done by the project team that disbanded at go-live. Enterprises that fund the run function from the first deployment sustain their gains and deploy faster afterwards, because the same team absorbs each new workflow. Those that treat operations as a residual activity absorbed by existing staff find, usually within two quarters, that the deployment's numbers have quietly reverted.
The roles and what each one actually does
Six roles cover a production estate. The evaluation specialist owns golden test sets, reviews failures, quantifies quality and drift, and gates releases — this is the role most often missing and the one whose absence most reliably produces decay. The conversation designer owns dialogue flow, escalation behaviour, tone and the handoff contract, working from real transcripts rather than from a design tool in isolation. The platform engineer owns the shared tool layer, retrieval infrastructure, deployment pipeline, observability and the enforcement of authority limits. The knowledge owner maintains the content the system retrieves, working the backlog that deployments generate — a role frequently assumed to be part-time and rarely is. The operations owner runs the exception and escalation queue, triages incidents, and is the interface to the business. The business owner accepts the outcome metric and holds the authority to pause. In smaller estates one person may hold two of these; what does not work is holding all of them, or assigning them as fractional additions to full-time roles elsewhere, which is the arrangement that produces a run function on paper and no run function in practice.
Ratios and how the team scales
Staffing scales with the number of distinct workflows and the rate of change, not with transaction volume — a workflow handling ten million interactions needs roughly the same maintenance attention as one handling one million, because the work is analytical rather than transactional. As a planning heuristic, a first production workflow typically needs the equivalent of two to three full-time roles across evaluation, platform and operations, plus part of a knowledge owner. Each subsequent workflow on the same platform adds substantially less, often a fraction of a role, provided the platform layers are genuinely shared. An estate of eight to twelve workflows commonly settles at a team of six to ten covering all six roles with some specialisation. Two factors break the heuristic upward: high change rate in the underlying business, which multiplies evaluation and content work, and regulatory tiering, which adds evidence and re-certification obligations. One factor breaks it downward: genuine platform reuse, which is the whole argument for building shared layers. Track the ratio deliberately — team size per production workflow, trending over time — because it is the clearest available measure of whether the platform investment is delivering leverage or whether each deployment is quietly a bespoke project.
Building these people from your existing operation
The scarce roles are hard to hire and unusually well suited to internal development. Evaluation specialists need domain judgement, comfort with data and the patience to categorise failures — a profile common among senior quality analysts and team leaders in contact centers and shared services, who already spend their days assessing whether an interaction was handled correctly. Conversation designers need to understand how customers actually phrase problems and where conversations break, which experienced agents know better than anyone in the organisation. Knowledge owners need to know why a policy exists and where the exceptions live, which sits with long-tenured operations staff. Build explicit paths: identify candidates, pair them with the delivery team during a deployment, give them a defined transition rather than an additional duty, and pay for the new role rather than the old one. The benefits compound — domain knowledge stays inside the automation, the operations floor sees a visible progression route at exactly the moment automation makes people anxious about their futures, and retention among the experienced staff you most need to keep improves. Enterprises that hire all these roles externally pay more, wait longer and lose the institutional knowledge that makes evaluation sets good.
What automation does to the remaining human work
Automation removes the simple, short, repetitive contacts first, which changes the nature of everything left. The residual queue is longer, more complex, more emotionally demanding and less predictable. Handle time rises, and it should. If targets, staffing models, quality frameworks and pay bands are unchanged, three things follow reliably: measured performance appears to worsen, coaching becomes punitive, and attrition among experienced staff rises at precisely the moment their judgement is most valuable. Reset deliberately. Rebase handle-time and occupancy expectations against the new mix. Rebuild the quality framework around judgement, empathy and resolution rather than adherence and speed. Protect training and coaching time, which the residual complexity demands more of, not less. Revisit pay bands where the role has genuinely become more skilled, and say publicly that it has. Adjust recruitment profiles toward problem-solving and communication over throughput. And staff for variance rather than average, since the residual queue is spikier once the smoothing effect of high-volume simple contacts is removed. Operations that make these adjustments preserve their experienced staff through the transition; those that do not discover that automation improved their cost per contact and degraded their capacity to handle the contacts that matter most.
The operating rhythm
Set the cadence before go-live and hold it. Daily during the first two weeks of any deployment: review the worst outcomes, fix what is fixable that day, and track the exception rate. Weekly thereafter: escalation reasons and their trend, quality against golden sets, knowledge backlog and its burn-down, cost per task, incidents and their causes, and the change queue. Monthly: business outcome metrics against the baseline with the business owner, model and provider review, and cost optimisation opportunities. Quarterly: evaluation set refresh, re-certification for tiered workflows, retirement of unused configuration, and a review of the team ratio against the workflow count. Annually: operating model review against the failure signals. The single most important property of this rhythm is that it visibly changes things — when agents report a bad answer and see it corrected within the week, reporting continues and the system improves; when nothing changes, reporting stops within a fortnight and the organisation loses its best source of signal. Assign each forum an owner with authority to act rather than only to escalate, and keep the number of forums small enough that people attend them.
Onboarding a new workflow into the run function
Treat the transition from build to run as a defined handover with acceptance criteria rather than a date. Before the run team accepts a workflow it should receive: an evaluation set with current scores and the release gate defined; runbooks for the known failure modes with named escalation paths; monitoring dashboards and alert thresholds already live; the authority configuration documented with its change process; a knowledge ownership assignment for the content the workflow retrieves; cost per task instrumented and a current baseline; and at least two weeks of shadow operation where the run team handles the queue while the build team is still available. Refusing acceptance until these exist is the only reliable way to prevent the pattern where a delivery team declares success, disperses, and operations inherits a system they cannot diagnose. Record the handover in the same way an infrastructure service transition would be recorded, and review the first month's incidents against the runbooks to close the gaps while the build team's knowledge is still retrievable. The same acceptance checklist then applies to every subsequent workflow, which is what makes the run function scale rather than absorb each deployment as a bespoke burden with its own undocumented behaviour, tribal diagnosis and single point of human dependency.
- AI in production is an operated system; the run function is permanent and is the most consistently under-budgeted line in the programme.
- Evaluation specialists and conversation designers are the scarce roles, and both can be developed from experienced operations staff.
- Ratios scale with workflow count and change rate, not with transaction volume.
- The human queue after automation is harder work; reset targets, coaching and pay bands accordingly or attrition will rise.
- A weekly review loop that visibly fixes what it finds is what keeps quality improving; without it, deployments decay quietly.
Questions leaders ask us
- How many people does it take to run one production AI workflow?
- As a planning figure, the equivalent of two to three full-time roles across evaluation, platform engineering and operations for the first workflow, plus part of a knowledge owner. Subsequent workflows on genuinely shared platform layers add far less. Staffing scales with workflow count and change rate rather than with transaction volume.
- Can existing contact center staff move into these roles?
- Yes, and it is usually the better route for evaluation, conversation design and knowledge ownership. Senior quality analysts, team leaders and long-tenured agents already hold the domain judgement these roles need. Pair them with the delivery team during a deployment, make the transition explicit rather than additive, and pay for the new role.
- What happens to handle time after automation?
- It rises, because the simple contacts are gone and the residual work is harder. Rebase targets against the new mix, rebuild the quality framework around judgement rather than speed, and protect coaching time. Holding old targets through an automation rollout is the most reliable way to trigger attrition among the experienced staff you need most.
- Who should own AI operations organisationally?
- Operationally it belongs with the function that owns the business outcome, supported by a shared platform team that owns the tool layer, evaluation harness and observability across workflows. Governance sits with risk. Ownership placed entirely inside technology tends to optimise for uptime rather than for the outcome the workflow exists to produce.
- What is the earliest sign that the run function is under-resourced?
- A growing knowledge backlog that nobody is burning down, evaluation sets that have not changed since launch, and escalation reviews that identify the same issues repeatedly. All three appear well before the outcome metrics decline, which is why the weekly review should track them explicitly.
Sources
- [1] Generative AI assistance produces the largest measured productivity gains for less-experienced workers, changing the shape of the residual human workload. Generative AI at Work — National Bureau of Economic Research, 2023
- [2] Organisations scaling AI cite talent and operating capability gaps, not model access, as the primary constraint. The state of AI: Global survey — McKinsey & Company, 2024