- 01
Cost per completed task is the only metric that connects AI spend to business value; cost per token does not.
- 02
Retrieval design usually drives more spend than model choice, because context size multiplies every call.
- 03
Model routing by step — small models for classification and extraction, large models for synthesis — is the highest-yield optimisation in most systems.
Why cost per task, not cost per token
Token pricing is the unit vendors bill in and the wrong unit for managing a business. A completed task — a resolved customer contact, a processed invoice, a prepared reconciliation, an assembled claim packet — may involve one model call or forty, depending on retrieval strategy, tool use, retries and how many turns a conversation takes. Two systems with identical token prices can differ by an order of magnitude in cost per outcome, and the difference is architecture rather than procurement. Define the task boundary explicitly for each workflow, then instrument everything inside it: model calls by model and step, input and output tokens, embedding and reranking calls, document processing, tool invocations, retries, and the fraction of tasks abandoned or escalated after consuming compute. Attribute all of it to the workflow and, where relevant, to the customer or business unit. With that instrumentation the important comparisons become possible: cost per task against the fully loaded human baseline, cost per successful outcome including the tasks that consumed compute and produced nothing, and the trend of both as volume grows. Without it, cost management degenerates into negotiating rates with a provider, which is the smallest available lever and the one most enterprises pull first.
Where the money actually goes
Instrumented systems consistently reveal a distribution that surprises the teams running them. Retrieval context is usually the largest single driver: every additional chunk passed into a prompt is charged on every call in the trajectory, so an over-generous retrieval configuration multiplies across the whole system. Conversation and case history is the second: naively appending full history to each turn makes late turns dramatically more expensive than early ones, and long sessions dominate the distribution. Multi-step trajectories are the third: agentic workflows that reason, call a tool, reason again and call another tool incur a full context charge at each step, so a workflow with six steps is not six times a single call but often considerably more. Then come the invisible categories most models omit — retries after tool failures or malformed outputs, trajectories abandoned midway, evaluation runs in the deployment pipeline, and the development and testing consumption of the team itself, which at scale is not trivial. Model selection matters, but usually less than these structural factors. This is why the productive first step is always measurement: teams that optimise by switching models before instrumenting typically achieve a modest reduction and leave the dominant driver untouched.
The optimisation levers that work
Five levers deliver most of the available reduction. Model routing by step is the highest-yield: classification, extraction, routing decisions and simple transformations rarely need a frontier model, while synthesis, complex reasoning and customer-facing generation may. Routing each step to the smallest model that passes its evaluation gate commonly reduces total spend substantially with no measurable quality loss — but it requires per-step evaluation, which is why teams without an evaluation harness cannot safely take this lever. Retrieval efficiency is second: better chunking, hybrid search and reranking allow far fewer chunks to be passed while improving answer quality, since precision matters more than volume. Context management is third: summarise history rather than appending it, drop content the step does not need, and cache stable system content where the provider supports it. Trajectory design is fourth: collapse steps that do not need separate reasoning, fail fast on inputs the workflow cannot handle, and cap retries. Output discipline is fifth: constrain response length and format, since output tokens are typically priced well above input. Apply them in this order and measure after each, because the interactions are real — a retrieval change alters what a routing decision can safely do.
Modelling cost before you build
Build a cost model during design rather than discovering the economics in production. Estimate the trajectory for a typical task: steps, model per step, expected input tokens including retrieved context and history, expected output tokens, tool calls, and the probability of retries. Multiply by expected volume with a peak factor. Add embedding and indexing costs for the content corpus, including re-indexing frequency, plus document processing, storage, observability and the evaluation runs the pipeline will execute. Then apply a distribution rather than an average, because task cost is typically long-tailed — a minority of complex cases can account for a large share of spend, and an average-based model will be wrong in the direction that matters. Compare against the fully loaded human baseline per task and identify the break-even volume. Finally, model three scenarios: pilot volume, first-year production, and steady state at full rollout, since engineering cost is fixed while consumption scales. The model will be imprecise, and that is acceptable — its purpose is to identify whether the economics are plausible and which variable to watch, not to produce a number for a spreadsheet. Workflows whose modelled cost approaches the human baseline should be redesigned before they are built, not after.
Governing consumption in production
Consumption architectures drift upward quietly, so governance must be continuous and lightweight. Set budget alerts per workflow and per environment, including development, where experimentation costs accumulate without anyone owning them. Report cost per task weekly alongside quality metrics, so the trade-off is visible and an optimisation that saved money by degrading outcomes is caught immediately. Attribute spend to the owning business unit, since unattributed central cost invites both waste and, eventually, an abrupt central cut that damages good workflows along with bad. Set rate limits per workflow to bound the blast radius of a runaway loop, which is a real failure mode in agentic systems and can consume a month's budget in hours. Review the top decile of expensive tasks monthly — they usually reveal either a design flaw or a category of input the workflow should decline. Re-evaluate provider pricing and model options quarterly, because the price-performance frontier moves quickly and a routing decision made a year ago is probably no longer optimal. Keep model selection behind an abstraction so switching is a configuration change validated by the evaluation suite rather than an engineering project.
Presenting AI economics to finance
Finance leaders are comfortable with consumption models — they run cloud infrastructure — but they expect the same discipline applied here. Present four things. Cost per task by workflow against the human baseline, with the retained human cost stated separately. Total spend by category — inference, document processing, platform, storage, observability — with the trend. The deployment cost curve across workflows, demonstrating that shared platform investment reduces the marginal cost of each new deployment. And the optimisation record: what was changed, what it saved, and what quality effect was measured. That last item builds more confidence than any forecast, because it shows the spend is managed rather than merely reported. Avoid presenting AI cost as a single line in a technology budget; the useful framing attributes it to the business processes it serves, where it can be compared to the cost it displaces. Enterprises that establish this reporting early rarely face the arbitrary spending caps that hit programmes whose cost appeared suddenly and without explanation, and they are far better positioned when the sponsor asks — as they eventually will — whether the programme should scale or stop.
Instrumentation: what to capture and where
Capture cost data in the trace rather than reconciling provider invoices afterwards, since invoices arrive late and cannot be attributed to workflows, customers or steps. For every model call record the workflow, the task identifier, the step name, the model and version, input and output token counts, latency, whether it was a retry and why, and the outcome. For every retrieval record the number of chunks, the source corpus and whether a rerank occurred. For every tool call record the tool, duration and success. Roll these up per task and store the total alongside the business outcome, so cost per successful outcome is a query rather than an analysis project. Emit the same data in development and testing environments, tagged separately, because experimentation spend is otherwise invisible until it appears on an invoice. Build three standing views: cost per task by workflow over time, cost distribution within a workflow to expose the expensive tail, and spend by component. Teams that put this in place before their first production release make optimisation decisions from evidence within weeks; teams that add it later spend their first optimisation cycle building the instrumentation they should have had.
- Cost per completed task is the only metric that connects AI spend to business value; cost per token does not.
- Retrieval design usually drives more spend than model choice, because context size multiplies every call.
- Model routing by step — small models for classification and extraction, large models for synthesis — is the highest-yield optimisation in most systems.
- Retries, failed trajectories and abandoned sessions are real cost and are missing from most cost models.
- Instrument before optimising; teams that guess at where spend accumulates are usually wrong.
Questions leaders ask us
- What should we measure first?
- Cost per completed task, broken down by model call, retrieval, tools and retries, attributed to the workflow. Add the share of tasks that consume compute without producing an outcome. Those two numbers reveal where optimisation effort will pay back and are the basis of every subsequent conversation with finance.
- Is switching to a cheaper model the best way to reduce cost?
- It is one lever and rarely the largest. In instrumented systems, retrieval context size, conversation history handling and trajectory length usually drive more spend than model choice. Routing each step to the smallest model that passes its evaluation gate is high-yield, but it depends on having per-step evaluation in place first.
- How do we handle the long tail of expensive tasks?
- Review the most expensive decile monthly. The pattern is usually either a design flaw — unbounded retries, runaway trajectories, unmanaged history — or a category of input the workflow should decline and route to a human immediately. Capping retries and failing fast on unsupported inputs typically removes most of the tail.
- Should AI cost sit in a central budget or with business units?
- Fund the shared platform centrally and attribute run cost to the owning workflow and business unit. Unattributed central spend invites waste and eventually an indiscriminate cut. Attributed spend makes the trade-off visible to the people who own the outcome and can decide whether it is worth paying for.
- How often should we revisit model and provider choices?
- Quarterly. The price-performance frontier moves quickly enough that a routing decision made a year ago is likely suboptimal. Keep model selection behind an abstraction and validate any change with your evaluation suite, so a switch is a configuration change with evidence rather than an engineering project with risk.
Sources
- [1] Cost and unclear value are among the leading reasons enterprises abandon generative AI initiatives after proof of concept. Gartner predictions on generative AI project abandonment — Gartner, 2024
- [2] Retrieval quality, not context volume, determines answer accuracy in grounded generation systems. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv, 2021