The cost of an AI job that actually gets done
A lower token bill can hide a more expensive operation. Enterprise AI economics starts with accepted work, including the time and infrastructure needed to finish it.
Editorial image for The cost of an AI job that actually gets doneVIBECIO Editorial Desk ·
An AI service can become cheaper to call while making the department that uses it more expensive to run. The model produces a draft for pennies; an employee spends twelve minutes checking it, repairing missing information, and discovering that the account has already changed. Procurement sees a favorable unit price. Operations sees another queue.
The useful unit is a completed piece of work that the organization accepts. That could be an invoice reconciled, a support case resolved, or a software change accepted into a release. Each needs an explicit finish line. A generated response is an intermediate event unless producing that response is the entire job.
Put a denominator under the investment case
For a defined period and workload, divide the full operating cost by the number of outcomes that meet the agreed acceptance criteria. Include model and tool charges, allocated infrastructure, engineering and support, human review, and rework. Count failed attempts in the cost even when they contribute nothing to completed output. Report initial development investment separately, or amortize it transparently over a stated horizon.
Consider an illustrative comparison. Two systems each finish 100 comparable cases at the required quality and deadline. One incurs $600 in model and infrastructure costs and $900 in review and correction. The other incurs $900 and $300 respectively. Their completed-case costs are $15 and $12. The more expensive technical component supports the less expensive process. These are hypothetical figures; the point is to make the accounting boundary visible.
That measure still needs context. A system could lower its average by accepting only easy cases and leaving specialists with the hardest work. Report completion coverage, exceptions, and the case mix alongside cost. Compare against the actual previous process. Freed time creates usable capacity; count financial savings when an expense is reduced or avoided.
The serving system changes the answer
NVIDIA’s inference metrics distinguish the delay before the first token, the pace of subsequent generation, total request latency, and aggregate throughput. Its documentation also explains that higher concurrency can increase system throughput while worsening an individual user’s latency. A large tokens-per-second number does not tell a service owner how long a particular customer waits.
Latency acquires a business meaning at the workflow boundary. A background reconciliation process can tolerate a different delay from a tool assisting an employee during a live call. The same model, prompt, and hardware can therefore have different economic value in two applications. Queueing, retrieval, tool calls, retries, and review all consume part of the time available to complete the job.
The DistServe research presented at USENIX OSDI examines separating prefill and decoding to increase the rate of requests served within first-token and per-token latency objectives. This is serving goodput: useful capacity under specified timing constraints. It is a better infrastructure question than raw throughput, while still leaving answer correctness and business acceptance to the application.
For an enterprise, the implication is to benchmark the workload it intends to operate: realistic input lengths, output lengths, arrival patterns, and concurrency. Capacity purchased for a peak can sit underused between peaks. An apparently cheap shared endpoint may struggle with the deadline that makes an interaction useful. Both effects belong in the investment case.
Acceptance needs an independent standard
A team cannot establish completion merely by asking the generating model whether it succeeded. Some outcomes have mechanical checks: a ledger balances, an API returns the expected state, or an approved test suite passes. Others require informed review. Google Cloud’s guidance on evaluating a judge model compares model-based scores with human ratings to establish whether the evaluator fits the use case.
The practical consequence is to budget for evaluation as part of delivery. Keep representative difficult cases, examine disagreements between automated and human judgments, and monitor whether accepted results later require correction. Otherwise the cost calculation rewards whatever the evaluator happens to overlook.
Buy the next improvement in the process
This changes the next investment decision. A better retrieval system might remove more review work than a model discount. A slower but more capable model may be appropriate for difficult exceptions, while a smaller model handles a bounded routine step. Sometimes the best intervention is correcting the source data that every model receives.
The purchase decision becomes concrete: which change increases accepted work at the required quality and speed, and what does that increment cost? Token prices remain useful inputs. The budget owner finally gets a measure that corresponds to the work being funded.
Sources