AI cost calculators usually begin and end with tokens. Production invoices do not.
A model call sits inside retrieval, storage, queues, observability, evaluations, moderation, retries, and support. Agent workflows may call several models and tools before producing one outcome.
Measure the Whole Run
Cost per generation hides retries and orchestration. Track cost per trace, successful task, active user, and product feature. Include embeddings, reranking, judge calls, and background summaries.
Latency creates cost too. Slow responses increase abandonment and may trigger impatient duplicate requests. Engineering time spent debugging untraceable outputs is another operational expense.
Watch the Multipliers
Small design choices compound:
- replaying full history on every turn;
- retrieving more documents than the model needs;
- running every evaluator on every request;
- using the strongest model for classification;
- and retrying malformed output without a limit.
Sampling and task-specific routing can reduce spend without removing visibility.
Connect Cost to Value
A feature costing ten cents per run may be excellent if it completes a valuable workflow. A one-cent feature is expensive if users retry five times and abandon.
Optimize cost per useful outcome, not cost per token. That requires joining model spend with evaluation and product data.
The goal is not to make AI as cheap as possible. It is to understand where cost creates value, where it compensates for a weak design, and which changes improve both quality and economics.