FinOps for AI: the cost of models and agents in production

A generative AI pilot usually costs little. A handful of people try an assistant, the provider's bill barely moves and nobody worries. The problem appears when that same assistant goes to production, becomes an agent that chains steps and tool calls, and serves thousands of users. The bill stops being a minor line item and, above all, becomes hard to explain: nobody knows for sure which agent, which use case or which prompt change made it grow.

FinOps already solved a version of this problem for the cloud: make spending visible, allocate it to whoever generates it and decide with data. The FinOps Foundation has been working for some time on extending those practices to AI. This article brings that idea down to operations: what makes up the cost of an agent, how to measure it per business unit of work, how to allocate it and which levers control it.

Why AI does not behave like the rest of the cloud

In traditional cloud, cost closely follows capacity: machines, storage, traffic. If you know how many servers you have and how long they run, you have a good estimate. With language models consumed as a service, cost follows usage, and usage depends on factors that do not show up in an inventory:

• You pay per token, the unit into which the model splits text. Providers publish different prices for input tokens (what you send) and output tokens (what the model generates), and output usually costs several times more per token.
• What gets sent is much more than the user's question. Each call includes system instructions, conversation history, retrieved documents and tool results. That context grows with every step.
• An agent makes several calls per request. It plans, queries a tool, evaluates the result and calls the model again. A single question can trigger ten calls.
• Behavior changes without changing infrastructure. A change to the instructions, a new model or a longer document in the knowledge base can double consumption without anyone touching a server.

That is why the classic controls (rightsizing, reservations, off-hours shutdown) remain useful for the infrastructure around the model, but they are not enough. If you already practice continuous rightsizing, the discipline is the same; what changes is the unit you need to watch.

What makes up the cost of a request

It pays to break down the cost of a request to an agent into its parts, because each one has a different lever.

• Input tokens. System instructions, history, retrieved documents and tool results. In agents this is usually the largest component, because the context is resent with every call.
• Output tokens. What the model generates, including intermediate reasoning when the model bills for it. Lower volume, higher unit price.
• Inference and capacity. If the model is consumed on demand, this cost is built into the per-token price. If you use reserved capacity or your own GPUs, you pay by the hour whether you use it or not, and the risk becomes idle capacity.
• Retrieval. Generating embeddings for documents and queries, storing them in a vector database and querying it. Usually a small cost per request, but fixed and growing with the volume of documents.
• Tool calls. Internal or third-party APIs the agent invokes. Some carry their own cost; all of them add input tokens when their result goes back to the model.
• Retries and loops. Format errors, timeouts or an agent that cannot close the task repeat entire calls. It is the easiest component to overlook and the one that causes spikes.
• Logging and evaluation. Storing traces for audit and evaluating quality, sometimes with another model acting as a judge. It is a necessary cost, but it has to be budgeted.

An illustrative example, with hypothetical prices that do not correspond to any provider: a support agent makes 6 model calls per conversation, averaging 8,000 input tokens and 500 output tokens each. That is 48,000 input tokens and 3,000 output tokens. If input cost USD 3 per million and output USD 15 per million, the conversation would cost about USD 0.14 in input and USD 0.05 in output. Note that input weighs more than three times as much as output, even though its price per token is five times lower.

Unit cost metrics

Total monthly cost answers "how much are we spending", but not "is it worth it". For that you need a unit cost tied to something the business recognizes. We recommend three levels:

1. Cost per request or per call. Useful for engineering: it detects behavior changes after a deployment.
2. Cost per conversation or per task. Useful for the product owner: it reflects the user's full experience.
3. Cost per outcome. Per resolved ticket, per fulfilled request, per processed document. This is the metric you compare with value.

Continuing the example: if a conversation costs USD 0.19 and only 60% of conversations resolve the ticket without human intervention, the cost per resolved ticket is about USD 0.32 (0.19 / 0.6). If a quality improvement raises the resolution rate, the cost per outcome drops even if the cost per conversation stays the same. That is the reason to measure per outcome: it prevents optimizing spend at the expense of quality.

Two caveats. First, "resolved" must be defined before measuring, with an explicit quality criterion. Second, the cost of an agent varies widely between requests, so it is better to report distributions (median and 95th percentile), not just averages. The value compared against this cost is measured with the same rigor as any automation; we cover it in how to measure the value of automation.

Allocation and tagging

Without allocation there is no ownership, and without ownership spending only grows. The challenge with AI is that the provider's bill usually shows consumption by model and by account, not by agent or use case. Allocation has to be built in two layers:

• In the infrastructure: separate accounts, projects or subscriptions by environment and business unit, and mandatory tags (agent, use case, owner, cost center, environment) on every resource that supports them: endpoints, vector databases, trace storage.
• In the application: each model call records who made it, for which agent, in which conversation and how many tokens it consumed. This telemetry is what lets you split the cost of a shared endpoint among several agents.

A common entry point for all calls (a model gateway) makes both easier: it applies tags, measures tokens and enforces limits in one place. It is the same pattern we recommend for governing agents and their tools with MCP.

To consolidate data from several providers, the FinOps Foundation's FOCUS specification (version 1.0, published in 2024) defines a common format for billing data, with columns such as billed cost, effective cost, service, consumed quantity and tags. Normalizing to FOCUS lets you combine the cloud bill, the model provider's bill and other services in a single analysis.

Budgets and limits per agent

In the cloud, a budget that alerts at 80% is usually enough. With agents it is not: a loop can burn a month's budget in hours. We recommend limits at several levels, enforced by the gateway or the agent platform:

• Per request: maximum number of steps, model calls and output tokens. When reached, the agent stops and escalates to a person.
• Per conversation or task: a cap on accumulated cost, with the same behavior.
• Per agent and period: a daily or monthly quota with alerts at 50%, 80% and 100%, and an explicit decision about what happens at the cap (fall back to a cheaper model, switch to recommend-only mode or stop).
• Anomaly detection: compare the cost per request with its baseline and alert on deviations, for example after a prompt or model change.

These limits are also security controls: an agent manipulated into repeating actions is bounded by the same caps. We go deeper in security guardrails for AI agents.

Levers to reduce cost without losing quality

No lever is applied blindly. Every change is validated against the same set of quality evaluations before it reaches production.

Lever	What it does	When it fits	Risk to watch
Model choice and routing	Uses a small model for simple tasks and reserves the large one for complex ones	When there are classification, extraction or summarization tasks	Quality drop on edge cases
Prompt caching	Reuses the repeated prefix (instructions, tool catalog) at a lower price	Long, stable instructions across calls	Frequent prefix changes cancel the savings
Context size	Trims history, summarizes conversations and retrieves fewer but more relevant chunks	Agents with long conversations or many documents	Losing information the model needed
Batch processing	Sends non-urgent jobs in batch mode, which some providers charge less for	Bulk classification, evaluations, overnight reprocessing	Higher latency; not suitable for live interaction
Output limits	Asks for structured, short answers with a token cap	Answers consumed by another system	Truncated answers
Retry control	Validates format before retrying and limits steps per task	Always	Escalating to a person too early
Reserved capacity	Pays for fixed capacity in exchange for guaranteed price or availability	High, stable volume	Idle capacity if demand changes

To understand your provider's billing model, check its pricing page: most distinguish input and output tokens, offer batch and reserved capacity options, and some charge cache reads and writes separately. Terms change often, so it is worth revisiting the estimate every time you change models.

One last criterion: when an agent's cost affects a material budget or a regulated service, the decision on limits and fallback is not just an engineering call. It should be agreed with Finance and, if the service touches customers, with Risk.