AI Workflow Cost Control: Cut Token Spend Without Losing Quality
Most AI workflows are not expensive because the model is expensive — they are expensive because nobody watches where the tokens go. This guide shows how to meter every step, cache what repeats, route each task to the cheapest capable model, and put hard caps on retries, so your monthly AI bill drops while results stay sharp.
Where the money actually leaks
When a team complains that "AI is too expensive," the bill almost never traces back to one costly call. It traces back to four structural leaks that appear in nearly every workflow we have audited:
- Context bloat. Every step of a chained workflow re-sends the full conversation history, the retrieved documents, and the system prompt. A workflow that started at 3,000 input tokens per call is quietly at 25,000 by step eight.
- Retries with no ceiling. A flaky step that fails 10% of the time will happily retry until it succeeds — sometimes 20 or 30 times — because nobody set a maximum.
- One model for everything. Teams use their strongest (and priciest) model for simple classification, extraction, and summarization steps that a small model could handle in half the time for a tenth of the cost.
- Dead runs nobody reads. Drafts, duplicate evaluations, and experimental branches generate output that is never used, yet each one bills the same as the run that shipped.
The fix is not to use AI less. It is to make every step accountable for its cost, the same way you would make a server team accountable for cloud spend. Pair this mindset with proper circuit breakers, retries and fallbacks so cost limits never turn into silent data loss.
Step 1: Meter every step, not just the total
You cannot control what you cannot see. Before changing anything, tag every workflow step with a name and log three numbers per run: input tokens, output tokens, and wall-clock latency. A flat log line is enough:
After two weeks of logs, one or two steps will own 70–80% of the bill. That is where you start. Teams that skip this measurement step almost always optimize the wrong thing — usually the flashy generation step, when the real leak is the retrieval step re-sending documents.
Set budgets, not just alerts
Alerts are passive; budgets are active. Give each workflow a per-run token budget (for example, 40,000 total tokens) and a daily team budget. When a run crosses its budget, the workflow should degrade gracefully — finish with a smaller model, skip the optional enrichment step, or queue for human review — instead of failing mid-run after spending everything. This is the same evaluation discipline described in our testing and evaluation loops guide: measure first, decide by the numbers.
Step 2: Stop paying for the same tokens twice
Three caching and compression habits remove a surprising share of waste:
- Prompt caching for stable context. System prompts, retrieved reference documents, and few-shot examples rarely change between runs. Providers now discount cached input tokens heavily — often 50–90% off. Mark the stable prefix of your prompt as cacheable and stop re-billing it at full price on every call.
- Rolling summaries instead of full history. For multi-step workflows, replace the raw transcript with a running summary once the context passes a threshold (say, 10,000 tokens). Keep the last step verbatim for accuracy, compress everything older. In practice this cuts chain inputs by 40–60% with no measurable quality drop.
- Cache by intent, not just by text. Exact-match caching misses near-duplicates: "summarize this invoice" on invoice A vs. invoice B. Hash the intent plus the document ID, and reuse the schema-detection or classification result when the same document type recurs. For small teams this is a cheap win — see our small teams workflow guide for lightweight implementations.
Step 3: Route each task to the cheapest capable model
Not every step needs your flagship model. Sort your steps into three tiers and match them to models deliberately:
| Tier | Typical steps | Model strategy | Typical saving |
|---|---|---|---|
| Mechanical | Classification, extraction, format validation, keyword matching | Smallest model that passes your evals | 70–90% |
| Standard | Summarization, drafting, translation, data mapping | Mid-size model, verified per step | 40–60% |
| Critical | Final judgment, code review, customer-facing answers | Best model, with review | Baseline |
The key word is capable: downgrade one step at a time and re-run your evaluation set before moving on. Keep a per-step record of which model won, so future workflow changes inherit the routing instead of resetting to "everything on the flagship model."
Escalate instead of defaulting up
An even cheaper pattern is escalation: run the small model first, and only escalate to the large model when the small one's confidence is low or its output fails validation. For classification and extraction steps, escalation paths routinely deliver flagship-level accuracy at near-small-model prices.
Step 4: Put a hard ceiling on retries and loops
Retries are the silent budget killer. Adopt these four rules:
- Cap attempts per step. Three attempts is a sane default; five is the absolute maximum. Beyond that you are burning money on a step that has a systemic problem.
- Cap tokens per retry loop. Even with few attempts, a step with a 100k-token context that retries three times costs 300k tokens. Give the loop its own budget and stop when it is spent.
- Retry smarter, not harder. Retry only transient errors (rate limits, timeouts). On content errors — validation failures, schema mismatches — fix the prompt or the input instead of re-sending the same request.
- Log and review every escalation. Each capped run is a signal: a step that hits its ceiling weekly has a prompt, data, or routing problem worth a proper fix, not another retry.
These ceilings pair naturally with resilience patterns like circuit breakers, which we cover in depth in our resilience guide.
Step 5: Run a weekly cost review
Cost control decays without a rhythm. Once a week, spend 15 minutes on three questions:
- Which step grew its per-run token cost the most since last week — and why?
- Which runs produced output nobody used? (Kill or gate those steps.)
- Is there a step still on the flagship model that has passed evals on a smaller one for two consecutive weeks? (Downgrade it.)
Track one number over time — cost per successful run — and make it visible to the team. When this number trends down while quality metrics hold steady, you know the program is working. Small teams can run this review with a spreadsheet; the discipline matters more than the tooling, as our small teams guide explains.
Quick reference: the 5-step checklist
- Log input tokens, output tokens, and latency per step for two weeks.
- Enable prompt caching for stable context; compress chains with rolling summaries.
- Tier every step (mechanical / standard / critical) and route to the cheapest model that passes evals.
- Cap retries at three attempts and give every loop its own token budget.
- Review cost per successful run weekly; downgrade, cache, or kill anything that drifts.
Related reading
Limits and notes
Saving percentages above are observed ranges from typical document-processing and content workflows, not guarantees — your numbers depend on context size, provider pricing, and step design. Never cut cost by silently dropping quality checks or audit logs: a cheap run that produces an unverified customer-facing answer is more expensive than a well-metered one. And keep a reserve in your budget for the flagship model on genuinely critical steps; the goal is to spend deliberately, not to spend zero.