Map costs before cutting them
Break the bill into four buckets: input tokens, output tokens, retries, and dead-end tool loops. Most teams over-index on picking a cheaper model when 80% of the spend is in input context they could trim.
Seven tactics that actually move the bill
- Prompt caching — stable system-prompt prefix; 50-90% savings on repeated calls.
- Context windowing — RAG over the last N turns instead of the full thread.
- Tool result truncation — cap tool outputs at 2-4KB; summarize the rest server-side.
- Cheap-model first, escalate — route to a small model; escalate only when confidence is low.
- Stop tokens and max_tokens — bound output explicitly; never trust the model to know when to stop.
- Retry budget — cap retries per request to 2; surface failures instead of looping.
- Per-user cost caps — hard ceiling per user per month; protects unit economics from outliers.
Measure success per dollar
A model that's 5× cheaper but needs three tries to succeed is not cheaper. Track success rate per dollar spent, not dollar per call.
FAQ
- Where do AI costs actually go?
- Usually 60-70% on input tokens (long context, no caching), 20% on output, 10% on retries and dead-end agent loops. Cut input first.
- Is a cheaper model always the answer?
- No. A model that's 5× cheaper but needs three retries to get it right is more expensive. Measure success rate per dollar, not dollar per call.
- Does caching actually help in production?
- Yes. Prompt caching on Anthropic and OpenAI saves 50-90% on repeated system prompts. Free money if your prompts have a stable prefix.