A server costs the same whether ten people use it or a thousand, up to a point. LLM API calls cost money every time. When an AI feature gets popular, the bill grows with it. When a bug sends a loop of requests, the bill grows faster. OWASP's 2025 list for LLM applications includes "unbounded consumption" as a risk for this reason.
We plan LLM costs before launch and put limits in the code.
Estimate per request, then per month
For each feature we measure the average input and output tokens per request with realistic data. Retrieval-based features send large contexts, so input tokens usually dominate. Multiply by the provider's price and the expected number of requests per user per month. That gives a cost per active user, which the business can compare with what that user pays.
If the numbers don't work at the expected scale, we find out on paper instead of on the invoice.
Send less text
The cheapest token is one you don't send.
- Retrieve fewer, better chunks. Reranking lets us pass three strong passages instead of ten weak ones.
- Trim the prompt. System prompts grow over time as people add rules. We review them and remove instructions the tests show make no difference.
- Summarise long conversations. After a few turns, we replace older messages with a short summary instead of resending the full history.
Use the smallest model that passes
Different tasks need different models. Classifying a support ticket or extracting fields from an invoice rarely needs the largest model. We route each task to the smallest model that passes its evaluation set and reserve the large model for tasks that need it. Our test suite makes this safe, because we can prove the smaller model's answers hold up.
Cache what repeats
Many requests repeat. Product descriptions, FAQ answers and document summaries can be generated once and stored. Providers also offer prompt caching for repeated context such as a long system prompt, which reduces the price of those tokens.
Put limits in the code
- Per-user rate limits, so one account can't generate thousands of requests
- A maximum output length on every call
- Timeouts, and a cap on retries
- A daily spending limit at the provider, plus an alert well before it
- Loop guards on agent-style features, so a tool-calling loop stops after a set number of steps
Report it with the rest of the bill
We tag LLM usage by feature and show it next to hosting costs in the monthly report. When a feature's cost per user drifts up, someone sees it within weeks. Usually the cause is a prompt that grew or a retrieval setting that now returns too much, and the fix is small.
