What an LLM feature actually costs

Model pricing is quoted per million tokens, which is a unit chosen to sound small. Three dollars per million is a real price and it is also completely uninformative until you know how many tokens one use of your feature takes.

Almost every unpleasant surprise in an AI bill comes from the same two places: the request is bigger than anyone estimated, and a conversation resends everything that came before it.

The unit you actually pay in #

Two numbers per request, priced separately, because output costs several times more than input:

Your system prompt is part of the input on every single call. A 400-token system prompt on a million requests is 400 million input tokens that produced no user-visible value, and at mid-tier pricing that is over a thousand dollars a month to say the same thing repeatedly.

The arithmetic #

One formula, and it is worth writing down rather than eyeballing:

// Prices are per million tokens, so the divisor is 1e6.
const requestCost = ({ inputTokens, outputTokens, inputPrice, outputPrice }) =>
  (inputTokens / 1e6) * inputPrice + (outputTokens / 1e6) * outputPrice;

// A 1,500-token prompt, a 400-token answer, at $3 / $15 per million.
requestCost({
  inputTokens: 1500,
  outputTokens: 400,
  inputPrice: 3,
  outputPrice: 15
});
// 0.0105 — just over one cent per request.

One cent is easy to dismiss. Multiply it by the volume you are actually planning for:

These are round-number price points spanning the range providers charge, not quotes from any specific provider. Substitute the rates on your own invoice.

A table rather than a chart, deliberately: the values span four orders of magnitude, and on any bar chart with a single honest zero-based axis the $8 bar is one pixel next to the $52,500 one. A chart that hides five of its nine numbers is worse than the table it replaced.

The spread across price points is 65×. That is the single largest lever you have, and it is worth knowing which parts of your feature genuinely need the expensive model before assuming all of them do.

The curve nobody budgets for #

Here is the one that catches people. A chat feature is stateless underneath: the model has no memory, so to continue a conversation you resend the entire conversation. Every turn.

Turn one sends a system prompt and one message. Turn twenty sends the system prompt and nineteen previous exchanges. The input per turn grows linearly, so the total grows with the square of the conversation length.

Output is a straight line. Input is a curve, and by turn twenty it is fourteen times larger than output. Compare that conversation against a hypothetical one where history was not resent:

Nine times the input tokens, three times the total cost, for the same twenty replies. And that is a short conversation. At fifty turns the multiple is far worse, because the growth is quadratic and your intuition is linear.

Watch out

This is also how you hit a context limit without sending anything large. Nobody sent a big message. The conversation simply accumulated, and turn forty was rejected for a length nobody typed.

Three things actually help, in descending order of effect:

1. Cap the history. Send the last N turns, not all of them. Most conversations do not need turn three by the time you are on turn thirty.
2. Summarise the middle. Replace older turns with a short running summary. The cost of producing that summary is repaid within a few turns.
3. Use prompt caching. Providers that offer it charge a fraction of the input rate for a prefix they have seen before, which is exactly the shape of this problem, since the system prompt and early history are identical every turn. It changes the constant, not the curve, so combine it with capping rather than relying on it alone.

Prompt size matters more than you think #

Because input is billed per token and there is a lot of it, the size of what you send dominates the bill long before the choice of model does at the cheap end:

A 500-token prompt costs $7.50 per thousand requests. The same feature with a 32,000-token prompt costs $102, a fourteen-fold increase for output the user may not notice is any better.

Retrieval systems are where this goes wrong quietly. Stuffing twenty retrieved documents into a prompt because the context window allows it turns a 500-token request into a 32,000-token one, and the chart above shows what that does. Retrieve fewer, better chunks.

Before you ship #

Do these three things and you will not be surprised:

• Count real tokens, not estimated words. Use your provider's tokenizer on ten actual payloads, including your worst case. The ratio for your data will not be the ratio for prose.
• Log input and output tokens per request from day one. Every provider returns them in the response. Without that log you cannot tell an expensive feature from a popular one.
• Set a spend limit at the provider. A retry loop that runs unattended overnight is a real failure mode and the limit is the only thing that stops it.

Then run the arithmetic at ten times your expected volume. If that number is survivable, ship it. If it is not, the problem is in the prompt size or the model choice, and both are much cheaper to change now than after launch.