What running a model actually costs

AI/ML engineering · capacity planning · caching strategy · cost modelling

A model feature is cheap to prototype and expensive to operate, and the gap between those is where most AI projects quietly die. The costs are knowable before you build: tokens in, tokens out, how much of it you can avoid paying for twice.

Your actual prompt

Tokenised as you type, with the real GPT pre-tokenizer. The token count below drives the costs — no rule of thumb.

Your workload

Every call that reaches the model.

Counted from the prompt above. Edit it and this follows.

What the model writes back.

Provider price in dollars.

Output is usually the expensive half.

Share of input tokens served from a cached prefix. A stable system prompt is the easiest win here, and it is usually the largest one.

Every month

Cached input is billed at roughly a tenth of the standard rate by most providers, which is why the cache slider moves the total more than any price negotiation will. The output side cannot be cached and is typically five times the input rate per token — so the cheapest optimisation available is usually asking the model to say less, not finding a cheaper model. Nothing you type is sent anywhere.

verso