What Is Inference?
Inference is the runtime phase of an AI model – every time it answers a question. It stands in contrast to the training phase, when the model was built. For companies, this is the cost that counts, since you almost always use a finished model and pay per run. The cost is driven by model size, token volume, and response-time requirements.
Inference is the runtime phase of an AI model – the actual moment it receives a question and produces an answer. It stands in contrast to the training phase, when the model was once built by learning patterns from data. Training is the manufacturing; inference is the use.
The difference matters, because it explains where the cost actually lands for most companies. The word inference means roughly “to draw a conclusion” – the model draws a conclusion from what it has learned and what you’re asking about. Every answer it gives is an inference.
Training cost versus operating cost
The two phases differ dramatically in both character and cost.
Training is a heavy one-time effort. Building a foundation model from scratch requires enormous amounts of data and computing power, and is done by a handful of large players. It’s extremely costly, but it happens once.
Inference is the ongoing operation. Every time someone asks a question, the model runs, and that costs a small amount each time. The amount is low per run but multiplies with every use.
For a typical company, the consequence is clear: you rarely train your own foundation model, but use a finished one instead. So in practice you pay almost exclusively for inference – to run the model, not to build it. It’s the inference cost that shows up in the budget and is worth understanding.
What drives the inference cost
Three factors decide what each run costs, and they can be influenced.
- Model size. A larger, more powerful model costs more per run than a smaller one. Choosing the largest model available, when a smaller one would do, is a common and expensive mistake.
- Token count. The cost is counted per token, meaning per piece of text sent in and generated. Long questions, lots of accompanying history, and detailed answers drive up the cost.
- Response time. Requirements for very fast answers can demand more resources and thereby increase the cost.
| Cost driver | Effect on the bill |
|---|---|
| Larger model | Higher cost per run |
| More tokens per question | Cost rises proportionally |
| Requirement for fast answers | Can require more resources |
The point is that the inference cost isn’t a fixed number but something shaped by how the solution is built. Choosing the right model for the task and keeping down the amount of text sent is often what has the biggest impact.
A concrete worked example
Picture an internal AI assistant that answers questions from staff. Every question means an inference: a certain number of tokens is sent in – the question plus any history and accompanying material – and a certain number is generated as the answer. The cost per question is small, but with hundreds of questions a day, the sum becomes noticeable over a month.
This is where it becomes clear why optimization pays off. If you send along the entire previous conversation with every question, the token count grows quickly, and the cost with it. If you also choose an unnecessarily large model, you pay a premium for capacity you don’t use. By choosing a well-sized model and only sending along the material the question actually needs, the same service can become significantly cheaper to run – without the answers getting worse. It’s rarely one big saving, but many small ones that add up per token.
Where inference runs
Finally: inference can run in two ways. Either via an API in the cloud, where you call a provider’s model and pay per use, or on your own hardware that you run yourself. The cloud is the easiest way to get started and scales with usage, while your own hardware can pay off at large, steady volumes or when data can’t leave your systems.
The choice between cloud and your own hardware is rarely obvious, but a trade-off. The cloud requires no investment in equipment and suits most companies starting out, but the cost follows usage and can get high at very large volumes. Your own hardware means an upfront investment and operations to manage, but can become cheaper per run when volume is large and steady, and gives full control over where the data is processed. It’s the same kind of trade-off as between renting and owning.
Understanding inference means understanding where the money goes in an AI solution – and therefore where it’s possible to save. Want to think through what running a model would cost in your case? Read more about our AI services or get in touch with a description of the expected usage.
Frequently asked questions
What's the difference between training and inference?
Training is when the model is built and learns patterns from data – a heavy one-time effort. Inference is when the finished model is used to answer a question, over and over in production. Training is done once by whoever builds the model; inference happens every time someone asks it a question. Most companies only ever touch inference.
Why do companies pay for almost nothing but inference?
Because most companies use a finished model instead of training their own. Training foundation models is extremely costly and done by a handful of players. You instead pay to run the finished model – for every question it answers. That's why it's the inference cost, not the training cost, that shows up in the budget for a typical AI solution.
What affects the inference cost?
Mainly three things. Model size: a larger model costs more per run. Token volume: the more text that's sent in and generated, the higher the cost. And response-time requirements: faster answers can require more resources. Choosing the right model size for the task, instead of the largest one available, is often what affects the cost the most.
Where does inference run?
Either via an API in the cloud, where you call a provider's model and pay per use, or on your own hardware that you run yourself. The cloud is the easiest way to get started and scales with usage. Your own hardware can pay off at large, steady volumes, or when data can't leave your systems for sensitivity reasons. The choice depends on volume, cost, and requirements.
Can the inference cost be reduced?
Often, yes. Common approaches are choosing a smaller model where it's enough, shortening what's sent in, and avoiding sending along more history than necessary with every question. Saving and reusing answers to recurring questions can also cut the number of runs. Since the cost is counted per token, such optimizations have a direct effect on the bill.