If Ollama feels slow on a CPU, first find out which part is slow. Loading the model, processing a long prompt and producing output are different jobs. A change that helps one may do little for the others.
Slow is a symptom, not a diagnosis
Start with one short, repeatable request and no other active workloads. Note the model, its size or quantisation, the available memory and whether the model was already loaded. Then repeat the same request. Comparing a cold first call with a warm second call helps reveal whether startup accounts for most of the delay.
This guide does not promise a token rate for your hardware. It gives you a way to measure your own workload and choose a change based on evidence. A smaller model that answers the required question correctly is often more useful than a larger model that is too slow for the interaction.
Read the timing fields before tuning
Ollama's generate API reports timing fields including load_duration, prompt_eval_duration and eval_duration, along with token counts. The durations are expressed in nanoseconds. These fields help separate loading, input processing and output generation instead of treating the entire response as one mysterious delay.
For a readable single-response experiment, use the documented generate endpoint with stream set to false. Keep the prompt and model constant. Your application's time to first visible token is a separate measurement; a non-streamed aggregate response does not directly give you that interaction metric.
Generation rate can be calculated as eval_count divided by eval_duration in seconds. For example, an invented trace with 100 generated tokens and an eval_duration of 5,000,000,000 nanoseconds works out to 20 tokens per second. That is an arithmetic example, not a result measured on a particular CPU.
- Large load_duration: investigate cold loading and model residency.
- Large prompt_eval_duration: investigate input length and prompt processing.
- Large eval_duration: investigate generated length, model choice and resource pressure.
- Long application wait despite reasonable API timings: investigate your queue, retrieval or UI before blaming inference.
Check whether the model is actually on CPU
Run ollama ps and inspect the processor information for the loaded model. Do not assume that installing a GPU driver means your current request is using the GPU, or that a machine without a suitable GPU can achieve the same throughput as a benchmark recorded on one.
Also watch memory and the machine's other work. If the model and its context push the system into heavy swapping, the experience can degrade severely. Close unnecessary workloads for the baseline test, then reintroduce normal workloads so your final estimate reflects the environment where the assistant will run.
Choose a model that fits the machine with room for the operating system and application. Compare quality on your own representative questions. Reducing the model size is a useful performance experiment; it is not a successful change if it produces unacceptable answers on the work you need it to do.
Fix long input before shrinking everything else
A chat application may send a short new question together with a large conversation history, retrieved documents and tool results. The user sees a five-word question while the model sees thousands of tokens. Log input size and retrieval size using safe metadata, not private document text.
Remove irrelevant material and deduplicate repeated passages. Keep the instructions, evidence and recent conversation needed for the task. If you summarise older history, preserve important decisions and references rather than replacing them with a vague paragraph that makes later answers less reliable.
Ollama's FAQ explains that memory requirements increase with context and parallel requests. Set the context deliberately for your workload instead of reserving a very large window because the model permits it. Test that legitimate inputs still fit; silently losing required history is not a performance improvement.
Warm loading, output length and concurrency
The keep_alive setting controls how long a model remains loaded. It can reduce repeated loading delays for intermittent requests, but it does not turn a CPU into faster generation hardware. Keeping a model resident also consumes memory, so choose a value that fits your workload rather than treating indefinite residency as free.
Check whether the task needs a long answer. A classification, extraction or routing step may need a small structured result instead of several paragraphs. Shortening the required output can reduce completion time without changing the underlying model. Verify that the shorter response still contains the fields your application needs.
If the model supports thinking controls, consult that model's documented options before changing them. Do not send unsupported settings copied from another model. For CPU deployments, test one request at a time before raising concurrency; more simultaneous work can increase memory pressure and make every user wait longer.
Keep a before-and-after worksheet
Record the model, input tokens, output tokens, loading time, input-processing time, generation time and total application time. Change one variable at a time. Compare a short prompt with your normal prompt, cold with warm loading and one active request with your planned concurrency.
Accept a change only when both speed and answer quality meet your requirements. For an interactive assistant, measure the time before useful text appears as well as the time until completion. Streaming can improve perceived responsiveness without making the full generation finish faster.
If the CPU cannot meet the target after these checks, choose a smaller adequate model, a different workload design or suitable remote or accelerated inference. The useful outcome is an honest operating limit. A local model that handles a narrow task reliably is better than a supposedly private assistant that users abandon because every interaction stalls.