How to Reduce LLM Latency (7 Levers, Real Numbers)

Most latency guides list the same abstract levers and never show you a real number. This one does: on identical coding tasks where 10 of 13 models scored a perfect 9/9, the fastest was not the smartest or the cheapest — it was a mid-priced, non-reasoning dense model that finished in 2.9 seconds while heavy reasoners took nearly seven times longer. Here is what actually moves the clock, ranked, with our first-party data.

Latency is the tax users feel on every request. But most advice about reducing it stops at a checklist — stream it, cache it, use a smaller model — without ever telling you how much each move is worth. We ran a first-party benchmark of 13 models on 9 identical coding tasks and timed every one. The results reframe the whole problem: the biggest latency lever is not a code change at all. It is which model you call, and whether you let it think.

TTFT vs total latency

Before you optimize anything, separate two numbers that people constantly conflate.

Time to First Token (TTFT) is the gap between sending a request and the first output token arriving. It is what users experience as the wait, and it is dominated by prompt length — the model has to prefill your entire input before it can emit a single token. Total latency is TTFT plus the time to generate every remaining token. Because decoding is sequential and memory-bandwidth-bound — each token needs a full forward pass — total latency scales roughly linearly with output length.

This distinction decides which lever helps you. Streaming attacks TTFT and perception. Shorter output and a faster model attack total wall-clock time. If you optimize the wrong one, you will make a demo feel snappier while the job still takes just as long.

Our benchmark: reasoning drives the clock

We gave 13 models the same 9 tasks — generate code, then run it against hidden tests — and recorded quality, cost per 1,000 tasks, average seconds per task, and reasoning-token spend. Quality converged: 10 of 13 scored a perfect 9/9. Latency did not. It spread from 2.9s to 19.2s, a 6-7x gap on identical work. Full write-up in our LLM coding cost benchmark.

The pattern that jumped out: every model that spent zero reasoning tokens was faster than every model that reasoned. The fastest overall, Mistral Medium 3.5, is a non-reasoning dense model that still hit 9/9 — faster than the cheapest 9/9 model and 6-7x faster than the heavy reasoners.

Read the two spreads together. Quality is roughly flat, but latency swings 6-7x and cost swings 88x for the same 9/9 result. The heavy reasoners were both the slowest and, in the cases that mattered, no more correct. That is the whole argument for treating model choice as your first latency lever, not your last.

The seven levers, ranked

Ordered by how much they move wall-clock latency for a typical API user, biggest first:

1. Model choice — the single biggest lever; route easy requests to a fast small model.
2. Turn off reasoning for simple tasks — removes a large, sequential thinking-token penalty.
3. Shorten output — total latency tracks output length almost linearly.
4. Prompt / prefix caching — cuts TTFT on long, repeated prefixes.
5. Streaming — cuts perceived latency, not total time.
6. Speculative decoding — 2-3x, but your provider runs it, not you.
7. Parallelization & routing — fan out independent calls, fail over on slow endpoints.

Lever 1: pick a fast model

Model choice is repeatedly described as the single biggest latency lever, and our data agrees. The trick is not to standardize on one model but to route: send latency-sensitive, easy requests — classification, lookups, short summaries, boilerplate code — to a fast tier, and reserve a large model for genuinely hard ones. As of mid-2026, speed leaderboards like Artificial Analysis show fast-tier models such as Gemini 2.5 Flash and Claude Haiku 4.5 delivering the lowest TTFT, consistently under roughly 600ms on medium prompts, and specialized hardware pushing hundreds of tokens per second. Those numbers move hourly, so link the live leaderboard rather than memorizing a millisecond figure.

The gateway pattern makes this cheap to adopt: one key, many models, switch by changing a string. See what is an LLM gateway and, for the routing and failover mechanics, our routing and failover guide.

Lever 2: turn off reasoning

A reasoning model generates a chain of internal thinking tokens before it writes your answer. Because decode is sequential, those tokens are pure added wall-clock time before the first useful word appears. Reported overheads vary widely — some studies cite 10s of extra seconds or 20x-plus latency versus non-reasoning models — so treat those as illustrative extremes. Our own 6-7x spread is the safer anchor, and it points the same direction.

Most providers let you disable reasoning per request for straightforward work, using a pattern like reasoning={'enabled': False}. Syntax varies by vendor and SDK version, so check current docs rather than copying verbatim. The rule of thumb: if the task is deterministic and you would not think out loud to do it, neither should the model.

Lever 3: shorten the output

Because total latency scales with output length, the cheapest tuning you have is telling the model to stop sooner. A simple be-brief instruction typically cuts output token count by roughly a third, which directly lowers total latency and cost at the same time. Cap max_tokens to a realistic ceiling, ask for structured or bulleted output instead of prose when you will parse it anyway, and avoid prompting patterns that invite the model to restate the question. Every token you do not generate is a forward pass you do not pay for — the same logic behind our guide to cutting token costs in coding agents.

Levers 4-5: streaming and caching

Streaming (stream=true) is a perception fix. It does not shorten total completion time; it lowers TTFT so users watch words appear instead of staring at a spinner. For chat UIs it is close to mandatory. For a batch job whose output nobody watches live, it changes nothing.

Prompt caching attacks TTFT on long, repeated prefixes. OpenAI caches prompts of 1024 tokens or more automatically, in 128-token increments, with no code change, and cites up to about 80 percent latency reduction on cached prefixes — a best-case figure, not a typical one. Anthropic caching similarly improves TTFT for long documents, with cache reads at 0.1x base input price, though it publishes no fixed percentage. Structure your prompts so the stable part (system prompt, few-shot examples, long context) comes first and the variable part comes last, and you turn latency into a function of output length rather than total context.

Levers 6-7: what your provider does

Speculative decoding uses a small draft model to propose several tokens that the large target model verifies in a single forward pass, preserving the output distribution while delivering roughly 2-3x speedup (NVIDIA reported up to 3.6x throughput on H200). It is powerful, but it is an infrastructure technique — most API users cannot toggle it. You benefit from it indirectly through faster endpoints.

Parallelization and routing is the last mile. Fan out independent requests concurrently instead of serially, and route around slow or failing endpoints automatically. If you are also chasing the bill, the same levers usually help — see how to cut LLM API costs, since faster non-reasoning models were frequently cheaper per task too.

FAQ