When you send a prompt to a large language model, you wait for a moment and then watch the response arrive one word at a time. Those are two distinct computations, and they place opposite demands on the hardware.
Reading your prompt, called prefill, processes every token at once, providing the GPU with enough parallel arithmetic to reach its compute ceiling. Generating the response is called decode, and that phase produces one token per step, which leaves most of those same arithmetic units idle.
A standard deployment runs both phases on the same GPU, where each one degrades the other. Separating them across two GPU pools lets each phase run on hardware matched to its demands. The DistServe paper measures that gain as serving "7.4x more requests" under the same latency targets.
This article explains what each phase does and why running them together reduces your throughput. It then covers how a disaggregated system works and when the added complexity is justified.
Prefill is the phase that processes your entire input prompt and produces the first output token. The model reads every prompt token in a single pass and computes the attention keys and values for each one.
Decode is the phase that generates the remaining output tokens, one at a time. Each step reads the stored keys and values from all previous tokens and produces one new token. It then appends that token’s own key and value to the store.
The store is called the KV cache. Attention needs every earlier token’s keys and values to compute the next token, so the cache grows throughout the response. A detailed explanation of the attention mechanism creating it is available in the transformer models explainer.
The two phases reach different limits, and arithmetic density explains why. Prefill has thousands of tokens to process simultaneously, which supplies enough parallel work to reach the GPU’s compute ceiling. Decode has one token per request, so the GPU reads its full set of model weights to produce that token and waits on memory through most of each step.
| Property | Prefill | Decode |
|---|---|---|
| What it processes | The whole prompt at once | One token per step |
| Tokens in flight | Thousands | One for each request |
| What limits it | Compute throughput | Memory bandwidth |
| Gain from larger batches | Small, since compute is already saturated | Large, since one weight read serves many requests |
| User-facing metric it sets | Time to first token | Time between output tokens |
| Hardware it suits | Highest available compute | High memory bandwidth at lower cost |
A colocated server interleaves the two phases across the requests it holds. A long prompt keeps the GPU busy while it computes, and every decode step queued behind it stalls.
DistServe names the effect directly, reporting "strong prefill-decoding interferences" in systems that run both phases together. One request with a long prompt raises the inter-token latency of other users sharing that GPU.
Batching sharpens that conflict further. Decode throughput improves with large batches, because the GPU reads the model weights once and applies them across many requests. Prefill gains almost nothing, since one prompt already occupies the compute units. Tuning the batch size for decode throughput lengthens the prefill queue, and tuning it for fast prefill leaves decode capacity unused.
Resource allocation creates the same bind. A deployment picks its parallelism strategy and its GPU type once, for two workloads with opposite profiles. Whichever phase you tune for, the other runs on a configuration built for something else.
The result is a tradeoff between two metrics your users feel separately. Time to first token measures how long someone waits before text appears. Time between output tokens measures how smoothly it then arrives. Colocation forces you to sacrifice one to protect the other.
Prefill-decode disaggregation runs the two phases on separate pools of GPUs and transfers the KV cache between them.
A request moves through four stages:
Each pool can now be configured for the work it actually does. Prefill nodes receive the hardware and parallelism that suit compute-bound processing. Decode nodes receive large batches and enough memory to hold many caches at once. Scaling becomes independent as well, so a workload with long prompts and short answers can add prefill nodes without provisioning decode capacity it will never use.
The transfer is the engineering cost the design introduces. Mooncake, the serving platform behind the Kimi assistant, describes "a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters." It also uses "the underutilized CPU, DRAM, and SSD resources" of the cluster to store caches outside GPU memory.
That last detail changes how you think about the cache. A managed tier has its own capacity and its own eviction policy, which lets a stored cache serve more than one request. Two requests that begin with the same text can reuse the same stored keys and values, so a repeated system prompt needs prefill only once.
Disaggregation is measured in goodput, which counts only the requests a system serves inside its latency targets. Those targets are stated formally as a service-level objective, or SLO. A system that emits a high volume of tokens while missing its deadlines scores poorly on goodput and well on raw throughput, which is the gap the metric exists to expose.
DistServe pairs the throughput figure from the opening with its alternative reading, where the same system holds a "12.6x tighter SLO" while staying "within latency constraints for > 90% of requests." The two numbers describe one result, because identical headroom either absorbs additional traffic or supports stricter latency promises.
Mooncake reports "up to a 525% increase in throughput in certain simulated scenarios." Its production number is the more informative one, where the architecture "enables Kimi to handle 75% more requests."
The distance between those two figures deserves attention. A simulation removes the network variance and the cache-transfer stalls that a live deployment contains, so a comparable workload should expect the production figure.
Disaggregation is one answer to interference. Chunked prefill is the other, and it works inside a single colocated server.
The technique splits a long prompt into smaller pieces and processes one piece per scheduling step, with decode work for other requests batched alongside it. No single prefill then occupies the GPU for an uninterrupted stretch, and decode steps proceed at a steadier rate.
Each approach optimizes something different. Chunked prefill reduces interference without any network transfer and keeps one pool of identical servers, which is simpler to operate. Total prefill time rises, because splitting a prompt means reading the model weights once for every chunk.
Disaggregation ends the contention outright and gives each pool the hardware its phase actually needs. Its costs are a KV cache transfer on every request and two clusters to size separately.
The scheduler change takes less work, which makes it the reasonable first attempt.
Disaggregation adds a network hop to the critical path of every request, and three conditions determine whether the gain exceeds it.
Prompt length against output length. The design suits workloads where prompts are long and responses are comparatively short, which describes document analysis and retrieval-augmented work. A chat application with brief prompts and long answers sits in decode for most of its runtime, so splitting off a short prefill changes little.
Request volume. Each pool carries its own minimum capacity. At low volume you operate two partly idle clusters where one busy GPU would have served the traffic.
Interconnect bandwidth. The KV cache holds a key and a value vector for every token in every attention layer, so its size grows with both prompt length and model depth. At production scale that reaches gigabytes per request. On a slow link, moving it adds more to time to first token than the removal of interference subtracts, which turns the optimization into a regression.
Operational maturity matters alongside those three. A disaggregated system has more failure modes and more components to monitor, with a cache lifecycle to manage on top. The practices in this guide to building scalable AI pipelines apply directly to running one.
Three numbers tell you whether colocation is costing you anything.
Measure these before you redesign anything. A colocated deployment that meets its targets at current volume needs capacity, and a new architecture would solve a problem it does not have. Instrumenting the two phases separately is the prerequisite, and this guide to AI model observability covers the monitoring that makes the comparison possible.
Prefill and decode need different things from a GPU. One wants arithmetic throughput and the other wants memory bandwidth, and a single server configuration cannot deliver both at once.
Disaggregation resolves that conflict by giving each phase its own hardware, at the price of moving a KV cache across the network for every request. The published results support the trade at scale, with DistServe measuring a large goodput gain and Mooncake reporting 75 percent more requests in production.
Measure your time to first token and your inter-token latency separately before you commit. Those two measurements show whether the phases are interfering, and that answer decides whether this architecture is worth building.
Don't miss this opportunity to share your voice and make an impact in the Ai community. Feature your blog on ARTiBA!
Contribute