Separating prompt processing from token generation can protect a streaming response from a large incoming prompt. It also creates a new obligation: move the prompt's KV state to the decoder without consuming the latency budget the separation was meant to protect.
DistServe assigns prefillThe phase that processes the prompt and builds attention state for generation. and decoding to different GPUs and selects resource allocations and parallelism with both time-to-first-token and time-per-output-tokenA decoding-speed measure whose averaging window must be stated. requirements in mind. Its placement decisions account for the bandwidth available between phases. The system's objective concerns requests meeting latency requirements, not throughput regardless of delay. DistServe paper
DistServe's evaluated settings report up to 7.4× more requests or 12.6× tighter service-level objectives while more than 90% of requests satisfy their latency constraints. These are conditional maxima from a resource-allocation and placement system. They should not be read as the expected benefit of adding a transfer between any prefill and decode workers.
Make the handoff visible
Consider a synthetic request whose prefill takes 40 milliseconds and whose subsequent decoding steps take 10 milliseconds each. Assume prefill emits the first output token, then the KV state is transferred before the next token can be decoded. There is no queuing and no overlap of transfer with decoding.
Set a first-token limit of 80 milliseconds and a maximum inter-token gap of 20 milliseconds. The first token arrives comfortably within its limit. Steady-state decoding also fits the token-gap limit. But the first gap after prefill is transfer_time + 10 milliseconds, so transfer has only 10 milliseconds of headroom.
At an assumed effective link rate of one GiB per second, a four-MiB cache takes 3.90625 milliseconds to transfer. The handoff gap is approximately 13.906 milliseconds, which passes. A sixteen-MiB cache takes 15.625 milliseconds, yielding a 25.625-millisecond gap that fails this deliberately strict requirement.
These are byte-count calculations using invented service times and a declared effective bandwidth. They are not measurements of DistServe or a particular interconnect. GiB and MiB use powers of two, making the transfer arithmetic auditable.
Which latency statistic does the service promise?
A maximum token-gap requirement is stricter than an average time-per-output-token requirement. For a long response, one slow handoff can disappear inside a favorable average. For an interactive application, that single pause may still matter. An evaluation should state whether it uses a per-request average, a percentile of gaps, or the worst gap.
The example also states where the first token is emitted. If a serving design withholds it until handoff, some delay moves into first-token latency instead. Moving the reporting boundary does not remove elapsed time. Both the first response and later gaps need to be evaluated under the actual delivery behavior.
Separation may avoid interference: in a serialized toy colocated worker, a 40-millisecond prefill inserted ahead of a 10-millisecond decode can create a 50-millisecond gap for an existing stream. Yet comparing one shared worker against two dedicated workers would be an unfair resource comparison. A throughput study must hold total resources fixed and vary how those resources are divided.
The accompanying calculator only checks an isolated handoff budget. It omits queuing, cache construction details, contention, overlapping transfers, and batching, so passing its test is necessary only under its stated model, not a deployment guarantee.
Draw the feasible region before choosing a link
The extended calculator sweeps seven cache sizes and six effective bandwidths. With a 10-millisecond transfer budget, the boundary is cache_MiB ≤ 10.24 × bandwidth_GiB_per_second. At 0.25 GiB/s the limit is 2.56 MiB; at one GiB/s it is 10.24 MiB; at eight GiB/s it is 81.92 MiB. The heatmap gives the actual handoff gap at each of the 42 tested points instead of reducing them to a single pass percentage.
The effective bandwidth matters. Peak link specifications do not include every source of contention, protocol overhead, or transfer scheduling delay. Adding any fixed handoff overhead consumes the same 10-millisecond budget before a byte moves. If the decoder can overlap some transfer work safely, the model must account for that overlap rather than subtracting it informally from the final number.
A second sweep fixes the 16-MiB, one-GiB/s case and varies response length. If there are N output tokens and the first is emitted by prefill, the average of the N − 1 following gaps is 10 + 15.625/(N − 1) milliseconds. It already falls below 20 milliseconds at N = 3, even though the maximum gap stays 25.625 milliseconds for every response length.
This is not an argument against an average-TPOT objective. It is an argument for naming it correctly and checking whether it matches the product promise. A deployment that passes DistServe-style average latency constraints can still need a separate worst-gap check for an application sensitive to a single streaming pause.