Where LLM inference benchmarks hide queueing delay

Overview

The latency of a dispatched request can stay flat while the time spent waiting to send it keeps growing. In a controlled queueing model, changing the arrival interval from 400 ms to 100 ms increased mean arrival-to-completion latency from 400 ms to 3,250 ms. The dispatch-to-completion measurement stayed at 400 ms. At a one-second deadline, the fraction of requests finishing on time fell from 100% to 15%.

The service did not get slower. The queue got longer, and one timer left that queue out.

The timing question is where measurement begins: before comparing inference systems, reported latency should include the interval the application actually waits. For an agent waiting on a model call or a person waiting for an answer, time spent before dispatch still delays the next step.

What the benchmark clock measures

NVIDIA AIPerf distinguishes request latency, measured from request start to response end, from effective latencyTime from a request's scheduled credit issuance until its response completes., measured from credit issuance to response end. Subtracting the two exposes the credit-to-start wait in the load generator [1].

Those boundaries give three useful measurements. Let a be the intended arrival, c the issued credit, s the request start and e the response end:

The distinction is about where the clock starts. Request latency in a real trace can already include network transit, server queueing and response delivery. This model isolates the additional waiting before dispatch.

Experiment design

The model uses a single FIFOA queue rule that serves requests in arrival order. processing slot and a constant 400 ms service time. Each scenario contains 20 requests. Their arrival interval is the only parameter changed: 100, 200, 400 or 800 ms, corresponding to scheduled rates of 10, 5, 2.5 and 1.25 requests per second.

Credits are issued at their scheduled arrival times. A request waits for the previous one to finish before dispatching, and every request is allowed to complete. The four scenarios produce 80 request records, each retaining its scheduled arrival, issued credit, dispatch and completion timestamps.

For request i, processing follows:

The first request starts at its credit time. Holding service constant makes the source of the change easy to identify: any additional latency is waiting, rather than a different kernel, model or response length. GPU batching, prefill and decoding are outside this model.

Results

At 800 ms between arrivals, each request finishes before the next one arrives. At 400 ms, arrivals and completions line up exactly. Both scenarios have a 400 ms mean latency and no queue wait.

At 200 ms, each arrival adds 200 ms to the waiting time of the next request. The mean credit-inclusive latency reaches 2,300 ms. At 100 ms, the queue adds 300 ms per request, raising the mean to 3,250 ms. The last request completes 6,100 ms after its scheduled arrival, including 5,700 ms of waiting before dispatch.

The figure below keeps the vertical scale fixed across scenarios. Select an arrival interval and inspect the individual requests to separate the constant service time from the accumulated wait.

The one-second deadline makes the consequence more concrete. Dispatch-based timing counts all 20 requests as passing in every scenario. Counting from issued credit, only 4 of 20 pass at 200 ms arrivals, and 3 of 20 at 100 ms arrivals. Requests completing at exactly 1,000 ms count as passing.

There is no uncertainty interval on these plots because they show exact outputs of a deterministic recurrence. The trace is deliberately small enough to inspect. Extending the overloaded scenarios would increase their mean and maximum latency even though dispatch latency remained 400 ms.

Why the queue grows

One slot taking 400 ms per request can complete 2.5 requests per second while continuously busy. Scheduling ten requests per second adds work faster than that slot can clear it.

For this regular arrival pattern, the waiting time of request i, counting the first as zero, is i × max(400 − arrival interval, 0) milliseconds. The 100 ms case therefore adds 300 ms of wait at every step: 0, 300, 600, and so on, up to 5,700 ms.

The zero-wait result at 400 ms is equally specific. It depends on perfectly regular arrivals and service times. A real workload with the same averages can still queue when requests arrive in bursts or take different amounts of time. Mean rate alone does not specify the workload.

When the benchmark changes its own workload

A fixed arrival stream asks how the system handles work that keeps arriving. A fixed concurrency asks what happens while a limited number of requests remain active. Both are useful experiments, but their latency numbers need that context.

This distinction has a longer history than LLM serving. In Open Versus Closed: A Cautionary Tale, Schroeder, Wierman and Harchol-Balter distinguish open workloads, whose arrivals are independent of completions, from closed workloads, whose new arrivals depend on completed work. Their implementation and simulation study shows why that choice changes system behavior [4]. The timing experiment here examines one small part of that broader measurement problem.

AIPerf documents rate-based scheduling separately from concurrency-only generation. It also describes cases where the client delivers less than its configured request rate [2]. A comparison should retain the observed dispatch trace rather than assuming that a command-line rate describes the load that reached the server.

Prefill limits add another control point. NVIDIA warns that a prefill concurrency cap can throttle the generator and produce optimistic latency relative to traffic that continues arriving independently [3]. The limit may be appropriate for the intended workload. It should still be reported, because it changes the question being tested.

This is where a queue-inclusive timer and a workload check complement each other. The timer accounts for recorded waiting. The workload check asks whether the expected requests entered the experiment in the first place.

A second gap before credit issuance

Consider one request intended for 0 ms whose credit is not issued until 300 ms. It starts immediately and finishes at 700 ms.

Its request latency is 400 ms. Its credit-inclusive latency is also 400 ms. Its schedule-inclusive latency is 700 ms.

Agreement between the first two measurements establishes that there was no recorded credit-to-start wait for this request. The 300 ms before credit issuance remains outside both. Keeping the intended schedule makes that gap visible; request records alone cannot describe arrivals that were never issued.

What to measure next

The next experiment is to keep average arrival rate fixed and change burstiness. That would test whether two workloads with identical mean load produce different queueing and deadline behavior. A useful comparison would preserve the actual arrival traces and show latency alongside queue depth, rather than presenting only an average.

Moving to an inference server adds a second question: how do scheduling and batching change the service process itself? That comparison needs the same model, hardware and input/output-length distribution, plus explicit warmup, concurrency and drain settings. Errors and timeouts belong alongside successful completions, not outside the denominator.

For GPU allocation and agent-serving work, this is the connection worth pursuing: more completed tokens or a faster kernel only matters to the application if useful requests finish within its time budget. The first step is making sure the benchmark has not started its clock after the waiting began.

References

[1] NVIDIA AIPerf v0.13.0. Effective vs Active Metrics. Definitions of request latency and effective_latency.

[2] NVIDIA AIPerf v0.13.0. Load Generator Options Reference. Rate-based scheduling, concurrency limits and under-delivery.

[3] NVIDIA AIPerf v0.13.0. Prefill Concurrency. Workload throttling and coordinated omission.

[4] Bianca Schroeder, Adam Wierman and Mor Harchol-Balter. Open Versus Closed: A Cautionary Tale. NSDI 2006.

Sources checked November 6, 2026.