Overview
The replay lab answers one question: which engine config serves my traffic for less, at the latency my users get today? It answers it on one of your own replicas, with the shape of your own recorded traffic, and puts the replica back the way it was.
What a replay does
Picks a piece of your traffic
A run of 300 consecutive requests the gateway recorded, chosen to look like the whole window: prompt and completion lengths, shared prefixes, and how bursty the arrivals were.
Rebuilds it without any text
No text was recorded, and none is needed. Each request becomes a prompt of token ids of exactly the recorded length, starting with the same tokens wherever the recorded prefix hashes matched, and asks for exactly the recorded number of completion tokens.
The sample is sent many times over, and each pass stands for new traffic of the same shape. Text that several conversations share, such as a system prompt, keeps its tokens from pass to pass, as it would stay in the engine's prefix cache. A conversation's own history gets new tokens on every pass, so the prefix cache serves the replay no more than it served your users.
Rules out what cannot fit
Configs that cannot hold the model's weights and its KV cache in GPU memory, on paper, are dropped before any restart.
Drains the replica
The gateway stops sending it new requests, and the replay waits for the ones in flight to finish. Other replicas take the traffic.
Tests each config on a fresh engine
The engine restarts under each config, the current one first. The replay sends the rebuilt requests at rising rates, doubling, then narrowing down, until the 95th percentile of the time to first token or of the gap between tokens breaks its limit. The highest rate that held, in tokens a second, gives that config's cost per million tokens. Every config, the current one included, is measured on a freshly started engine, so they are compared under the same conditions.
A config is recommended only if it measures at least 10% cheaper than the current one. One measurement of a config varies from run to run: the same config measured twice on a GeForce RTX 2060 held rates 8.5% apart. Configs cheaper by less are reported as within that variation, not as savings. --min-saving changes the margin.
Puts everything back
The original flags are restored and the replica is undrained, whatever happened in between: a config that crashed, a failed health check, Ctrl-C. A watchdog process does the same if the command itself is killed. A replica whose original config will not start again stays drained, so it gets no traffic until you look at it.
What it leaves behind
A table in your terminal, a record in ClickHouse, and a section in the next report on that traffic. From a replay on an H100 serving Qwen3-8B, of 16 minutes of traffic from four simulated customers:
config requests/s tokens/s per million tokens
current 22.50 46,747 $0.0237 cheapest
max-num-batched-tokens=16384 22.50 46,747 $0.0237
max-num-batched-tokens=2048 22.50 46,747 $0.0237
kv-cache-dtype=fp8 22.50 46,747 $0.0237 quality unverified
enable-prefix-caching=False 15.91 33,055 $0.0335
quantization=fp8 not tested: the time budget ran out
max-num-seqs=128 not tested: the time budget ran out
max-num-seqs=256 not tested: the time budget ran out
Here vLLM's defaults were already the cheapest config that holds. Neither prefill chunk size changed what the card could hold, nor did the FP8 KV cache, which changes precision and so could not have won anyway until its answers are checked. Turning off the prefix cache would make each token 41% dearer. The same traffic, sent live, had been comfortable at about 22 requests a second.
These costs are at the highest rate each config sustains, so they compare configs with each other. They hold no idle time and are lower than the costs at the top of a report.
When to run one
When your traffic is low, since the tested replica takes none during the run; when a new model or a new vLLM version ships; or when the report shows a replica busy all the time. You start it, from the command line; there is no schedule unless you set one.
Next: Run a replay.