What a replay cannot tell you yet
Capacity to plan a fleet by
The replay sends the requests of its sample at steady rates: each goes out on schedule, whether or not the ones before it have finished. Real users send their next message after they have read the last answer, so under load they slow down with the engine; and a sample of 300 requests is a few seconds to a few minutes of your traffic.
On the H100 test, the replay found 22.5 requests a second where the same traffic, sent live, was comfortable at about 22: one test, on one traffic mix. Use a replay's rates to compare configs with each other, and before you plan a fleet by them, for example to run fewer replicas, confirm the rate on live traffic.
Answer quality
The replay sends token ids, not text, and does not look at the answers. A config that changes precision, such as FP8 weights or an FP8 KV cache, is measured but never recommended. Check its answers on your own evaluation before you adopt it.
Speculative decoding
How much speculative decoding speeds up generation depends on the content of the text. Synthetic prompts have none, so the replay cannot measure it.
Downtime on a single replica
The tested replica takes no traffic for the length of the run. With one replica per model, that model is unavailable meanwhile: --single-replica asks you to accept that, so run it when traffic is low.
GPUs you do not have
A replay runs on the replica's own GPUs. To compare another GPU type, you need a machine with it, set up like the others.
Kubernetes
There is no launcher for Kubernetes yet. See Launchers.