Sweep files
A sweep file lists the configs to try. Each config is the replica's flags in production with some of them changed. The current config is always measured too, first.
# Configs listed one by one.
configs:
- max-num-batched-tokens: 16384
- max-num-batched-tokens: 2048
- enable-prefix-caching: false
- kv-cache-dtype: fp8
- quantization: fp8
# Every combination of these values, on top of the list above.
grid:
max-num-seqs: [128, 256]
How flags are written
Flags are written as vLLM takes them, without the leading dashes:
| In the sweep file | On vLLM's command line |
|---|---|
max-num-seqs: 128 | --max-num-seqs 128 |
enable-prefix-caching: true | --enable-prefix-caching |
enable-prefix-caching: false | --no-enable-prefix-caching |
max-model-len: null | the flag is removed |
A config that comes out the same as one already listed, or as the current config, is tried once.
Configs that change precision
A config that changes quantization, dtype, kv-cache-dtype or model is measured, but it cannot be named the cheapest: it may answer worse, and the replay does not check answer quality yet. It shows in the results as quality unverified. Check its answers yourself before you adopt it.
What is worth trying
Start with the flags that trade latency for throughput on your traffic:
max-num-seqs: how many requests run at once. Fewer shortens the wait for the first token; more serves more at a time, as long as the KV cache holds them.max-num-batched-tokens: how many tokens one engine step processes. Larger chunks prefill long prompts in fewer steps but stall the decoding of others for longer.gpu-memory-utilization: how much GPU memory vLLM takes, most of it for the KV cache.kv-cache-dtype: fp8: twice the KV cache in the same memory. Changes precision.enable-prefix-caching: false: to see what the prefix cache is worth on your traffic.
Configs that cannot fit the model in GPU memory on paper are ruled out before any restart, so a generous grid costs nothing for those.