---
title: "Sweep files"
description: "List the engine configs a replay tries."
canonical_url: "https://inference-cost-optimizer.com/replay/sweep-file"
---
# Sweep files

> List the engine configs a replay tries.

A sweep file lists the configs to try. Each config is the replica's flags in production with some of them changed. The current config is always measured too, first.

```yaml [sweep.yaml]
# Configs listed one by one.
configs:
  - max-num-batched-tokens: 16384
  - max-num-batched-tokens: 2048
  - enable-prefix-caching: false
  - kv-cache-dtype: fp8
  - quantization: fp8

# Every combination of these values, on top of the list above.
grid:
  max-num-seqs: [128, 256]
```

## How flags are written

Flags are written as vLLM takes them, without the leading dashes:

<table>
<thead>
  <tr>
    <th>
      In the sweep file
    </th>
    
    <th>
      On vLLM's command line
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        max-num-seqs: 128
      </code>
    </td>
    
    <td>
      <code>
        --max-num-seqs 128
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        enable-prefix-caching: true
      </code>
    </td>
    
    <td>
      <code>
        --enable-prefix-caching
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        enable-prefix-caching: false
      </code>
    </td>
    
    <td>
      <code>
        --no-enable-prefix-caching
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        max-model-len: null
      </code>
    </td>
    
    <td>
      the flag is removed
    </td>
  </tr>
</tbody>
</table>

A config that comes out the same as one already listed, or as the current config, is tried once.

## Configs that change precision

A config that changes `quantization`, `dtype`, `kv-cache-dtype` or `model` is measured, but it cannot be named the cheapest: it may answer worse, and the replay does not check answer quality yet. It shows in the results as **quality unverified**. Check its answers yourself before you adopt it.

## What is worth trying

Start with the flags that trade latency for throughput on your traffic:

- **max-num-seqs**: how many requests run at once. Fewer shortens the wait for the first token; more serves more at a time, as long as the KV cache holds them.
- **max-num-batched-tokens**: how many tokens one engine step processes. Larger chunks prefill long prompts in fewer steps but stall the decoding of others for longer.
- **gpu-memory-utilization**: how much GPU memory vLLM takes, most of it for the KV cache.
- **kv-cache-dtype: fp8**: twice the KV cache in the same memory. Changes precision.
- **enable-prefix-caching: false**: to see what the prefix cache is worth on your traffic.

Configs that cannot fit the model in GPU memory on paper are ruled out before any restart, so a generous grid costs nothing for those.


## Sitemap

See the full [sitemap](https://inference-cost-optimizer.com/sitemap.md) for all pages.