The replay lab

Run a replay

Set up a GPU machine for replays, start one, and read its plan.

A replay runs on the GPU machine of the replica it tests: it restarts that machine's engine and sends load to it directly, so latency is measured without a network in between.

Prepare the GPU machine

Install the CLI

The ico CLI is a Python package in the repository. With uv:

gpu-1

cd inference-cost-optimizer/py
uv sync --no-dev
source .venv/bin/activate

Or with pip, in a virtual environment: pip install ./inference-cost-optimizer/py.

Extend the machine's config

The replay reads the same file as the gateway on this machine. Add what it needs: how to restart the engine, the latency limits, prices, the model's size, and where ClickHouse is.

gateway.yaml

replicas:
  - name: gpu-1-r1
    url: http://10.0.0.11:8000
    host: gpu-1
    gpu_model: H100-SXM
    gpu_count: 1
    launcher:                       # how to restart the engine; see Launchers
      kind: docker
      container: vllm

# ... headers and gateway, as before ...

slo:                                # the limits a config must hold
  ttft_p95_ms: 1000
  itl_p95_ms: 50

gpu_price_per_hour:
  H100-SXM: 2.99

models:
  Qwen/Qwen3-8B:
    params: 8_190_735_360
    weight_bytes_per_param: 2
    kv_bytes_per_token: 147_456

clickhouse:
  host: 10.0.0.5                    # the ico host
  port: 8123
  username: ico
  database: ico

The gateway ignores the keys it does not use.

Write a sweep file

The configs to try, as changes to the flags in production. See Sweep files.

sweep.yaml

configs:
  - max-num-batched-tokens: 16384
  - kv-cache-dtype: fp8
grid:
  max-num-seqs: [128, 256]

Start it

gpu-1

export ICO_ADMIN_TOKEN=...           # the gateway's admin token
export ICO_CLICKHOUSE_PASSWORD=...   # the ico user's password
ico replay --config gateway.yaml --replica gpu-1-r1 --configs sweep.yaml --window 24h --budget 45m

--window and --until choose the recorded traffic the replay is built from.
--budget is the most the whole run may take, restore included.
--slo ttft_p95=1000ms,itl_p95=50ms overrides the limits in the config.
--min-saving 10% is how much cheaper than the current config a config must measure to be recommended.
--single-replica is required when this replica is the only one serving its model: it accepts that the model is unavailable for the length of the run.

All options are in the CLI reference.

Read its plan, and say yes

The replay first measures the current config. Then it knows how long loading a config and testing one take, and prints its plan:

current: holds 22.50 requests/s, 46,747 tokens/s, $0.0237 per million tokens
Loading a config takes about 40s and a full test about 131s. Within the budget that leaves time for 5 of the 7 configs:
  test  max-num-batched-tokens=16384
  test  max-num-batched-tokens=2048
  test  enable-prefix-caching=False
  test  kv-cache-dtype=fp8
  test  quantization=fp8
  skip  max-num-seqs=128
  skip  max-num-seqs=256

It waits for a yes; --yes skips the question. When a full test of every config does not fit the budget but successive halving does, every config gets a short test, the better half a longer one, and the last two a full test. Only a full test can win.

Pick latency limits with room

The replay prints the time to first token your traffic saw in the window. Set the limits above it, with room. When the limit sits right at what the traffic sees today, whether a rate holds comes down to a few requests in each step, and the rate found swings from run to run. If your traffic is already over the limit, the replay says so before it starts.

Overview

What the replay lab tests, how, and what it leaves behind.

Launchers

How the replay restarts your engine under other flags, set per replica.