Reference

Configuration

Every key of the YAML config file, its default, and which part of ico reads it.

The gateway, the collector and the CLI read the same YAML format. Each reads the keys it needs and ignores the rest, so one file can serve all three on a machine, and each machine can have its own. Secrets never go in the file: they come from the environment.

Section	Read by
`replicas`	all
`headers`, `gateway`	the gateway (and the CLI, for `gateway.admin_url`)
`gpu_price_per_hour`, `models`, `slo`, `clickhouse`	the collector, `ico report`, `ico replay`
`collector`	the collector

`replicas`

The engine replicas. At least one; names must be unique.

name

string required

The replica's name. vLLM's OTEL_SERVICE_NAME must be the same, and every file that lists the replica must use the same name.

url

string required

How the program reading this file reaches the engine, http:// or https://. The gateway sends requests there, the collector scrapes <url>/metrics, and ico replay sends its load there unless --replica-url says otherwise.

host

string

The machine the replica runs on. It ties the replica to the GPU exporter with the same host.

gpu_model

string

The card, as a key of gpu_price_per_hour.

gpu_count

integer

Default 1. How many GPUs the replica uses: its tensor-parallel size.

models

string[]

Gateway only. Send only these models here. Empty: any model.

launcher

object

How ico replay restarts this engine. See Launchers.

kind: manual (default), command or docker.
container: for docker, the engine's container.
start, stop, env, cwd: for command.
flags: for command and manual, the engine's flags in production, without the leading dashes.

`gpu_price_per_hour`

USD per GPU-hour, by gpu_model. Put in what you pay. Idle hours are billed like busy ones.

gpu_price_per_hour:
  H100-SXM: 2.99
  L40S: 0.86

`models`

What each served model weighs, for the replay's check of whether a config fits in GPU memory. Every key is required for a model that is listed.

params

integer

Number of parameters.

weight_bytes_per_param

number

2 for BF16 or FP16, 1 for FP8.

kv_bytes_per_token

integer

KV cache per token: 2 × layers × KV heads × head size × bytes per value.

`headers`

tenant

string

Default X-Tenant-Id. The request header that names the customer.

feature

string

Default X-Feature. The request header that names the part of your product.

`slo`

The latency limits a config must hold in a replay, at the 95th percentile. --slo overrides them.

ttft_p95_ms

number

Default 500. Time to first token, in milliseconds.

itl_p95_ms

number

Default 50. Gap between tokens, in milliseconds.

`gateway`

listen

string

Default :8080. Where the gateway takes requests.

admin_listen

string

Default :8081. The admin API. Keep it on 127.0.0.1 or a private address.

admin_url

string

Default http://localhost:8081. Where the CLI reaches the admin API. ICO_GATEWAY_ADMIN overrides it.

collector

string

The collector's gRPC address, host:port. Empty: records are not exported.

string

Default: the machine's hostname. Names this gateway to the collector.

state_dir

string

Default /var/lib/ico. Where the gateway keeps the salt of its hashes, unless ICO_HASH_SALT is set.

max_body_bytes

integer

Default 32 MiB. The largest request body accepted.

shutdown_timeout

duration

Default 30s. How long requests in flight get to finish when the gateway stops.

prefix_hash

object

How request text is hashed to find shared prefixes: block_chars (default 256) characters per block, up to max_blocks (default 64) blocks.

`collector`

grpc_listen

string

Default 0.0.0.0:4317. Where gateway records and engine traces arrive.

scrape_interval_s

number

Default 5. How often engines and GPU exporters are scraped, in seconds.

gpu_exporters

object[]

The GPU exporters to scrape, each with kind (dcgm for dcgm-exporter, nvidia_smi for nvidia_gpu_exporter), url (its /metrics page) and host (matching the replicas' host).

flush_interval_s

number

Default 1. How often rows are written to ClickHouse.

max_buffered_rows

integer

Default 500000. Rows held while ClickHouse is unreachable. Past this, the oldest are dropped.

`clickhouse`

host

string

Default localhost. ICO_CLICKHOUSE_HOST overrides it.

port

integer

Default 8123, the HTTP port. ICO_CLICKHOUSE_PORT overrides it.

username

string

Default default. The Compose deployment uses ico.

database

string

Default ico. Created by the collector if it does not exist.

secure

boolean

Default false. HTTPS to ClickHouse.

The password comes only from ICO_CLICKHOUSE_PASSWORD.

Environment variables

Variable	Read by	Meaning
`ICO_CONFIG`	all	Path of the config file. Default `/etc/ico/config.yaml`.
`ICO_ADMIN_TOKEN`	gateway, `ico replay`	Bearer token of the admin API. Unset: the API refuses every call.
`ICO_HASH_SALT`	gateway	Salt of the prefix hashes. Use the same on every gateway.
`ICO_CLICKHOUSE_PASSWORD`	collector, CLI	ClickHouse password.
`ICO_CLICKHOUSE_HOST`, `ICO_CLICKHOUSE_PORT`	collector, CLI	Override `clickhouse.host` and `clickhouse.port`.
`ICO_GATEWAY_ADMIN`	CLI	Overrides `gateway.admin_url`.
`ICO_SCHEMA_DIR`	collector	Where the ClickHouse migrations are. Default `/opt/ico/schema`.

What a replay cannot tell you yet

Where the replay's numbers stop, so you can use them safely.

CLI

The ico command and its options.