---
title: "Configuration"
description: "Every key of the YAML config file, its default, and which part of ico reads it."
canonical_url: "https://inference-cost-optimizer.com/reference/configuration"
---
# Configuration

> Every key of the YAML config file, its default, and which part of ico reads it.

The gateway, the collector and the CLI read the same YAML format. Each reads the keys it needs and ignores the rest, so one file can serve all three on a machine, and each machine can have its own. Secrets never go in the file: they come from the environment.

<table>
<thead>
  <tr>
    <th>
      Section
    </th>
    
    <th>
      Read by
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        replicas
      </code>
    </td>
    
    <td>
      all
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        headers
      </code>
      
      , <code>
        gateway
      </code>
    </td>
    
    <td>
      the gateway (and the CLI, for <code>
        gateway.admin_url
      </code>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        gpu_price_per_hour
      </code>
      
      , <code>
        models
      </code>
      
      , <code>
        slo
      </code>
      
      , <code>
        clickhouse
      </code>
    </td>
    
    <td>
      the collector, <code>
        ico report
      </code>
      
      , <code>
        ico replay
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        collector
      </code>
    </td>
    
    <td>
      the collector
    </td>
  </tr>
</tbody>
</table>

## `replicas`

The engine replicas. At least one; names must be unique.

<field-group>
<field name="name" type="string" :required="true">

The replica's name. vLLM's `OTEL_SERVICE_NAME` must be the same, and every file that lists the replica must use the same name.

</field>

<field name="url" type="string" :required="true">

How the program reading this file reaches the engine, `http://` or `https://`. The gateway sends requests there, the collector scrapes `<url>/metrics`, and `ico replay` sends its load there unless `--replica-url` says otherwise.

</field>

<field name="host" type="string">

The machine the replica runs on. It ties the replica to the GPU exporter with the same `host`.

</field>

<field name="gpu_model" type="string">

The card, as a key of `gpu_price_per_hour`.

</field>

<field name="gpu_count" type="integer">

Default `1`. How many GPUs the replica uses: its tensor-parallel size.

</field>

<field name="models" type="string[]">

Gateway only. Send only these models here. Empty: any model.

</field>

<field name="launcher" type="object">

How `ico replay` restarts this engine. See [Launchers](https://inference-cost-optimizer.com/replay/launchers).

- `kind`: `manual` (default), `command` or `docker`.
- `container`: for `docker`, the engine's container.
- `start`, `stop`, `env`, `cwd`: for `command`.
- `flags`: for `command` and `manual`, the engine's flags in production, without the leading dashes.

</field>
</field-group>

## `gpu_price_per_hour`

USD per GPU-hour, by `gpu_model`. Put in what you pay. Idle hours are billed like busy ones.

```yaml
gpu_price_per_hour:
  H100-SXM: 2.99
  L40S: 0.86
```

## `models`

What each served model weighs, for the replay's check of whether a config fits in GPU memory. Every key is required for a model that is listed.

<field-group>
<field name="params" type="integer">

Number of parameters.

</field>

<field name="weight_bytes_per_param" type="number">

`2` for BF16 or FP16, `1` for FP8.

</field>

<field name="kv_bytes_per_token" type="integer">

KV cache per token: 2 × layers × KV heads × head size × bytes per value.

</field>
</field-group>

## `headers`

<field-group>
<field name="tenant" type="string">

Default `X-Tenant-Id`. The request header that names the customer.

</field>

<field name="feature" type="string">

Default `X-Feature`. The request header that names the part of your product.

</field>
</field-group>

## `slo`

The latency limits a config must hold in a replay, at the 95th percentile. `--slo` overrides them.

<field-group>
<field name="ttft_p95_ms" type="number">

Default `500`. Time to first token, in milliseconds.

</field>

<field name="itl_p95_ms" type="number">

Default `50`. Gap between tokens, in milliseconds.

</field>
</field-group>

## `gateway`

<field-group>
<field name="listen" type="string">

Default `:8080`. Where the gateway takes requests.

</field>

<field name="admin_listen" type="string">

Default `:8081`. The admin API. Keep it on `127.0.0.1` or a private address.

</field>

<field name="admin_url" type="string">

Default `http://localhost:8081`. Where the CLI reaches the admin API. `ICO_GATEWAY_ADMIN` overrides it.

</field>

<field name="collector" type="string">

The collector's gRPC address, `host:port`. Empty: records are not exported.

</field>

<field name="id" type="string">

Default: the machine's hostname. Names this gateway to the collector.

</field>

<field name="state_dir" type="string">

Default `/var/lib/ico`. Where the gateway keeps the salt of its hashes, unless `ICO_HASH_SALT` is set.

</field>

<field name="max_body_bytes" type="integer">

Default 32 MiB. The largest request body accepted.

</field>

<field name="shutdown_timeout" type="duration">

Default `30s`. How long requests in flight get to finish when the gateway stops.

</field>

<field name="prefix_hash" type="object">

How request text is hashed to find shared prefixes: `block_chars` (default `256`) characters per block, up to `max_blocks` (default `64`) blocks.

</field>
</field-group>

## `collector`

<field-group>
<field name="grpc_listen" type="string">

Default `0.0.0.0:4317`. Where gateway records and engine traces arrive.

</field>

<field name="scrape_interval_s" type="number">

Default `5`. How often engines and GPU exporters are scraped, in seconds.

</field>

<field name="gpu_exporters" type="object[]">

The GPU exporters to scrape, each with `kind` (`dcgm` for dcgm-exporter, `nvidia_smi` for nvidia_gpu_exporter), `url` (its `/metrics` page) and `host` (matching the replicas' `host`).

</field>

<field name="flush_interval_s" type="number">

Default `1`. How often rows are written to ClickHouse.

</field>

<field name="max_buffered_rows" type="integer">

Default `500000`. Rows held while ClickHouse is unreachable. Past this, the oldest are dropped.

</field>
</field-group>

## `clickhouse`

<field-group>
<field name="host" type="string">

Default `localhost`. `ICO_CLICKHOUSE_HOST` overrides it.

</field>

<field name="port" type="integer">

Default `8123`, the HTTP port. `ICO_CLICKHOUSE_PORT` overrides it.

</field>

<field name="username" type="string">

Default `default`. The Compose deployment uses `ico`.

</field>

<field name="database" type="string">

Default `ico`. Created by the collector if it does not exist.

</field>

<field name="secure" type="boolean">

Default `false`. HTTPS to ClickHouse.

</field>
</field-group>

The password comes only from `ICO_CLICKHOUSE_PASSWORD`.

## Environment variables

<table>
<thead>
  <tr>
    <th>
      Variable
    </th>
    
    <th>
      Read by
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        ICO_CONFIG
      </code>
    </td>
    
    <td>
      all
    </td>
    
    <td>
      Path of the config file. Default <code>
        /etc/ico/config.yaml
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_ADMIN_TOKEN
      </code>
    </td>
    
    <td>
      gateway, <code>
        ico replay
      </code>
    </td>
    
    <td>
      Bearer token of the admin API. Unset: the API refuses every call.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_HASH_SALT
      </code>
    </td>
    
    <td>
      gateway
    </td>
    
    <td>
      Salt of the prefix hashes. Use the same on every gateway.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_CLICKHOUSE_PASSWORD
      </code>
    </td>
    
    <td>
      collector, CLI
    </td>
    
    <td>
      ClickHouse password.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_CLICKHOUSE_HOST
      </code>
      
      , <code>
        ICO_CLICKHOUSE_PORT
      </code>
    </td>
    
    <td>
      collector, CLI
    </td>
    
    <td>
      Override <code>
        clickhouse.host
      </code>
      
       and <code>
        clickhouse.port
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_GATEWAY_ADMIN
      </code>
    </td>
    
    <td>
      CLI
    </td>
    
    <td>
      Overrides <code>
        gateway.admin_url
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_SCHEMA_DIR
      </code>
    </td>
    
    <td>
      collector
    </td>
    
    <td>
      Where the ClickHouse migrations are. Default <code>
        /opt/ico/schema
      </code>
      
      .
    </td>
  </tr>
</tbody>
</table>


## Sitemap

See the full [sitemap](https://inference-cost-optimizer.com/sitemap.md) for all pages.