CLI
ico is the Python package in the repository's py/ folder. It is installed in the collector's image, so on the ico host it runs with docker compose exec collector ico .... Anywhere else, install it with uv sync or pip install ./py.
Every command takes --config FILE, the config file to read (default: $ICO_CONFIG, else /etc/ico/config.yaml).
ico collector
Receives gateway records and engine traces, scrapes engines and GPU exporters, and writes everything to ClickHouse. Applies ClickHouse's migrations at startup. The Compose deployment runs it for you.
| Option | Default | Meaning |
|---|---|---|
--schema-dir DIR | $ICO_SCHEMA_DIR, else /opt/ico/schema | The ClickHouse migrations to apply. |
ico report
Writes the cost and waste report as one static HTML file. See Read the report.
| Option | Default | Meaning |
|---|---|---|
--window TEXT | 24h | How far back the report looks: 90m, 24h, 7d. |
--until TEXT | now | Where the window ends, an ISO time, UTC unless it says otherwise. |
--out FILE | report.html | The file to write; - for standard output. |
ico replay
Tests engine configs on recorded traffic and names the cheapest that holds the latency limits, if it beats the current config by the margin. Run it on the GPU machine of the replica. See Run a replay.
Needs ICO_ADMIN_TOKEN and ICO_CLICKHOUSE_PASSWORD in the environment.
| Option | Default | Meaning |
|---|---|---|
--replica TEXT | required | The replica to drain and test. |
--configs FILE | required | The sweep file. |
--window TEXT | 1h | How far back the recorded traffic is taken from. |
--until TEXT | now | Where that window ends. |
--slo TEXT | from the config | Latency limits, like ttft_p95=500ms,itl_p95=50ms. |
--budget TEXT | 45m | The most the whole run may take, restore included. |
--min-saving TEXT | 10% | How much cheaper than the current config a config must measure to be recommended. |
--single-replica | off | Go ahead although this replica is the only one serving its model. |
--yes | off | Do not wait for a yes after the plan. |
--replica-url TEXT | the replica's url | Where this machine reaches the engine. |
--sample-requests N | 300 | How many consecutive recorded requests the replay is built from. |
--health-timeout TEXT | 15m | The longest an engine may take to come up after a restart. |
--step-seconds N | 15 | How long the load stays at each rate. |
--min-step-requests N | 40 | The fewest requests a rate is judged on, however low the rate. |