ico
Install on rented GPUs

Set up the ico host

ClickHouse and the collector on one small machine, with Docker Compose.

The ico host runs two containers: ClickHouse, which stores everything, and the collector, which receives telemetry from the GPU machines and writes it to ClickHouse. It needs Docker with Compose and about ten minutes.

Get the code

ico host
git clone --branch v0.1.1 https://github.com/Gudjon01/inference-cost-optimizer.git
cd inference-cost-optimizer/deploy

Set the secrets

ico host
cp .env.example .env

In .env, set ICO_ADMIN_TOKEN and ICO_CLICKHOUSE_PASSWORD to long random strings, such as the output of openssl rand -hex 24. Keep the admin token: every gateway needs the same one for ico replay to drain replicas. The ClickHouse password is the password of the ico user.

Write the config

ico host
cp config.example.yaml config.yaml

The collector reads four things from it: your replicas, and how it reaches each engine's /metrics; your GPU exporters; what your GPUs cost; and where ClickHouse is. For two GPU machines on a private network, with one vLLM replica each:

deploy/config.yaml
replicas:
  - name: gpu-1-r1             # unique; the gateway and vLLM's traces use the same name
    url: http://10.0.0.11:8000 # how the ico host reaches this engine
    host: gpu-1                # matches the exporter's host below
    gpu_model: H100-SXM        # matches a key of gpu_price_per_hour
    gpu_count: 1               # GPUs this replica uses (tensor parallel size)
  - name: gpu-2-r1
    url: http://10.0.0.12:8000
    host: gpu-2
    gpu_model: H100-SXM
    gpu_count: 1

# USD per GPU-hour, what you pay. Idle hours are billed too.
gpu_price_per_hour:
  H100-SXM: 2.99

# For the replay's memory check: what each served model weighs.
models:
  Qwen/Qwen3-8B:
    params: 8_190_735_360
    weight_bytes_per_param: 2      # BF16
    kv_bytes_per_token: 147_456    # 2 x layers(36) x kv_heads(8) x head_dim(128) x 2 bytes

collector:
  grpc_listen: 0.0.0.0:4317
  scrape_interval_s: 5
  gpu_exporters:
    - kind: dcgm                   # nvidia_smi for nvidia_gpu_exporter on GeForce cards
      url: http://10.0.0.11:9400/metrics
      host: gpu-1
    - kind: dcgm
      url: http://10.0.0.12:9400/metrics
      host: gpu-2

clickhouse:
  host: clickhouse                 # the ClickHouse container, inside Compose
  port: 8123
  username: ico
  database: ico

Every key is described in the configuration reference.

Let the GPU machines reach the collector

Compose publishes the collector (4317) and ClickHouse (8123) on 127.0.0.1 only. Publish them on the ico host's private address instead, in an override file Compose reads next to its own:

deploy/docker-compose.override.yml
services:
  collector:
    ports: !override
      - "10.0.0.5:4317:4317"   # the ico host's private address
  clickhouse:
    ports: !override
      - "10.0.0.5:8123:8123"   # for ico replay, which runs on the GPU machines
The collector accepts writes without authentication. Publish it only on an address your GPU machines reach privately. ClickHouse asks for the ico user's password.

Start ClickHouse and the collector

ico host
docker compose up -d clickhouse collector

Compose pulls the collector's image, ghcr.io/gudjon01/ico-collector, at the release you cloned; --build builds it from the checkout instead. The collector applies ClickHouse's schema on its first start. Until the GPU machines are set up, its log says that every scrape failed; that is expected:

ico host
docker compose logs -f collector

Next

Set up the GPU machines.

Overview

What runs where on a fleet of rented GPU machines, and what has to reach what.

Set up the GPU machines

On every GPU machine, turn on tracing in vLLM, run a GPU exporter, and run the gateway.

Copyright © 2026