How it works

ico has four parts. Only the gateway sits in the path of your users' requests. The rest reads what the gateway, the engines and the GPUs report, and stays out of the way.

 your apps or router
         │  OpenAI-compatible requests
         ▼
 ┌──────────────┐  one record      ┌──────────────┐        ┌────────────┐
 │ ico gateway  │ ───────────────▶ │  collector   │ ─────▶ │ ClickHouse │
 └──────┬───────┘  per request     └──────────────┘        └─────┬──────┘
        │                            ▲   ▲   ▲                    │
        ▼                   traces   │   │   │ /metrics           ├──▶ ico report
 ┌──────────────┐  ──────────────────┘   │   │ every 5 s          │
 │ vLLM replica │  ──────────────────────┘   │                    └──▶ ico replay
 └──────────────┘                            │                          │
 ┌──────────────┐   GPU counters, every 5 s  │      restarts one replica│
 │ GPU exporter │  ──────────────────────────┘      under other flags ◀─┘
 └──────────────┘

Where each part runs

Only what has to be next to a GPU runs on a GPU machine. The collector and ClickHouse need CPU, memory and disk, so they run elsewhere and never take anything from your engines. The gateway stays next to the engine because it times every request, and the replay because it measures latency. See Install on rented GPUs.

The gateway

An OpenAI-compatible pass-through, written in Go, in front of your engine replicas. Your clients or your router send requests to it instead of to the engine. It forwards them unchanged, streams included, and writes one record per request:

• who it was for: the tenant and the feature, from two request headers;
• what it was: the model, the endpoint, prompt, completion and cached token counts;
• how it went: time to first token, end-to-end time, the gaps between streamed chunks, the status;
• which earlier requests it shares a prefix with, as salted hashes of 256-character blocks.

It never stores prompt or completion text. It adds about 0.2 ms per request at the median. When the collector is unreachable it keeps serving and queues up to 16,384 records; while the queue is full, new records are dropped and counted. It drops telemetry, never requests.

The collector

A Python service that receives the gateway's records and the engine's OpenTelemetry traces on one gRPC port (4317), scrapes each engine's /metrics page and each GPU exporter every 5 seconds, and writes everything to ClickHouse under its own names and units. A report reads the same whichever vLLM version or exporter produced the numbers.

Engine traces carry the time each request spent queued, in prefill and in decode. Joined to the gateway's record by trace id, they let the report split a GPU's cost across tenants by the work each one caused, not just by token count.

ClickHouse

Holds everything for 30 days: requests, engine spans, engine metrics and GPU metrics. Replay results are kept without a time limit. The report's numbers come from views you can query yourself.

The report and the replay lab

Two commands of the ico CLI:

• ico report turns a time window into one static HTML file: what the GPUs cost, how much of it was idle, and how it splits across models, replicas and tenants. The feature of each request is recorded too, for your own queries; the report does not break costs down by feature yet. See Read the report.
• ico replay drains one replica at the gateway, restarts its engine under each candidate config, replays the shape of your recorded traffic at rising rates until a latency limit breaks, and names the cheapest config that held. Then it restores the original. See The replay lab.

How much it can see

Idle time always comes from the gateway's requests: time with none in flight. What else the report can say depends on what your GPUs report about themselves. Each host gets a telemetry level:

Datacenter cards such as the H100, A100 or L40S reach L3 with dcgm-exporter. GeForce cards reach L2 with nvidia_gpu_exporter.