Introduction
ico, the inference cost optimizer, finds out what your self-hosted LLMs cost per million tokens and per customer, how much of that cost pays for GPUs with nothing to do, and which engine config serves the same traffic for less. It runs on your machines, next to your engines, and reads your real traffic.
Who it is for
Teams that serve open-weight or fine-tuned models from their own or rented GPUs, usually to many customers from the same replicas, and that want to know:
- what each customer costs to serve, to price them or to see the margin on them;
- how much of the GPU bill pays for idle time;
- whether their engine flags are leaving capacity on the table.
ico is built first for GPU machines you rent or own, from RunPod, Lambda, CoreWeave or your own racks, running vLLM in Docker or as plain processes. Kubernetes comes later.
What you get
What it works with today
| Supported | Run on real hardware | |
|---|---|---|
| Engine | vLLM | 0.31.0 on an H100 SXM; 0.10.1.1 on a GeForce RTX 2060 |
| GPU telemetry | NVIDIA dcgm-exporter (datacenter GPUs), nvidia_gpu_exporter (GeForce) | dcgm-exporter 4.5.2-4.8.1 on the H100, with profiling counters; nvidia_gpu_exporter 1.15.1 on the RTX 2060 |
| Deployment | Docker Compose, or plain processes | RunPod (H100), a Windows desktop with Docker Desktop (RTX 2060) |
Not built yet: SGLang and AMD support, Kubernetes and a Helm chart, replays that mirror live requests, and a check of answer quality. Until that check exists, configs that change precision, such as FP8, are measured but cannot be recommended.