What your self-hosted LLMs really cost

ico, the inference cost optimizer, measures what each customer you serve from your own GPUs costs, shows how much of the bill pays for idle time, and tests engine configs on a replay of your own traffic to find the one that serves it for less.

Runs on your machines. No prompt or completion ever leaves them.

Get started Install on rented GPUs

Three questions, answered from your own traffic

What does each customer cost?

The gateway tags every request with the tenant it served. The report splits each GPU's cost across tenants by the engine time their requests took, prefill and decode weighed apart.

How much pays for idle GPUs?

Rented GPUs bill every hour. ico counts the hours your engines had no request to serve, and prices them. Where the card has DCGM's profiling counters, it also shows what you paid for cores that sat unused while serving.

Which config costs less?

The replay lab restarts one replica under other engine flags, replays the shape of your recorded traffic, and names the config that holds your latency limits for clearly less than the current one. Then it puts everything back.

Nothing leaves your network

The gateway stores token counts and timings, never text. Replays use synthetic prompts rebuilt from lengths and salted prefix hashes.

Out of the way of your users

The gateway adds about 0.2 ms per request. When the collector is down it keeps serving and drops telemetry, never requests.

Install it yourself, or with us

On rented machines with Docker, installing takes an afternoon, and the first report is worth reading after a day of traffic. If you would rather not do it alone, we install it with you.