How costs are counted

What a replica costs

cost = GPUs × price per GPU-hour × hours seen

• GPUs is the replica's gpu_count in the config.
• Price per GPU-hour comes from gpu_price_per_hour, by the replica's gpu_model. Put in what you pay: the on-demand rate, or your contract or reserved rate.
• Hours seen run from the first to the last sign of the replica in the window: a request it served, a scrape of its engine's metrics, or a reading of its host's GPU exporter. Gaps in between are billed. An engine that was down for an hour still had its GPUs rented.

A replica with no sign at all in the window carries no cost, and the report says so.

What part of it was idle

Idle time is time with no request in flight on the replica: paid for, with nothing to serve. The rest is serving time. Cost × serving share is serving cost; the rest is idle cost.

Where the compute cores were active for longer than requests were in flight, as DCGM's profiling counters show, the replica had work the gateway did not see, such as traffic sent straight to the engine. The cores' active time then counts as serving time instead.

Cores unused while serving

Serving is not the same as using the whole GPU. With one small request in flight, most of the compute cores have nothing to run. Where the host reports profiling counters (telemetry level L3) and runs one replica, the report shows this too:

cores unused while serving = serving share − share of time the compute cores were active

The cores' active share is DCGM's SM_ACTIVE counter, averaged over the window. On the H100 run, requests were in flight for 94% of the billed time and the cores were active 85% of it: 6% of the cost was idle, and another 9% paid for cores that had nothing to run while requests were being served. That 9% is room a fuller batch could use, not time with nothing to do.

With several replicas on one host, the exporter's counters cannot be told apart by replica, so this figure is left out for them, and the report says so.

Cost per million tokens

cost per million tokens = cost ÷ (prompt tokens + completion tokens) × 1,000,000

Token counts come from vLLM, through the gateway. The headline's "without idle time" figure divides serving cost by the same tokens: what they would have cost on GPUs that were never idle.

How a replica's cost is split across tenants

Each tenant gets a share of the replica's whole cost, idle time included, in proportion to the work its requests took. The work of a request is its tokens, weighted by what a token costs on that replica:

work = prompt tokens × (prefill ms per prompt token) + completion tokens × (decode ms per completion token)

The milliseconds per token come from vLLM's traces of the same requests on the same replica, over the window. A prompt token, processed in bulk, usually costs far less than a generated token, so a tenant that generates a lot pays more per token than one that sends long prompts.

When the engine sends no traces, or traces without prefill and decode time (older vLLM versions such as 0.10), every token weighs the same. The Tenants split by column of the Replicas section says which was used.

Models are split the same way when one replica serves several.

What is not counted

• CPUs, memory, storage and network of the GPU machines: only GPU-hours are priced.
• The ico host itself.
• Requests that never reached the gateway.