Reference
Stored data
The ClickHouse tables and views ico writes and reads, and the names it stores metrics under.
Everything lives in one ClickHouse database, ico by default. Raw rows expire after 30 days, in whole days; replay results are kept until you delete them.
Tables
| Table | Written from | One row per | Kept |
|---|---|---|---|
requests | the gateways | request the gateway proxied | 30 days |
engine_spans | the engines' OpenTelemetry traces | engine span: queue, prefill, decode and end-to-end time, token counts | 30 days |
engine_metrics | each engine's /metrics | scraped sample | 30 days |
gpu_metrics | the GPU exporters | scraped sample, with its telemetry level | 30 days |
replay_runs | ico replay | replay run: replica, window, limits, budget, margin, status | always |
replay_results | ico replay | config tested in a run, at each stage | always |
Views
| View | Gives |
|---|---|
request_spans | Every request with its engine span, joined on trace id; has_span is false where the engine sent none. |
usage_by_tenant(since, until) | Requests, errors and tokens per replica, model and tenant. |
replica_busy(since, until) | Per replica, the seconds with at least one request in flight, and the span of time its requests cover. |
replica_phases(since, until) | Per replica, engine time in prefill and decode, with the tokens of the same requests. |
replica_engine(since, until) | Per replica, when its engine was first and last heard from, and the model it reports. |
host_gpus(since, until) | Per host, its GPUs, telemetry level and the window's average of each counter. |
The parameterized views are queried like functions:
SELECT * FROM host_gpus(since = '2026-10-14 00:00:00', until = '2026-10-15 00:00:00')
Metric names
Engine and GPU metrics are stored under ico's own names and units, so queries and reports read the same across vLLM versions and exporters.
GPU metrics
| Name | Unit | From dcgm-exporter | From nvidia_gpu_exporter | Level |
|---|---|---|---|---|
sm_active_ratio | 0 to 1 | DCGM_FI_PROF_SM_ACTIVE | L3 | |
sm_occupancy_ratio | 0 to 1 | DCGM_FI_PROF_SM_OCCUPANCY | L3 | |
tensor_active_ratio | 0 to 1 | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | L3 | |
dram_active_ratio | 0 to 1 | DCGM_FI_PROF_DRAM_ACTIVE | L3 | |
util_ratio | 0 to 1 | DCGM_FI_DEV_GPU_UTIL | nvidia_smi_utilization_gpu_ratio | L2 |
memory_used_bytes | bytes | DCGM_FI_DEV_FB_USED | nvidia_smi_memory_used_bytes | L1 |
memory_total_bytes | bytes | DCGM_FI_DEV_FB_TOTAL | nvidia_smi_memory_total_bytes | L1 |
power_watts | watts | DCGM_FI_DEV_POWER_USAGE | nvidia_smi_power_draw_watts | L1 |
Engine metrics
| Name | What it counts |
|---|---|
requests_running, requests_waiting | Requests in the engine's batch, and queued for it |
kv_cache_usage | Share of the KV cache in use |
prompt_tokens_total, generation_tokens_total | Tokens processed and generated, cumulative |
prefix_cache_queries_total, prefix_cache_hits_total | Prefix cache lookups and hits, cumulative |
preemptions_total | Requests evicted to free KV cache, cumulative |
ttft_seconds, inter_token_latency_seconds, queue_time_seconds, prefill_time_seconds, decode_time_seconds | Histograms, stored as _bucket, _sum and _count |
Each maps every name vLLM has used for it, from 0.10 to 0.31.