ico
Install on rented GPUs

Set up the GPU machines

On every GPU machine, turn on tracing in vLLM, run a GPU exporter, and run the gateway.

Do this on every GPU machine. The examples use the addresses of the ico host page: the ico host at 10.0.0.5, and this machine, gpu-1, at 10.0.0.11 with one vLLM replica named gpu-1-r1.

Start by getting the code; the steps below use files from it:

gpu-1
git clone --branch v0.1.1 https://github.com/Gudjon01/inference-cost-optimizer.git

1. Turn on tracing in vLLM

vLLM sends one OpenTelemetry span per request, with the time it spent queued, in prefill and in decode. Add one flag and two environment variables to the way you start it:

docker run -d --name vllm --gpus all -p 10.0.0.11:8000:8000 \
  -e OTEL_SERVICE_NAME=gpu-1-r1 \
  -e OTEL_EXPORTER_OTLP_TRACES_INSECURE=true \
  vllm/vllm-openai:v0.31.0 \
  --model Qwen/Qwen3-8B \
  --otlp-traces-endpoint grpc://10.0.0.5:4317
  • OTEL_SERVICE_NAME must be the replica's name in the config. The collector files each span under it.
  • OTEL_EXPORTER_OTLP_TRACES_INSECURE=true: the collector's port is plain gRPC, on your private network.
  • Keep your other flags as they are.

If vLLM exits saying that OpenTelemetry is not available, its environment lacks the packages. Add them:

FROM vllm/vllm-openai:v0.31.0
RUN pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp
vLLM 0.31 puts prefill and decode time in its spans. Older versions, such as 0.10, leave them out; the report then splits costs across tenants by token counts instead.

2. Run a GPU exporter

The collector reads the GPUs through NVIDIA's own exporters; ico's containers never touch the GPU driver.

For H100, H200, A100, L40S, L4 and other datacenter cards, run NVIDIA's dcgm-exporter with the counters file from ico's repository:

gpu-1
docker run -d --name dcgm-exporter --restart unless-stopped \
  --gpus all --cap-add SYS_ADMIN \
  -p 10.0.0.11:9400:9400 \
  -v "$PWD/inference-cost-optimizer/deploy/dcgm-counters.csv:/etc/dcgm-exporter/ico-counters.csv:ro" \
  nvcr.io/nvidia/k8s/dcgm-exporter:4.5.2-4.8.1-ubuntu22.04 \
  -f /etc/dcgm-exporter/ico-counters.csv

SYS_ADMIN lets it read the profiling counters, which tell busy cores from idle ones (telemetry level L3). Some hosts allow them without it.

Check that the profiling counters are there, about 30 seconds after the start:

gpu-1
curl -s 10.0.0.11:9400/metrics | grep -E '^DCGM_FI_PROF_(SM_ACTIVE|PIPE_TENSOR_ACTIVE|DRAM_ACTIVE)'

Three lines mean telemetry level L3. None means the host keeps them closed; ico still works, at level L2.

3. Run the gateway

The gateway has its own config file on each machine, listing only the replicas it sends requests to:

gateway.yaml
replicas:
  - name: gpu-1-r1                # the same name as in the ico host's config
    url: http://10.0.0.11:8000    # how the gateway reaches the engine
    host: gpu-1
    gpu_model: H100-SXM
    gpu_count: 1

headers:
  tenant: X-Tenant-Id             # see Send traffic through the gateway
  feature: X-Feature

gateway:
  listen: "10.0.0.11:8080"        # where your router or clients reach it
  admin_listen: "127.0.0.1:8081"  # drain and undrain; keep it local
  admin_url: http://127.0.0.1:8081 # where ico replay on this machine finds it
  collector: 10.0.0.5:4317
  state_dir: /var/lib/ico

Start it next to vLLM:

gpu-1
docker run -d --name ico-gateway --restart unless-stopped --network host \
  -e ICO_ADMIN_TOKEN="<the token from the ico host's .env>" \
  -e ICO_HASH_SALT="<the same salt on every gateway>" \
  -v "$PWD/gateway.yaml:/etc/ico/config.yaml:ro" \
  -v ico-gateway-state:/var/lib/ico \
  ghcr.io/gudjon01/ico-gateway:0.1.1
  • ICO_ADMIN_TOKEN turns on the admin API, which ico replay uses to drain this replica. Without it the API refuses every call.
  • ICO_HASH_SALT keys the hashes of request text. Give every gateway the same one, made once with openssl rand -hex 32, so that a prompt prefix seen on two machines hashes the same. Treat it as a secret. Without it, each gateway makes its own salt in state_dir.

Next

Send traffic through the gateway.

Set up the ico host

ClickHouse and the collector on one small machine, with Docker Compose.

Send traffic through the gateway

Point your router or clients at the gateway, and tell it who each request is for.

Copyright © 2026