---
title: "Set up the GPU machines"
description: "On every GPU machine, turn on tracing in vLLM, run a GPU exporter, and run the gateway."
canonical_url: "https://inference-cost-optimizer.com/install/gpu-machines"
---
# Set up the GPU machines
> On every GPU machine, turn on tracing in vLLM, run a GPU exporter, and run the gateway.
Do this on every GPU machine. The examples use the addresses of the [ico host page](https://inference-cost-optimizer.com/install/ico-host): the ico host at `10.0.0.5`, and this machine, `gpu-1`, at `10.0.0.11` with one vLLM replica named `gpu-1-r1`.
Start by getting the code; the steps below use files from it:
```bash [gpu-1]
git clone --branch v0.1.1 https://github.com/Gudjon01/inference-cost-optimizer.git
```
## 1. Turn on tracing in vLLM
vLLM sends one OpenTelemetry span per request, with the time it spent queued, in prefill and in decode. Add one flag and two environment variables to the way you start it:
```bash [docker run]
docker run -d --name vllm --gpus all -p 10.0.0.11:8000:8000 \
-e OTEL_SERVICE_NAME=gpu-1-r1 \
-e OTEL_EXPORTER_OTLP_TRACES_INSECURE=true \
vllm/vllm-openai:v0.31.0 \
--model Qwen/Qwen3-8B \
--otlp-traces-endpoint grpc://10.0.0.5:4317
```
```bash [vllm serve]
OTEL_SERVICE_NAME=gpu-1-r1 \
OTEL_EXPORTER_OTLP_TRACES_INSECURE=true \
vllm serve Qwen/Qwen3-8B --host 0.0.0.0 --port 8000 \
--otlp-traces-endpoint grpc://10.0.0.5:4317
```
- `OTEL_SERVICE_NAME` must be the replica's name in the config. The collector files each span under it.
- `OTEL_EXPORTER_OTLP_TRACES_INSECURE=true`: the collector's port is plain gRPC, on your private network.
- Keep your other flags as they are.
If vLLM exits saying that OpenTelemetry is not available, its environment lacks the packages. Add them:
```dockerfile [Dockerfile]
FROM vllm/vllm-openai:v0.31.0
RUN pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp
```
```bash [pip]
pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp
```
vLLM 0.31 puts prefill and decode time in its spans. Older versions, such as 0.10, leave them out; the report then splits costs across tenants by token counts instead.
## 2. Run a GPU exporter
The collector reads the GPUs through NVIDIA's own exporters; ico's containers never touch the GPU driver.
For H100, H200, A100, L40S, L4 and other datacenter cards, run NVIDIA's dcgm-exporter with the counters file from ico's repository:
```bash [gpu-1]
docker run -d --name dcgm-exporter --restart unless-stopped \
--gpus all --cap-add SYS_ADMIN \
-p 10.0.0.11:9400:9400 \
-v "$PWD/inference-cost-optimizer/deploy/dcgm-counters.csv:/etc/dcgm-exporter/ico-counters.csv:ro" \
nvcr.io/nvidia/k8s/dcgm-exporter:4.5.2-4.8.1-ubuntu22.04 \
-f /etc/dcgm-exporter/ico-counters.csv
```
`SYS_ADMIN` lets it read the profiling counters, which tell busy cores from idle ones (telemetry level L3). Some hosts allow them without it.
GeForce cards have no DCGM support. Run nvidia_gpu_exporter:
```bash [gpu-1]
docker run -d --name nvidia-gpu-exporter --restart unless-stopped \
--gpus all -p 10.0.0.11:9835:9835 \
utkuozdemir/nvidia_gpu_exporter:1.15.1
```
In the ico host's config, list it with `kind: nvidia_smi` and `url: http://10.0.0.11:9835/metrics`.
Check that the profiling counters are there, about 30 seconds after the start:
```bash [gpu-1]
curl -s 10.0.0.11:9400/metrics | grep -E '^DCGM_FI_PROF_(SM_ACTIVE|PIPE_TENSOR_ACTIVE|DRAM_ACTIVE)'
```
Three lines mean telemetry level L3. None means the host keeps them closed; ico still works, at level L2.
## 3. Run the gateway
The gateway has its own config file on each machine, listing only the replicas it sends requests to:
```yaml [gateway.yaml]
replicas:
- name: gpu-1-r1 # the same name as in the ico host's config
url: http://10.0.0.11:8000 # how the gateway reaches the engine
host: gpu-1
gpu_model: H100-SXM
gpu_count: 1
headers:
tenant: X-Tenant-Id # see Send traffic through the gateway
feature: X-Feature
gateway:
listen: "10.0.0.11:8080" # where your router or clients reach it
admin_listen: "127.0.0.1:8081" # drain and undrain; keep it local
admin_url: http://127.0.0.1:8081 # where ico replay on this machine finds it
collector: 10.0.0.5:4317
state_dir: /var/lib/ico
```
Start it next to vLLM:
```bash [gpu-1]
docker run -d --name ico-gateway --restart unless-stopped --network host \
-e ICO_ADMIN_TOKEN="" \
-e ICO_HASH_SALT="" \
-v "$PWD/gateway.yaml:/etc/ico/config.yaml:ro" \
-v ico-gateway-state:/var/lib/ico \
ghcr.io/gudjon01/ico-gateway:0.1.1
```
The gateway is one static binary. Take it from the release, for `amd64` or `arm64`, and check it against the release's checksums:
```bash [gpu-1]
curl -fLO https://github.com/Gudjon01/inference-cost-optimizer/releases/download/v0.1.1/ico-gateway-linux-amd64
curl -fLO https://github.com/Gudjon01/inference-cost-optimizer/releases/download/v0.1.1/SHA256SUMS
sha256sum --check --ignore-missing SHA256SUMS
install -m 755 ico-gateway-linux-amd64 gateway
ICO_ADMIN_TOKEN="" \
ICO_HASH_SALT="" \
nohup ./gateway -config gateway.yaml > gateway.log 2>&1 &
```
`state_dir` in `gateway.yaml` must be a directory the gateway can write to. Nothing restarts it for you: start it again the same way after the machine restarts.
- `ICO_ADMIN_TOKEN` turns on the admin API, which `ico replay` uses to drain this replica. Without it the API refuses every call.
- `ICO_HASH_SALT` keys the hashes of request text. Give every gateway the same one, made once with `openssl rand -hex 32`, so that a prompt prefix seen on two machines hashes the same. Treat it as a secret. Without it, each gateway makes its own salt in `state_dir`.
## Next
[Send traffic through the gateway](https://inference-cost-optimizer.com/install/send-traffic).
## Sitemap
See the full [sitemap](https://inference-cost-optimizer.com/sitemap.md) for all pages.