Install on rented GPUs

Overview

What runs where on a fleet of rented GPU machines, and what has to reach what.

This section installs ico on GPU machines you rent, from RunPod, Lambda, CoreWeave or any other provider, or run in your own racks, with vLLM already serving on them. It assumes Docker on every machine; where a provider gives you a container without Docker, the same programs run as plain processes.

What runs where

Machine	What runs there	Why there
The ico host: one small machine without a GPU	The collector and ClickHouse	They need CPU, memory and disk, not a GPU. Keeping them off the GPU machines means they never take anything from your engines.
Every GPU machine	Your vLLM replicas (already there), the ico gateway, a GPU exporter	The exporter reads the local GPUs. The gateway times every request, so it sits as close to the engine as possible.

For the ico host, 2 vCPUs, 8 GB of memory and 50 GB of disk are a good start for a fleet of a few dozen GPUs. It can be any machine your GPU machines can reach: a small VM at the same provider, ideally in the same region.

The replay lab (ico replay) runs on the GPU machine of the replica it tests, and only while it runs. See The replay lab.

Where the gateway goes

Your clients or your router have to send requests to an ico gateway instead of to vLLM. There are two ways to place it.

Use this when you already have a router or load balancer in front of your replicas.

clients ──▶ your router ──▶ gateway on gpu-1 ──▶ vLLM on gpu-1
                        └──▶ gateway on gpu-2 ──▶ vLLM on gpu-2

Your router keeps doing what it does; each of its upstreams moves from vLLM's port to the gateway's port on the same machine. Every gateway knows only the replicas on its own machine.

Use this when clients talk to vLLM directly and you have no router yet.

clients ──▶ gateway ──┬──▶ vLLM on gpu-1
                      └──▶ vLLM on gpu-2

The gateway becomes your load balancer: round robin across the replicas of each model, skipping drained ones and any that refuse the connection. Run it on one of the GPU machines or on the ico host.

Do not put the gateway in front of an existing router. It would see only the router, so it could not tell which replica served a request, and costs could no longer be split by replica.

What has to reach what

From	To	Port	What crosses
Clients or your router	Gateway	8080	Your users' requests
Gateway	vLLM	8000	The same requests, forwarded
Gateway	Collector on the ico host	4317	One record per request (gRPC)
vLLM	Collector on the ico host	4317	One trace span per request (OpenTelemetry over gRPC)
Collector	vLLM on every GPU machine	8000	`/metrics`, every 5 seconds
Collector	GPU exporter on every GPU machine	9400 (dcgm-exporter) or 9835 (nvidia_gpu_exporter)	GPU counters, every 5 seconds
`ico replay` on a GPU machine	ClickHouse on the ico host	8123	Recorded traffic in, results out

Keep all of these on a private network. The gateway's admin API (port 8081) needs only to be reached from the machine it runs on.

The collector accepts writes without authentication. Publish port 4317 only on an interface your GPU machines reach privately, or behind a firewall that admits only them.

Machines without a private network

Some providers give each machine a public IP and nothing else; a RunPod pod, for example, is a container that is reached over SSH. There, one SSH tunnel from the ico host to each GPU machine carries everything, in both directions:

On the ico host

ssh -N -p <ssh port> root@<gpu machine> \
  -L 127.0.0.1:18000:127.0.0.1:8000 \
  -L 127.0.0.1:19400:127.0.0.1:9400 \
  -R 127.0.0.1:4317:127.0.0.1:4317 \
  -R 127.0.0.1:8123:127.0.0.1:8123

The gateway, vLLM and ico replay on the GPU machine then find the collector and ClickHouse at 127.0.0.1, and everything on the GPU machine can listen on 127.0.0.1 only. On the ico host, keep Compose's default ports, published on 127.0.0.1: skip the override file of Set up the ico host, since the tunnel is the only way in. The collector scrapes vLLM and the exporter at the tunnel's local ends; give each GPU machine's tunnel its own pair of local ports (18000 and 19400, 18001 and 19401, and so on).

The collector runs in a container, so it reaches the tunnel's local ends at host.docker.internal, for example http://host.docker.internal:18000:

With Docker Desktop that works as it is.
On a Linux host, bind the two -L forwards to Docker's bridge address instead of 127.0.0.1 (-L 172.17.0.1:18000:127.0.0.1:8000), and map the name for the collector in docker-compose.override.yml:
deploy/docker-compose.override.yml
```
services:
  collector:
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

Each request in flight through a tunnel holds an open file of the GPU machine's SSH server. Raise that server's limit before heavy traffic, for example with prlimit --pid $(cat /run/sshd.pid) --nofile=65536:65536, then connect. With the default of 1,024, a tunnel refuses new connections at about a thousand requests in flight.

A first report in ten minutes on your laptop, with two fake engines and no GPU.

Set up the ico host

ClickHouse and the collector on one small machine, with Docker Compose.

Overview

What runs where

Where the gateway goes

What has to reach what

Machines without a private network

The steps

Set up the ico host

Set up every GPU machine

Send traffic through the gateway

Check that everything arrives