Quickstart
This runs the whole of ico on one machine with Docker, with two fake engines standing in for vLLM. It shows what the gateway records and what the report looks like before you install anything next to a real GPU.
You need Docker with Compose and git.
ico, so its last step would delete a real install's data. To be safe anyway, add -p ico-quickstart to every docker compose command below.Get the code
git clone --branch v0.1.1 https://github.com/Gudjon01/inference-cost-optimizer.git
cd inference-cost-optimizer/deploy
Set the two secrets
cp .env.example .env
Set ICO_ADMIN_TOKEN and ICO_CLICKHOUSE_PASSWORD in .env to long random strings, for example the output of openssl rand -hex 24. The first unlocks the gateway's admin API, the second is the password of ClickHouse's ico user.
Start the stack with fake engines
ICO_CONFIG_FILE=./config.mock.yaml docker compose --profile mock up -d
This starts the gateway on port 8080, the collector, ClickHouse, and two fake engines, mock-1 and mock-2. The first start downloads the images, which takes a few minutes.
Send some traffic
Send requests through the gateway, tagged with a tenant and a feature:
for i in $(seq 1 200); do
tenant=$([ $((i % 3)) -eq 0 ] && echo globex || echo acme)
curl -s http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-Tenant-Id: $tenant" -H "X-Feature: chat" \
-d '{"model": "mock-model", "max_tokens": 16,
"messages": [{"role": "user", "content": "Summarize our refund policy."}]}' > /dev/null
done
The fake engines take about a third of a second per request, so this runs for a little over a minute. The gateway spreads the requests over both engines.
Write the report
docker compose exec collector ico report --window 1h --out /tmp/report.html
docker compose cp collector:/tmp/report.html report.html
On Windows, in Git Bash, put MSYS_NO_PATHCONV=1 before both commands, so /tmp/report.html is not rewritten as a Windows path.
Open report.html in a browser. It is one file with no scripts and nothing loaded from elsewhere. The fake engines report no GPU counters, so it measures idle time from request timing; see How much it can see.
Look at the raw records
Every request the gateway proxied is one row in ClickHouse:
docker compose exec clickhouse sh -c 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" -d ico -q "SELECT ts, replica, tenant, prompt_tokens, completion_tokens, ttft_ms, e2e_ms FROM requests ORDER BY ts DESC LIMIT 10"'
Clean up
docker compose --profile mock down -v
-v also deletes the ClickHouse volume and everything recorded in it.
Next
When you are ready for real GPUs, start with Install on rented GPUs.