Check the install

Send some traffic through the gateway, then check each source in turn. The commands run on the ico host from deploy/, unless they say otherwise.

The gateway exports its records

curl -s -H "Authorization: Bearer $ICO_ADMIN_TOKEN" 127.0.0.1:8081/admin/stats

{"exported": 1520, "dropped": 0, "queued": 0, "failed_calls": 0}

exported grows with traffic. dropped or failed_calls above zero mean the gateway cannot reach the collector at the address in gateway.collector.

The collector scrapes the engines and the GPUs

docker compose logs collector | grep -E 'telemetry level|scraping again|scrape failed'

For each exporter, look for a line like:

dcgm:gpu-1: reports dram_active_ratio, memory_total_bytes, ..., sm_active_ratio, tensor_active_ratio, util_ratio at telemetry level L3

scrape failed that does not go away means the ico host cannot reach that URL: check the address in the config, the port the engine or exporter publishes, and the firewall between them.

Requests and their engine traces arrive

docker compose exec clickhouse sh -c 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" -d ico -q "
  SELECT replica, count() AS requests, round(avg(has_span) * 100) AS traced_percent
  FROM request_spans WHERE ts > now() - INTERVAL 1 HOUR GROUP BY replica"'

Every replica should show requests, and traced_percent near 100.

Write a first report

docker compose exec collector ico report --window 1h --out /tmp/report.html
docker compose cp collector:/tmp/report.html report.html

See Read the report. After a day of traffic, the same command with --window 24h gives the first report worth acting on.

When something is missing