---
title: "Check the install"
description: "Make sure every part reports, and what to do when one does not."
canonical_url: "https://inference-cost-optimizer.com/install/check"
---
# Check the install

> Make sure every part reports, and what to do when one does not.

Send some traffic through the gateway, then check each source in turn. The commands run on the ico host from `deploy/`, unless they say otherwise.

## The gateway exports its records

```bash [GPU machine]
curl -s -H "Authorization: Bearer $ICO_ADMIN_TOKEN" 127.0.0.1:8081/admin/stats
```

```json
{"exported": 1520, "dropped": 0, "queued": 0, "failed_calls": 0}
```

`exported` grows with traffic. `dropped` or `failed_calls` above zero mean the gateway cannot reach the collector at the address in `gateway.collector`.

## The collector scrapes the engines and the GPUs

```bash [ico host]
docker compose logs collector | grep -E 'telemetry level|scraping again|scrape failed'
```

For each exporter, look for a line like:

```text
dcgm:gpu-1: reports dram_active_ratio, memory_total_bytes, ..., sm_active_ratio, tensor_active_ratio, util_ratio at telemetry level L3
```

`scrape failed` that does not go away means the ico host cannot reach that URL: check the address in the config, the port the engine or exporter publishes, and the firewall between them.

## Requests and their engine traces arrive

```bash [ico host]
docker compose exec clickhouse sh -c 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" -d ico -q "
  SELECT replica, count() AS requests, round(avg(has_span) * 100) AS traced_percent
  FROM request_spans WHERE ts > now() - INTERVAL 1 HOUR GROUP BY replica"'
```

Every replica should show requests, and `traced_percent` near 100.

## Write a first report

```bash [ico host]
docker compose exec collector ico report --window 1h --out /tmp/report.html
docker compose cp collector:/tmp/report.html report.html
```

See [Read the report](https://inference-cost-optimizer.com/report/read-the-report). After a day of traffic, the same command with `--window 24h` gives the first report worth acting on.

## When something is missing

<accordion>
<accordion-item icon="i-lucide-circle-help" label="No requests recorded at all">

Traffic does not go through the gateway yet, or the gateway cannot reach the collector. Check that requests arrive at the gateway's port, then `/admin/stats` above.

</accordion-item>

<accordion-item icon="i-lucide-circle-help" label="Requests recorded, but traced_percent is 0">

vLLM sends no spans, or files them under another name. Check that vLLM was restarted with `--otlp-traces-endpoint`, that `OTEL_SERVICE_NAME` is exactly the replica's name in the config, and that `OTEL_EXPORTER_OTLP_TRACES_INSECURE=true` is set. vLLM's own log names the problem at startup.

</accordion-item>

<accordion-item icon="i-lucide-circle-help" label="Telemetry level L2 on a datacenter GPU">

The profiling counters are closed. Give dcgm-exporter the `SYS_ADMIN` capability; on some providers they stay closed regardless. ico still works: idle time comes from the gateway's requests either way, and only the figure for cores unused while serving is missing.

</accordion-item>

<accordion-item icon="i-lucide-circle-help" label="The gateway will not start: address already in use">

Something else holds the port. Some provider images run their own services, for example nginx on 8081 on RunPod. Move `gateway.listen` or `gateway.admin_listen` to a free port, and `gateway.admin_url` with it.

</accordion-item>

<accordion-item icon="i-lucide-circle-help" label="dropped grows on the gateway">

The collector is unreachable, or was for a while. Requests are unaffected; records made while the gateway's queue of 16,384 was full are lost, and the report undercounts that stretch.

</accordion-item>
</accordion>


## Sitemap

See the full [sitemap](https://inference-cost-optimizer.com/sitemap.md) for all pages.