Gateway
Proxied endpoints
| Endpoint | What the gateway does |
|---|---|
POST /v1/chat/completions | Forwards to a replica and records the request. |
POST /v1/completions | Forwards and records. |
POST /v1/embeddings | Forwards and records. |
GET /v1/models | Answers from the next replica; no record. |
GET /healthz | Answers ok itself. |
| anything else | 404, with an OpenAI-style error. |
Replicas are chosen round robin among those that serve the request's model and are not drained. A replica that refuses the connection is skipped for that request. When every replica of a model is drained, the gateway answers 503; when none can be reached, 502.
What a request record holds
| Field | Meaning |
|---|---|
request_id, trace_id | The gateway's id for the request, sent upstream as X-Request-Id, and the W3C trace id it starts, sent as traceparent |
ts, replica, model, endpoint | When, where and what |
tenant, feature | From the request headers named in headers |
prompt_tokens, completion_tokens, cached_tokens | From the engine's usage figures |
ttft_ms, e2e_ms | Time to the first streamed token, and to the end of the response |
gap_p50_ms, gap_p95_ms | Median and 95th percentile of the gaps between streamed chunks |
status, stream, max_tokens, prompt_chars | The response status, whether it streamed, the requested limit, and the request text's length |
prefix_hashes | Salted hashes of the request text, block by block |
No prompt or completion text is kept. See Privacy and security.
Streams
Streams are relayed event by event. The gateway turns on stream_options.include_usage upstream, so the engine reports token counts at the end of a stream, and withholds that extra chunk from clients that did not ask for it.
Admin API
On gateway.admin_listen, with the bearer token from ICO_ADMIN_TOKEN. Without a token set, it refuses every call.
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" 127.0.0.1:8081/admin/replicas
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" -X POST 127.0.0.1:8081/admin/replicas/gpu-1-r1/drain
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" -X POST 127.0.0.1:8081/admin/replicas/gpu-1-r1/undrain
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" 127.0.0.1:8081/admin/stats
| Call | Answer |
|---|---|
GET /admin/replicas | Each replica: name, url, host, gpu_model, gpu_count, drained, and inflight, its requests in flight. |
POST /admin/replicas/{name}/drain | The replica, and was_drained. A drained replica gets no new requests; it is idle once inflight reads 0. |
POST /admin/replicas/{name}/undrain | The replica, and was_drained. |
GET /admin/stats | The export: exported, dropped, queued and failed_calls. |
When the collector is down
Records leave the request path at once, into a queue of 16,384 that is sent in batches of up to 512, with retries. When the queue is full, new records are dropped and counted in dropped. Requests never wait for the collector.
Running it
gateway -config /etc/ico/config.yaml
| Variable | Meaning |
|---|---|
ICO_CONFIG | The config file, if -config is not given. |
ICO_ADMIN_TOKEN | Turns on the admin API. |
ICO_HASH_SALT | The salt of the prefix hashes. Without it, the gateway makes one in state_dir on first start. |