---
title: "Gateway"
description: "What the gateway proxies, records and exposes."
canonical_url: "https://inference-cost-optimizer.com/reference/gateway"
---
# Gateway

> What the gateway proxies, records and exposes.

## Proxied endpoints

<table>
<thead>
  <tr>
    <th>
      Endpoint
    </th>
    
    <th>
      What the gateway does
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        POST /v1/chat/completions
      </code>
    </td>
    
    <td>
      Forwards to a replica and records the request.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        POST /v1/completions
      </code>
    </td>
    
    <td>
      Forwards and records.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        POST /v1/embeddings
      </code>
    </td>
    
    <td>
      Forwards and records.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        GET /v1/models
      </code>
    </td>
    
    <td>
      Answers from the next replica; no record.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        GET /healthz
      </code>
    </td>
    
    <td>
      Answers <code>
        ok
      </code>
      
       itself.
    </td>
  </tr>
  
  <tr>
    <td>
      anything else
    </td>
    
    <td>
      <code>
        404
      </code>
      
      , with an OpenAI-style error.
    </td>
  </tr>
</tbody>
</table>

Replicas are chosen round robin among those that serve the request's model and are not drained. A replica that refuses the connection is skipped for that request. When every replica of a model is drained, the gateway answers `503`; when none can be reached, `502`.

## What a request record holds

<table>
<thead>
  <tr>
    <th>
      Field
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        request_id
      </code>
      
      , <code>
        trace_id
      </code>
    </td>
    
    <td>
      The gateway's id for the request, sent upstream as <code>
        X-Request-Id
      </code>
      
      , and the W3C trace id it starts, sent as <code>
        traceparent
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ts
      </code>
      
      , <code>
        replica
      </code>
      
      , <code>
        model
      </code>
      
      , <code>
        endpoint
      </code>
    </td>
    
    <td>
      When, where and what
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        tenant
      </code>
      
      , <code>
        feature
      </code>
    </td>
    
    <td>
      From the request headers named in <code>
        headers
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        prompt_tokens
      </code>
      
      , <code>
        completion_tokens
      </code>
      
      , <code>
        cached_tokens
      </code>
    </td>
    
    <td>
      From the engine's usage figures
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ttft_ms
      </code>
      
      , <code>
        e2e_ms
      </code>
    </td>
    
    <td>
      Time to the first streamed token, and to the end of the response
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        gap_p50_ms
      </code>
      
      , <code>
        gap_p95_ms
      </code>
    </td>
    
    <td>
      Median and 95th percentile of the gaps between streamed chunks
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        status
      </code>
      
      , <code>
        stream
      </code>
      
      , <code>
        max_tokens
      </code>
      
      , <code>
        prompt_chars
      </code>
    </td>
    
    <td>
      The response status, whether it streamed, the requested limit, and the request text's length
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        prefix_hashes
      </code>
    </td>
    
    <td>
      Salted hashes of the request text, block by block
    </td>
  </tr>
</tbody>
</table>

No prompt or completion text is kept. See [Privacy and security](https://inference-cost-optimizer.com/reference/security).

## Streams

Streams are relayed event by event. The gateway turns on `stream_options.include_usage` upstream, so the engine reports token counts at the end of a stream, and withholds that extra chunk from clients that did not ask for it.

## Admin API

On `gateway.admin_listen`, with the bearer token from `ICO_ADMIN_TOKEN`. Without a token set, it refuses every call.

```bash
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" 127.0.0.1:8081/admin/replicas
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" -X POST 127.0.0.1:8081/admin/replicas/gpu-1-r1/drain
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" -X POST 127.0.0.1:8081/admin/replicas/gpu-1-r1/undrain
curl -H "Authorization: Bearer $ICO_ADMIN_TOKEN" 127.0.0.1:8081/admin/stats
```

<table>
<thead>
  <tr>
    <th>
      Call
    </th>
    
    <th>
      Answer
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        GET /admin/replicas
      </code>
    </td>
    
    <td>
      Each replica: <code>
        name
      </code>
      
      , <code>
        url
      </code>
      
      , <code>
        host
      </code>
      
      , <code>
        gpu_model
      </code>
      
      , <code>
        gpu_count
      </code>
      
      , <code>
        drained
      </code>
      
      , and <code>
        inflight
      </code>
      
      , its requests in flight.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        POST /admin/replicas/{name}/drain
      </code>
    </td>
    
    <td>
      The replica, and <code>
        was_drained
      </code>
      
      . A drained replica gets no new requests; it is idle once <code>
        inflight
      </code>
      
       reads 0.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        POST /admin/replicas/{name}/undrain
      </code>
    </td>
    
    <td>
      The replica, and <code>
        was_drained
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        GET /admin/stats
      </code>
    </td>
    
    <td>
      The export: <code>
        exported
      </code>
      
      , <code>
        dropped
      </code>
      
      , <code>
        queued
      </code>
      
       and <code>
        failed_calls
      </code>
      
      .
    </td>
  </tr>
</tbody>
</table>

## When the collector is down

Records leave the request path at once, into a queue of 16,384 that is sent in batches of up to 512, with retries. When the queue is full, new records are dropped and counted in `dropped`. Requests never wait for the collector.

## Running it

```bash
gateway -config /etc/ico/config.yaml
```

<table>
<thead>
  <tr>
    <th>
      Variable
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        ICO_CONFIG
      </code>
    </td>
    
    <td>
      The config file, if <code>
        -config
      </code>
      
       is not given.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_ADMIN_TOKEN
      </code>
    </td>
    
    <td>
      Turns on the admin API.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ICO_HASH_SALT
      </code>
    </td>
    
    <td>
      The salt of the prefix hashes. Without it, the gateway makes one in <code>
        state_dir
      </code>
      
       on first start.
    </td>
  </tr>
</tbody>
</table>


## Sitemap

See the full [sitemap](https://inference-cost-optimizer.com/sitemap.md) for all pages.