---
title: "Send traffic through the gateway"
description: "Point your router or clients at the gateway, and tell it who each request is for."
canonical_url: "https://inference-cost-optimizer.com/install/send-traffic"
---
# Send traffic through the gateway

> Point your router or clients at the gateway, and tell it who each request is for.

This is the one change in your request path. The gateway speaks the same OpenAI-compatible API as vLLM, so moving traffic to it is a change of address.

## Point traffic at the gateway

<tabs>
<tabs-item icon="i-lucide-layers" label="Through your router">

Change each upstream of your router from vLLM's port to the gateway's port on the same machine, for example from `10.0.0.11:8000` to `10.0.0.11:8080`. With nginx:

```nginx [nginx.conf]
upstream llm {
  server 10.0.0.11:8080;   # was 10.0.0.11:8000
  server 10.0.0.12:8080;   # was 10.0.0.12:8000
}
```

</tabs-item>

<tabs-item icon="i-lucide-code" label="Clients directly">

Change the base URL your clients use:

```python [client.py]
from openai import OpenAI

client = OpenAI(base_url="http://10.0.0.11:8080/v1", api_key="unused")
```

</tabs-item>
</tabs>

The gateway proxies these endpoints and answers everything else with 404:

<table>
<thead>
  <tr>
    <th>
      Endpoint
    </th>
    
    <th>
      Recorded
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        POST /v1/chat/completions
      </code>
    </td>
    
    <td>
      Yes
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        POST /v1/completions
      </code>
    </td>
    
    <td>
      Yes
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        POST /v1/embeddings
      </code>
    </td>
    
    <td>
      Yes
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        GET /v1/models
      </code>
    </td>
    
    <td>
      No, answered from the next replica
    </td>
  </tr>
</tbody>
</table>

Health checks of your load balancer can use the gateway's own `GET /healthz`, which answers `ok` without touching an engine.

Streams pass through event by event. The gateway asks vLLM for token counts at the end of each stream and keeps that extra chunk from clients that did not ask for it, so nothing changes for them.

## Say who each request is for

Two request headers tell the gateway who a request was for and what for:

<table>
<thead>
  <tr>
    <th>
      Header
    </th>
    
    <th>
      Meaning
    </th>
    
    <th>
      Example
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        X-Tenant-Id
      </code>
    </td>
    
    <td>
      Your customer: whom the cost belongs to
    </td>
    
    <td>
      <code>
        acme-corp
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        X-Feature
      </code>
    </td>
    
    <td>
      The part of your product that sent it
    </td>
    
    <td>
      <code>
        chat
      </code>
      
      , <code>
        search
      </code>
      
      , <code>
        agent
      </code>
    </td>
  </tr>
</tbody>
</table>

The report splits each replica's cost across tenants. The feature is recorded for your own queries.

Set them where you know the answer. Usually that is the service that authenticates your customers:

<code-group>

```python [OpenAI SDK]
client = OpenAI(
    base_url="http://10.0.0.11:8080/v1",
    api_key="unused",
    default_headers={"X-Tenant-Id": customer.id, "X-Feature": "chat"},
)
```

```nginx [nginx]
location /v1/ {
  proxy_set_header X-Tenant-Id $customer_id;   # from your auth layer
  proxy_set_header X-Feature   chat;
  proxy_pass http://llm;
}
```

```bash [curl]
curl http://10.0.0.11:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Tenant-Id: acme-corp" -H "X-Feature: chat" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Hello"}]}'
```

</code-group>

If your stack already carries the customer in a header of another name, tell the gateway its name instead of adding one:

```yaml [gateway.yaml]
headers:
  tenant: X-Customer
  feature: X-Product-Area
```

Requests without the header are still recorded and still cost money; the report lists them as **(no tenant header)**.

<note>

The gateway never stores request or response text. It hashes the text of each request in memory, in blocks of 256 characters with a per-install salt, and keeps only the hashes, which show which requests share a prefix. See [Privacy and security](https://inference-cost-optimizer.com/reference/security).

</note>

## Next

[Check the install](https://inference-cost-optimizer.com/install/check).


## Sitemap

See the full [sitemap](https://inference-cost-optimizer.com/sitemap.md) for all pages.