Send traffic through the gateway
This is the one change in your request path. The gateway speaks the same OpenAI-compatible API as vLLM, so moving traffic to it is a change of address.
Point traffic at the gateway
Change each upstream of your router from vLLM's port to the gateway's port on the same machine, for example from 10.0.0.11:8000 to 10.0.0.11:8080. With nginx:
upstream llm {
server 10.0.0.11:8080; # was 10.0.0.11:8000
server 10.0.0.12:8080; # was 10.0.0.12:8000
}
Change the base URL your clients use:
from openai import OpenAI
client = OpenAI(base_url="http://10.0.0.11:8080/v1", api_key="unused")
The gateway proxies these endpoints and answers everything else with 404:
| Endpoint | Recorded |
|---|---|
POST /v1/chat/completions | Yes |
POST /v1/completions | Yes |
POST /v1/embeddings | Yes |
GET /v1/models | No, answered from the next replica |
Health checks of your load balancer can use the gateway's own GET /healthz, which answers ok without touching an engine.
Streams pass through event by event. The gateway asks vLLM for token counts at the end of each stream and keeps that extra chunk from clients that did not ask for it, so nothing changes for them.
Say who each request is for
Two request headers tell the gateway who a request was for and what for:
| Header | Meaning | Example |
|---|---|---|
X-Tenant-Id | Your customer: whom the cost belongs to | acme-corp |
X-Feature | The part of your product that sent it | chat, search, agent |
The report splits each replica's cost across tenants. The feature is recorded for your own queries.
Set them where you know the answer. Usually that is the service that authenticates your customers:
client = OpenAI(
base_url="http://10.0.0.11:8080/v1",
api_key="unused",
default_headers={"X-Tenant-Id": customer.id, "X-Feature": "chat"},
)
location /v1/ {
proxy_set_header X-Tenant-Id $customer_id; # from your auth layer
proxy_set_header X-Feature chat;
proxy_pass http://llm;
}
curl http://10.0.0.11:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-Tenant-Id: acme-corp" -H "X-Feature: chat" \
-d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Hello"}]}'
If your stack already carries the customer in a header of another name, tell the gateway its name instead of adding one:
headers:
tenant: X-Customer
feature: X-Product-Area
Requests without the header are still recorded and still cost money; the report lists them as (no tenant header).