Privacy and security
Nothing leaves your network
Every part of ico runs on your machines. Nothing is sent to us or to anyone else: there is no telemetry, no licence check, no call home.
What is stored, and what is not
| Stored | Never stored |
|---|---|
| Token counts, latencies, status codes, models, replicas | Prompt text |
| The tenant and feature values your requests carry | Completion text |
| Salted hashes of request text, in blocks | Request or response headers other than the two you name |
| Engine metrics and GPU counters | Client IP addresses, API keys or Authorization headers |
The gateway reads each request's text only to hash it, in memory, in blocks of 256 characters (64 blocks at most), with HMAC-SHA256 keyed by a per-install salt, then drops the text. The hashes only show that two requests begin the same way. Without the salt they cannot be compared with anything outside your install, and they cannot be turned back into text.
Tenant and feature values are stored as your requests send them. Use ids rather than names if your customers' names are themselves sensitive.
The replay sends no real text
The replay rebuilds your traffic as prompts of token ids, with the recorded lengths and the shared prefixes the hashes reveal. The engine sees synthetic tokens, never your users' text, and the answers are thrown away.
What crosses your network
| Connection | Protected by |
|---|---|
| Clients to the gateway | Your network. The gateway adds no authentication of its own: keep the access control you have in front of it. |
| Gateway and vLLM to the collector (gRPC, 4317) | Your private network. The collector accepts writes without authentication. |
| Collector to vLLM and GPU exporters (HTTP) | Your private network. |
| CLI to ClickHouse (HTTP, 8123) | The ico user's password. |
| CLI to the gateway's admin API (8081) | A bearer token. The token can drain replicas: keep the API on 127.0.0.1 or a private address. |
Secrets
| Secret | Where it lives |
|---|---|
Admin token (ICO_ADMIN_TOKEN) | The environment of every gateway, and of ico replay |
ClickHouse password (ICO_CLICKHOUSE_PASSWORD) | The environment of the collector and of the CLI |
Hash salt (ICO_HASH_SALT) | The environment of every gateway, or its state_dir |
None of them goes in the config file.
Access to your GPUs
ico's own containers never touch a GPU driver. GPU counters come from NVIDIA's own exporters. The replay restarts engines only through the launcher you configure, and only for the replica you name.