---
title: "Launchers"
description: "How the replay restarts your engine under other flags, set per replica."
canonical_url: "https://inference-cost-optimizer.com/replay/launchers"
---
# Launchers

> How the replay restarts your engine under other flags, set per replica.

A launcher stops a replica's engine and starts it again with other flags. The replay does nothing else to your engine. Health checks, timeouts and restoring the original are the same whichever launcher you use. Set it per replica under `launcher` in the config the replay reads.

<tabs>
<tabs-item icon="i-simple-icons-docker" label="docker">

**For vLLM running in a Docker container.** The simplest one.

```yaml
launcher:
  kind: docker
  container: vllm      # the name of the engine's container
```

Each config runs in a copy of the engine's container: the same image, GPUs, volumes, environment, ports and networks, with only the engine's flags changed. The flags in production are read from the running container itself. The original container is only stopped, never changed or removed, so restoring it is starting it again: it comes back with whatever it was started with.

The replay needs access to the Docker socket, so run it as a user in the `docker` group or as root.

</tabs-item>

<tabs-item icon="i-lucide-terminal" label="command">

**For vLLM run any other way:** systemd, a process manager, a script of yours.

```yaml
launcher:
  kind: command
  start: /opt/llm/vllm.sh start {flags}
  stop: /opt/llm/vllm.sh stop
  env:
    VLLM_EXTRA: "{flags}"        # {flags} works in env values too
  cwd: /opt/llm
  flags:                         # the flags in production today
    gpu-memory-utilization: 0.9
    max-model-len: 32768
```

`{flags}` is replaced by the config's flags as they are typed on vLLM's command line, such as `--gpu-memory-utilization 0.9 --max-model-len 32768`. `flags` must list the flags in production: every config is these with some changed, and they are what gets restored.

- **start** must return once the engine is starting; it should not wait for it to be ready. The replay waits for `/health` itself.
- **stop** must return once the engine is gone and its GPU memory is free. The next config will need it.
- Each must exit with 0 when it worked. Either may take up to 10 minutes.

</tabs-item>

<tabs-item icon="i-lucide-hand" label="manual">

**When restarts must go through a person,** or to try the replay before automating anything.

```yaml
launcher:
  kind: manual
  flags:
    gpu-memory-utilization: 0.9
```

The replay prints the flags of each config and waits for you to restart the engine with them and press Enter. A watchdog cannot restore the original through a person, so if the replay is killed, restore it yourself.

</tabs-item>
</tabs>

<note>

Kubernetes is not supported yet: there is no launcher that restarts a Deployment under other flags.

</note>


## Sitemap

See the full [sitemap](https://inference-cost-optimizer.com/sitemap.md) for all pages.