Launchers
A launcher stops a replica's engine and starts it again with other flags. The replay does nothing else to your engine. Health checks, timeouts and restoring the original are the same whichever launcher you use. Set it per replica under launcher in the config the replay reads.
For vLLM running in a Docker container. The simplest one.
launcher:
kind: docker
container: vllm # the name of the engine's container
Each config runs in a copy of the engine's container: the same image, GPUs, volumes, environment, ports and networks, with only the engine's flags changed. The flags in production are read from the running container itself. The original container is only stopped, never changed or removed, so restoring it is starting it again: it comes back with whatever it was started with.
The replay needs access to the Docker socket, so run it as a user in the docker group or as root.
For vLLM run any other way: systemd, a process manager, a script of yours.
launcher:
kind: command
start: /opt/llm/vllm.sh start {flags}
stop: /opt/llm/vllm.sh stop
env:
VLLM_EXTRA: "{flags}" # {flags} works in env values too
cwd: /opt/llm
flags: # the flags in production today
gpu-memory-utilization: 0.9
max-model-len: 32768
{flags} is replaced by the config's flags as they are typed on vLLM's command line, such as --gpu-memory-utilization 0.9 --max-model-len 32768. flags must list the flags in production: every config is these with some changed, and they are what gets restored.
startmust return once the engine is starting; it should not wait for it to be ready. The replay waits for/healthitself.stopmust return once the engine is gone and its GPU memory is free. The next config will need it.- Each must exit with 0 when it worked. Either may take up to 10 minutes.
When restarts must go through a person, or to try the replay before automating anything.
launcher:
kind: manual
flags:
gpu-memory-utilization: 0.9
The replay prints the flags of each config and waits for you to restart the engine with them and press Enter. A watchdog cannot restore the original through a person, so if the replay is killed, restore it yourself.