Skip to content

feat(cli): add bioengine worker start to launch a worker container - #182

Draft
nilsmechtel wants to merge 1 commit into
mainfrom
feat/worker-cli
Draft

feat(cli): add bioengine worker start to launch a worker container#182
nilsmechtel wants to merge 1 commit into
mainfrom
feat/worker-cli

Conversation

@nilsmechtel

Copy link
Copy Markdown
Collaborator

Closes svamp issue #55.

Starting a BioEngine worker meant copying a ~10-line docker run block out of docs/deployment-guide.md and adapting the GPU flag, the uid/gid pair and the bind mount by hand. This adds bioengine worker start, which builds that command instead.

pip install "bioengine[cli]"
bioengine worker start -- --mode single-machine --head-num-cpus 4

Plus bioengine worker stop and bioengine worker logs -f for the container it started.

What it is, and what it deliberately is not

A container launcher, not an in-process worker. bioengine/cli/worker.py imports os, shutil, subprocess, pathlib, click and bioengine.__version__ — nothing else. It never imports ray, so pip install "bioengine[cli]" stays light and ray only ever runs inside the worker image. --runtime native is the escape hatch for an environment that already has the worker extra installed; it drops the container entirely and runs python -m bioengine.worker.

Worker arguments are forwarded verbatim after --. python -m bioengine.worker has roughly fifty options. Re-declaring them here would mean a second copy that silently rots the first time one is added, so the command declares none of them: everything after -- is passed through untouched, and bioengine worker start -- --help shows the worker's own help. There is a test for the awkward case — -- --workspace-dir /data/ws configures the worker, not the container, even though the CLI defines an option by that name.

The image tag pins to __version__, not latest. The CLI and the worker it starts cannot silently diverge. --image overrides it.

The token is named on the command line, never valued. The container command carries -e HYPHA_TOKEN; the value travels in the subprocess environment, so it does not appear in ps output. Apptainer only forwards variables it is told about explicitly, so it gets APPTAINERENV_HYPHA_TOKEN instead. Two tests assert the literal token value appears in no runtime's argv.

Runtime handling

--runtime auto (default) picks the first of docker, podman, apptainer on PATH. The three differ in exactly the ways the deployment guide already documents: --gpus=all vs --device nvidia.com/gpu=all vs --nv, and apptainer's --bind in place of -v with no container to name, detach or size.

GPUs default to on when nvidia-smi is present, because passing --gpus=all on a host without the NVIDIA container toolkit makes the runtime refuse to start outright — it cannot simply be on by default. --gpus only decides whether the container sees GPUs; Ray still needs --head-num-gpus after the --, same as in the current docs.

--dry-run prints the command and does nothing else. It deliberately does not require the runtime to be installed and does not create the workspace directory — the point of a dry run is to produce a command for a host you are not on. That is what makes the podman and apptainer paths verifiable on a docker-only machine:

$ bioengine worker start --runtime apptainer --gpus --dry-run -- --mode single-machine
apptainer exec --nv --bind /home/user/.bioengine:/.bioengine \
  docker://ghcr.io/aicell-lab/bioengine-worker:0.16.5 python -m bioengine.worker --mode single-machine

stop and logs refuse a resolved apptainer runtime with a hint rather than a stack trace, since there is no named container to act on.

Testing

21 new tests in tests/cli/test_worker_cli.py, pinning the argv for each runtime against the shape documented in the deployment guide, the token's absence from argv, verbatim argument forwarding, and the dry-run contract.

Nine positive controls, each breaking one behaviour, each failing exactly the tests that name it — token value moved onto argv (2 fail), podman given docker's GPU flag (1), always detached (1), GPU flag forced on (1), worker args dropped (4), image unpinned (1), dry run creating the workspace (1), dry run demanding the runtime (1), apptainer env prefix dropped (1).

Full suite: 226 passed on this branch vs 205 on main under the same invocation (python -m pytest tests --noconftest -q), 21 new, with the same 55 pre-existing collection errors on both.

Docker and native were also exercised end-to-end via --dry-run on this machine; podman and apptainer only through --dry-run and unit tests, as neither is installed here.

Docs

docs/deployment-guide.md Mode 1 now leads with the CLI and keeps the raw docker run and apptainer exec blocks under "Running the container directly" as the explicit fallback. README gains the one-liner in both the quickstart and the CLI section.

Starting a worker meant copying a ~10-line `docker run` block out of the
deployment guide and adapting the GPU flag, the uid/gid and the bind mount
by hand. `bioengine worker start` builds that command instead, picking the
first of docker, podman and apptainer on PATH.

The CLI is a launcher, not an in-process worker: it never imports ray, so
`pip install "bioengine[cli]"` stays light and ray only ever runs inside
the image. `--runtime native` is the escape hatch for environments that
already have the worker extra installed.

Worker arguments are forwarded verbatim after `--`, so the ~50 options on
`python -m bioengine.worker` are available without being re-declared here
and cannot drift out of sync. The image tag pins to the installed
`bioengine` version so the CLI and the worker it starts cannot diverge.
The Hypha token travels in the environment and is only ever named on the
command line, never valued.

Also adds `bioengine worker stop` / `logs`, and leads the deployment guide
and README with the CLI while keeping the raw container commands as the
documented fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant