feat(cli): add bioengine worker start to launch a worker container - #182
Draft
nilsmechtel wants to merge 1 commit into
Draft
feat(cli): add bioengine worker start to launch a worker container#182nilsmechtel wants to merge 1 commit into
bioengine worker start to launch a worker container#182nilsmechtel wants to merge 1 commit into
Conversation
Starting a worker meant copying a ~10-line `docker run` block out of the deployment guide and adapting the GPU flag, the uid/gid and the bind mount by hand. `bioengine worker start` builds that command instead, picking the first of docker, podman and apptainer on PATH. The CLI is a launcher, not an in-process worker: it never imports ray, so `pip install "bioengine[cli]"` stays light and ray only ever runs inside the image. `--runtime native` is the escape hatch for environments that already have the worker extra installed. Worker arguments are forwarded verbatim after `--`, so the ~50 options on `python -m bioengine.worker` are available without being re-declared here and cannot drift out of sync. The image tag pins to the installed `bioengine` version so the CLI and the worker it starts cannot diverge. The Hypha token travels in the environment and is only ever named on the command line, never valued. Also adds `bioengine worker stop` / `logs`, and leads the deployment guide and README with the CLI while keeping the raw container commands as the documented fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes svamp issue #55.
Starting a BioEngine worker meant copying a ~10-line
docker runblock out ofdocs/deployment-guide.mdand adapting the GPU flag, the uid/gid pair and the bind mount by hand. This addsbioengine worker start, which builds that command instead.pip install "bioengine[cli]" bioengine worker start -- --mode single-machine --head-num-cpus 4Plus
bioengine worker stopandbioengine worker logs -ffor the container it started.What it is, and what it deliberately is not
A container launcher, not an in-process worker.
bioengine/cli/worker.pyimportsos,shutil,subprocess,pathlib,clickandbioengine.__version__— nothing else. It never imports ray, sopip install "bioengine[cli]"stays light and ray only ever runs inside the worker image.--runtime nativeis the escape hatch for an environment that already has the worker extra installed; it drops the container entirely and runspython -m bioengine.worker.Worker arguments are forwarded verbatim after
--.python -m bioengine.workerhas roughly fifty options. Re-declaring them here would mean a second copy that silently rots the first time one is added, so the command declares none of them: everything after--is passed through untouched, andbioengine worker start -- --helpshows the worker's own help. There is a test for the awkward case —-- --workspace-dir /data/wsconfigures the worker, not the container, even though the CLI defines an option by that name.The image tag pins to
__version__, notlatest. The CLI and the worker it starts cannot silently diverge.--imageoverrides it.The token is named on the command line, never valued. The container command carries
-e HYPHA_TOKEN; the value travels in the subprocess environment, so it does not appear inpsoutput. Apptainer only forwards variables it is told about explicitly, so it getsAPPTAINERENV_HYPHA_TOKENinstead. Two tests assert the literal token value appears in no runtime's argv.Runtime handling
--runtime auto(default) picks the first of docker, podman, apptainer onPATH. The three differ in exactly the ways the deployment guide already documents:--gpus=allvs--device nvidia.com/gpu=allvs--nv, and apptainer's--bindin place of-vwith no container to name, detach or size.GPUs default to on when
nvidia-smiis present, because passing--gpus=allon a host without the NVIDIA container toolkit makes the runtime refuse to start outright — it cannot simply be on by default.--gpusonly decides whether the container sees GPUs; Ray still needs--head-num-gpusafter the--, same as in the current docs.--dry-runprints the command and does nothing else. It deliberately does not require the runtime to be installed and does not create the workspace directory — the point of a dry run is to produce a command for a host you are not on. That is what makes the podman and apptainer paths verifiable on a docker-only machine:stopandlogsrefuse a resolved apptainer runtime with a hint rather than a stack trace, since there is no named container to act on.Testing
21 new tests in
tests/cli/test_worker_cli.py, pinning the argv for each runtime against the shape documented in the deployment guide, the token's absence from argv, verbatim argument forwarding, and the dry-run contract.Nine positive controls, each breaking one behaviour, each failing exactly the tests that name it — token value moved onto argv (2 fail), podman given docker's GPU flag (1), always detached (1), GPU flag forced on (1), worker args dropped (4), image unpinned (1), dry run creating the workspace (1), dry run demanding the runtime (1), apptainer env prefix dropped (1).
Full suite: 226 passed on this branch vs 205 on
mainunder the same invocation (python -m pytest tests --noconftest -q), 21 new, with the same 55 pre-existing collection errors on both.Docker and native were also exercised end-to-end via
--dry-runon this machine; podman and apptainer only through--dry-runand unit tests, as neither is installed here.Docs
docs/deployment-guide.mdMode 1 now leads with the CLI and keeps the rawdocker runandapptainer execblocks under "Running the container directly" as the explicit fallback. README gains the one-liner in both the quickstart and the CLI section.