Skip to content

fix(cluster): strip the accelerator_type prefix, not its characters - #173

Draft
nilsmechtel wants to merge 1 commit into
mainfrom
fix/accelerator-type-prefix
Draft

fix(cluster): strip the accelerator_type prefix, not its characters#173
nilsmechtel wants to merge 1 commit into
mainfrom
fix/accelerator-type-prefix

Conversation

@nilsmechtel

Copy link
Copy Markdown
Collaborator

_get_accelerator_type used str.lstrip("accelerator_type:") to remove a prefix. lstrip takes a character set, not a prefix, so it strips every leading character that appears anywhere in {a, c, e, l, r, t, o, _, y, p, :}. The line is correct today only by luck — A40, T4, G and GeForce-RTX-3090 each begin with a character outside that set. Any accelerator type starting with one of them is silently mangled:

"accelerator_type:tesla".lstrip("accelerator_type:")   # -> "sla"

removeprefix is the operation that was meant. Python ≥3.11 is already required (pyproject.toml:9), so it is available unconditionally.

tests/test_accelerator_type.py pins it: an ordinary type is read back unchanged, a type whose leading characters all appear in the prefix survives intact, and a node with no accelerator resource still yields None. A positive control was run — restoring lstrip fails exactly the middle test and leaves the other two passing, so the new test discriminates rather than passing for free.

I checked whether this mistake appears elsewhere before fixing the one line: every other lstrip/rstrip in bioengine/ (apps/builder.py:224, datasets/http_zarr_store.py:119,120,149,294, datasets/datasets.py:153,312,453, worker/worker.py:240) strips "/", a genuine single-character set, which is what the method is for. This was the only misuse.

The accelerator_type: "G" display is upstream, and the obvious workaround does not work

This PR does not change what the single-machine Europa worker reports for a GeForce RTX 3090, which is the literal string G. That is worth stating because it was filed alongside the lstrip bug and the two are three lines apart.

BioEngine never sets the accelerator resource; it only reads Ray's. Ray derives the label in ray/_private/accelerators/nvidia_gpu.py with

NVIDIA_GPU_NAME_PATTERN = re.compile(r"\w+\s+([A-Z0-9]+)")

against the NVML device name. On "NVIDIA GeForce RTX 3090" the capture group matches uppercase-or-digit characters only, so it stops after the G of GeForce and the node resource really is accelerator_type:G. "NVIDIA A40" and "Tesla T4" capture cleanly, which is why KTH and deNBI look right on the same BioEngine version. I verified the regex is identical in Ray 2.55.1 (the version the workers run) rather than only in the 2.33.0 I happen to have installed locally.

The fix direction suggested when this was filed — have the worker pass an explicit accelerator_type:GeForce-RTX-3090 in ray start --resources — turns out to be unsound here. In ray/_private/resource_spec.py the user-supplied resources are copied in first (resources = (self.resources or {}).copy()) and the autodetected constraint is then added unconditionally under its own key. Different key, so the two coexist: the node would advertise both accelerator_type:G and accelerator_type:GeForce-RTX-3090, and _get_accelerator_type returns whichever it reaches first while Ray's own @ray.remote(accelerator_type=...) scheduling still sees the wrong one. There is no Ray setting that replaces the autodetected value — RESOURCE_CONSTRAINT_PREFIX is the only accelerator-related constant in ray_constants.py.

So the honest options are an upstream fix to that regex, or a BioEngine-private resource under a non-reserved key (say gpu_model:) that the status path prefers. The latter would only cover nodes BioEngine itself starts, leaving KubeRay-managed nodes on Ray's label, and the display is cosmetic. Not doing it here; recorded so the next person does not re-derive it.

_get_accelerator_type used str.lstrip("accelerator_type:"), which strips
any leading character in that set rather than the literal prefix. It is
correct for A40, T4 and G only because each starts with an out-of-set
character; "accelerator_type:tesla" yields "sla".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant