Skip to content

accelerator_type shows "G" for GeForce RTX 3090 (Ray auto-detect passthrough) + latent lstrip prefix-strip bug #159

Description

@nilsmechtel

Summary

The worker's per-node accelerator_type reports "G" for an NVIDIA GeForce RTX 3090 (seen on the single-machine Europa worker, bioengine 0.15.0, Ray 2.55.1). Datacenter cards are unaffected — KTH reports A40 and deNBI reports T4 correctly on the same code version.

While investigating I also found a separate latent string-handling bug in the same function that isn't the cause of "G" but should be fixed alongside it.

Evidence

get_status() on the Europa worker (RTX 3090):

{
  "node_ip": "172.17.0.3",
  "total_gpu": 1.0,
  "accelerator_type": "G",
  ...
}

KTH (A40) and deNBI (T4) report correctly, so this is specific to the GeForce card, not a regression in the field plumbing.

Root cause of "G" — upstream in Ray, passed through verbatim

BioEngine never sets the accelerator resource; it only reads Ray's accelerator_type:<TYPE> node resource in bioengine/cluster/proxy_actor.py::_get_accelerator_type. The one string transform there (see below) cannot turn GeForce-RTX-3090 into G. Therefore the raw Ray resource on that node is literally accelerator_type:GRay's own GPU auto-detection labels the RTX 3090 as G, and BioEngine faithfully passes it through.

Datacenter GPUs map cleanly to a model code (A40 → A40, T4 → T4); the consumer GeForce product name does not, and Ray collapses it to G.

Suggested fix for the label

Give BioEngine a clean, explicit accelerator label instead of trusting Ray's auto-detection for unrecognized GPUs — e.g. set accelerator_type:GeForce-RTX-3090 as a custom resource when the worker starts Ray on such a node (this is what our Chiron worker on the same physical box does, and it displays the full name correctly).

Separate latent bug in the same function

bioengine/cluster/proxy_actor.py:334:

return resource_name.lstrip("accelerator_type:")

str.lstrip(chars) strips any leading character in the set {a, c, e, l, r, t, o, _, y, p, :} — it does not strip the literal prefix "accelerator_type:". It happens to be correct for A40, T4, and G (each starts with an out-of-set char), but it would silently corrupt any accelerator whose value begins with one of those letters. Examples:

"accelerator_type:A40".lstrip("accelerator_type:")            # -> "A40"   (ok, by luck)
"accelerator_type:GeForce-RTX-3090".lstrip("accelerator_type:") # -> "GeForce-RTX-3090" (ok, by luck)
# a hypothetical value starting with an in-set char would be truncated, e.g.
"accelerator_type:tesla".lstrip("accelerator_type:")          # -> "sla"   (BUG)

Fix:

return resource_name.removeprefix("accelerator_type:")

Note: fixing lstrip alone does not change the "G" display — the G comes from Ray's resource value, so the label fix above is also needed.

Impact

Cosmetic/observability: the dashboard and bioengine cluster status show G instead of the GPU model, and the lstrip misuse is a latent correctness bug for future accelerator names.

Repro

Run get_status() (or bioengine cluster status) against a worker on a GeForce RTX 3090 node and observe accelerator_type == "G".

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions