feat(cluster): report VRAM_MB booking in cluster status

[nilsmechtel]

get_status reports used_gpu: 0.04 on a GPU that is 83% reserved. Four federated-unet replicas are running on bioengine-worker-europa, each holding 5120 MB of a 24576 MB RTX 3090 — but the cluster block reads as an almost entirely idle GPU, which makes the status output actively misleading for anyone deciding whether another GPU app will fit.

That number is not wrong, it is answering a different question. BioEngine apps declare gpu_memory_mb, never num_gpus. On a cluster that advertises the VRAM_MB custom resource, AppBuilder._apply_gpu_memory books num_gpus = _GPU_HANDLE_EPSILON (0.01) purely to bind a device and populate CUDA_VISIBLE_DEVICES, and puts the real reservation in resources["VRAM_MB"]. So used_gpu on such a cluster is a count of device handles — 4 replicas × 0.01 — and the quantity that actually bounds packing is invisible.

This PR surfaces that quantity. get_cluster_state now carries total_vram_mb / used_vram_mb per node and, via the existing rollup loop, at cluster level. Both are derived the same way as used_cpu / used_gpu (total − available), and the _mb suffix keeps them distinct from the byte-valued, NVML-measured total_gpu_memory / used_gpu_memory, which report what the driver observes rather than what the scheduler has booked. The three numbers are genuinely different and the PR does not merge them.

On Europa the new fields read used_vram_mb: 20480 / total_vram_mb: 24576 next to used_gpu: 0.04.

The same value has a second exit path, so bioengine cluster status is updated too: it printed GPU: 0.0/1 for that saturated GPU (.1f of 0.04). It now prints the fraction at .2f and appends the booking on nodes that advertise VRAM_MB:

  172.17.0.3 [HEAD] G GPU: 0.04/1 booked: 20480/24576 MB  VRAM: 5.8/24.0 GB  CPU: 8/8
  10.0.0.5 [worker] T4 GPU: 0.99/1 VRAM: 5.8/24.0 GB  CPU: 6/8

Nodes with no VRAM_MB resource report 0 for both fields and their CLI line is unchanged. The keys are always present rather than conditional, so table renderers over nodes do not have to special-case their absence — but a 0 there means "this cluster does not use VRAM_MB booking", not "this GPU is free", and on those clusters the GPU fraction remains the real reservation.

What is verified, and what is not

tests/test_cluster_status_vram.py drives the plain class behind the Ray actor decorator with stubbed node resources, so the branch runs without a GPU cluster. Two tests: the Europa case (VRAM_MB advertised, booking reported where the GPU fraction is not), and the deNBI-shaped case (no VRAM_MB, both fields 0, used_gpu carrying a real 0.33 fraction). A positive control was run — neutering used_vram_mb to 0 fails the first test while the second still passes, so the pair discriminates rather than both passing for free.

Two things are honestly not proven here:

No live-cluster validation of the VRAM_MB branch. It is only exercised on a cluster that advertises the resource. Europa is currently running the federated consortium (4 replicas, CPU 8/8) and rolling it needs explicit sign-off; a CPU-only e2e run would satisfy the new e2e assertions vacuously. The additions to tests/end_to_end/test_worker.py are guarded by if field in … like every field around them, so they type-check the new keys where present and pass silently where not.

The new tests cannot run under pytest in this environment. tests/conftest.py has a session-scoped autouse validate_environment fixture that walks requirements-worker.txt and calls pytest.exit() on the first package missing from the running interpreter. aiortc==1.14.0 is declared (pyproject.toml:62, requirements-worker.txt:1) but not installed here, so any test inheriting that fixture exits before collection — tests/test_gpu_sizing.py, an existing neighbour of the new file, fails identically on a clean checkout. Under xdist this surfaces as a crashed worker rather than the exit message, which is worth knowing before debugging it as a test bug.

This is not suite-wide: tests/_app/conftest.py:11 overrides the fixture with a no-op, and tests/_app/test_decorators.py runs its 26 tests green here. The new file sits at tests/ top level like its neighbours and inherits the gate along with them, so it is left as-is rather than opting itself out of a deliberate check. Its two tests were executed standalone, bypassing conftest. Environment gap, not introduced by this PR, and CI installs the worker requirements so it is unaffected.

Draft: no version bump yet, per the repo's bump-just-before-ready rule.

🤖 Generated with Claude Code