GPU VRAM read from the wrong device when CUDA_VISIBLE_DEVICES restricts visibility (over-advertises mesh capacity)

[ndizazzo]

Summary

When CUDA_VISIBLE_DEVICES restricts which GPUs are visible, a node reports the visible device's
name alongside a different device's memory, then advertises that wrong number to the mesh as its
placement budget.

Observed on a host with an RTX 5090 (device 0, 32607 MiB) and an RTX 3080 (device 1, 10240 MiB) with
CUDA_VISIBLE_DEVICES=1:

"my_vram_gb": 33.676066816,
"gpus": [{
  "name": "NVIDIA GeForce RTX 3080",
  "vram_bytes": 34190917632,          // 32607 MiB = the RTX 5090's total
  "rated_vram_gb": 32,                // also the 5090
  "reserved_bytes": 514850816,
  "allocatable_vram_bytes": 33676066816
}]

libcuda under the identical environment sees only the 3080:

cuda visible device count: 1
cuda[0] NVIDIA GeForce RTX 3080 total=10354032640 bytes (9874 MiB)

So the node advertises ~33.7 GB on a card holding ~10.35 GB, about 3.3x over-advertisement, and it
propagates: the peer entry on another node showed the same vram_gb: 33.676066816.

Not a regression. The released v0.76.0 binary reproduces the identical numbers, verified by
running the published mesh-llm-v0.76.0-x86_64-unknown-linux-gnu-cuda-13.tar.gz (checksum verified)
against a locally built candidate on the same host with the same environment.

Root cause

Two positional joins between enumerations that do not share visibility semantics, all in
crates/mesh-llm-system/src/hardware/enrichers.rs (module linux):

1. cuda_device_infos() enumerates via libcuda, which honours CUDA_VISIBLE_DEVICES. With
   =1, infos has one entry: the 3080.
2. merge_nvml_device_infos() iterates NVML with for index in 0..count. NVML ignores
   CUDA_VISIBLE_DEVICES, so count == 2. It merges each NVML device into infos.get(index), so
   NVML device 0 (the 5090) is merged into the 3080's slot.
3. match_nvidia_device() tries a PCI-BDF match, then falls back to infos.get(gpu.index). The
   skippy-enumerated 3080 has index == 0, so it matches the now-contaminated slot, and
   enrich_nvidia_gpu_facts() overwrites gpu.vram_bytes with the 5090's total.

The GPU name is unaffected because it comes from the skippy backend-device enumeration, which is
why the two disagree and make the bug visible.

Two things worth noting for the fix:

• The pre-enrichment value is already correct. gpu_facts_from_backend_devices() in
  skippy_devices.rs sets vram_bytes = device.memory_total from skippy, which reflects the visible
  device. Enrichment then overwrites a correct value with a wrong one. Declining to enrich is
  therefore a safe fallback, not a degradation.
• The PCI-BDF path is the correct one and it silently failed here. skippy does populate
  pci_bdf = device.device_id, and is_placeholder_pci_bdf() exists, implying placeholders occur.
  Establishing why the identity match missed is part of the work below.

Consequence

vram_bytes is the mesh placement budget and context size auto-scales to it, so the inflated budget
produced a real failure loading a 2.8 GB model:

[debug] backend: allocating 8704.00 MiB on device 0: cudaMalloc failed: out of memory
[warn ] warning: Native runtime reported a handled model-open failure
[error] error: Failed to start model unsloth/Qwen3.5-4B-GGUF:UD-Q4_K_XL: ... RuntimeError

Handled without a panic and the node stayed in the mesh. The /v1/models union correctly excluded
the failed model, so routing was not misled, but the peer still reported
serving_models: ["unsloth/Qwen3.5-4B-GGUF:UD-Q4_K_XL"] while llama_ready: false.

Scope

Any host where CUDA visibility is restricted and the visible device is not at the same global NVML
index: containers, Kubernetes with device plugins, and shared or scheduled multi-GPU machines. Those
are exactly the environments where a node is most likely to be handed a subset of the GPUs.

Introduced by

#509 (Report backend GPUs and bundle GPU benchmarks, merged 2026-05-12, commit 57cdac988), which
added enrichers.rs, the gpu.vram_bytes = total_bytes overwrite, and the infos.get(gpu.index)
fallback. git log -S places all three there.

---

Fix plan (TDD)

enrichers.rs currently has no test module and no test coverage. The plan below writes the tests
first at every stage. enrich_nvidia_gpu_facts() and match_nvidia_device() are already pure
functions taking &[NvidiaDeviceInfo], so most of this needs no refactor to become testable;
merge_nvml_device_infos() does need a seam.

Real two-GPU hardware is required for phases 0 and 4. Unit tests cannot prove the fix, because the
whole bug lives in the disagreement between two real driver enumerations. They can only lock in the
logic once the hardware tells us what the real inputs look like.

Phase 0 — Characterize on real hardware (before writing any fix)

Goal: capture the real GpuFacts and both driver enumerations on a two-GPU host, and answer the one
open question: why did the PCI-BDF match miss?

On a host with two NVIDIA GPUs of different memory sizes (validated here on an RTX 5090 +
RTX 3080), for each of CUDA_VISIBLE_DEVICES unset, 0, 1, and 1,0, record:

• skippy device_id / pci_bdf and device.memory_total per device, pre-enrichment
• cuda_device_infos() output: count, names, pci_bdf, total_bytes
• NVML output: count, UUIDs, pci_bdf, total_bytes
• the final /api/status gpus[]
• ground truth from libcuda and nvidia-smi --query-gpu=index,name,memory.total,pci.bus_id,uuid

Deliverable: a short table in this issue showing, for each setting, whether identities matched and
which slot each device landed in. Do not proceed until it is clear whether the PCI-BDF miss is a
formatting mismatch, a placeholder id, or an absent id, because that decides whether the fix must
also repair identity extraction rather than only remove the positional fallback.

Phase 1 — RED: unit tests that fail on today's code (no refactor needed)

Create mod tests in enrichers.rs, gated #[cfg(all(test, target_os = "linux"))], with a fixture
builder for GpuFacts and NvidiaDeviceInfo. Use the real observed numbers so the tests read as
the incident: 34_190_917_632 (5090) and 10_354_032_640 (3080).

1. index_fallback_does_not_borrow_memory_from_a_different_device
   Given one visible GpuFacts (3080, index: 0, vram_bytes: 10_354_032_640, no usable pci_bdf)
   and an infos list whose slot 0 is the 5090 (total_bytes: 34_190_917_632), assert after
   enrich_nvidia_gpu_facts() that vram_bytes is still 10_354_032_640.
   Fails today: becomes 34_190_917_632.

2. enrichment_declines_when_no_identity_matches
   Same shape, asserting reserved_bytes, vendor_uuid, and stable_id are also left untouched, so
   the fix does not simply special-case memory.
   Fails today.

3. pci_bdf_match_wins_over_index_position
   Visible 3080 at index: 0 with a valid pci_bdf, infos ordered so the 5090 sits at slot 0 and
   the 3080 at slot 1. Assert the 3080's total_bytes is applied.
   Should pass today (guards the correct path against regression while phases 2-3 change matching).

4. uuid_match_used_when_pci_bdf_is_a_placeholder
   Placeholder pci_bdf plus a matching vendor_uuid. Assert identity match still succeeds.
   Fails today: match_nvidia_device() never consults UUID.

5. single_visible_device_still_requires_identity_agreement
   The existing infos.len() == 1 convenience path, but with the single info being a different
   device. Assert no enrichment.
   Fails today.

Phase 2 — RED: introduce a seam for the NVML merge, test-first

merge_nvml_device_infos() dlopens libnvidia-ml.so.1 inline, so it cannot be tested. Write the
failing test first:

6. nvml_merge_keys_by_identity_not_position
   Given a CUDA list of length 1 (3080) and an NVML list of length 2 (5090 at 0, 3080 at 1), assert
   the merged result attributes the 3080's memory to the 3080 and does not write the 5090's
   memory into slot 0.

To make it compile, split the function:

fn nvml_device_infos() -> Vec<NvidiaDeviceInfo>            // FFI only, untested
fn merge_device_infos(cuda: &mut Vec<NvidiaDeviceInfo>,    // pure, unit-tested
                      nvml: &[NvidiaDeviceInfo])
fn merge_nvml_device_infos(infos: &mut Vec<NvidiaDeviceInfo>) {
    let nvml = nvml_device_infos();
    merge_device_infos(infos, &nvml);
}

This is a pure extraction with no behavior change, so tests 1-5 must stay at their existing red/green
status across it.

Phase 3 — GREEN: minimum change to pass

In dependency order:

1. merge_device_infos() joins on identity (uuid, else normalized pci_bdf). An NVML device with
   no identity match is appended rather than merged into an unrelated slot.
2. match_nvidia_device() matches on normalized pci_bdf, then uuid, then returns None.
   Delete the infos.get(gpu.index) fallback and the infos.len() == 1 shortcut, or gate the
   shortcut on both lists having length 1 and identities agreeing.
3. enrich_nvidia_gpu_facts() keeps its existing behavior for matched devices; unmatched devices
   retain skippy's vram_bytes.
4. If phase 0 shows identity extraction is the real gap, fix normalize_pci_bdf() and/or populate
   vendor_uuid earlier, with its own red test first.

Emit a warn once per survey when a device cannot be identified and enrichment is skipped, so this
degrades loudly instead of silently. Add a test asserting the warning fires.

Phase 4 — Real two-GPU hardware validation (the actual proof)

Unit tests cannot prove this fix. Run on a host with two NVIDIA GPUs of different memory sizes, with
the packaged binary from a release bundle, not target/release.

For each setting, assert the reported gpus[] and my_vram_gb against libcuda ground truth:

CUDA_VISIBLE_DEVICES	Expect visible	Expect vram_bytes	Expect my_vram_gb
unset	both GPUs	each device's own total	budget over both
0	5090 only	5090's total	5090-based
1	3080 only	3080's total (~10.35 GB)	3080-based, not ~33.7 GB
1,0	both, reordered	each device's own total, correctly attributed	budget over both
0,1	both	each device's own total	budget over both

Then confirm the downstream consequence is gone: with CUDA_VISIBLE_DEVICES=1, mesh-llm serve --model <~2.8 GB model> must load and serve without --max-vram, where today it dies with
cudaMalloc failed: out of memory. Capture /api/status, the model-load log, and nvidia-smi
before and after.

Finally, verify propagation: join a second node and confirm the peer entry's vram_gb matches the
corrected local value, since the wrong number was mesh-visible.

Phase 5 — Regression guard

• Keep tests 1-6 as the permanent guard; test 1 encodes the exact incident numbers.
• Add a CI test asserting match_nvidia_device() has no positional fallback, for example by
  constructing mismatched-length lists and asserting None. This is the invariant that broke.
• Consider a doctor check that flags a mismatch between skippy's memory_total and the enriched
  vram_bytes beyond a small tolerance, which would have surfaced this from a single command.

Acceptance criteria

• Phase 0 table posted here, with the PCI-BDF miss explained
• Tests 1-6 exist and initially fail for the stated reasons
• All six pass after phase 3, and the whole suite stays green
• Phase 4 table verified on real two-GPU hardware, evidence attached
• A ~2.8 GB model loads under CUDA_VISIBLE_DEVICES=1 without --max-vram
• Peer-visible vram_gb matches the corrected local value
• No device is silently enriched from an unidentified source; unmatched devices warn

Workaround until fixed

--max-vram <GB> caps advertisement, planning, and local-fit together, which corrects the advertised
figure and makes model load fit.

---

Found during an evidence-backed release validation of a12b535d7 against v0.76.0 on three real
hosts, including the A/B against the released build that established this as pre-existing.

[ndizazzo]

Phase 0 and Phase 4 on carrack (RTX 5090 at 00000000:01:00.0, RTX 3080 at 00000000:06:00.0, driver 610.x). Fix is #1760.

Phase 0: the PCI-BDF match never missed

The open question in the plan was why the identity match failed. It did not. libcuda returns a usable bus id for every visible device:

CUDA_VISIBLE_DEVICES	libcuda	NVML
unset	[0] 5090 0000:01:00.0 33676197888, [1] 3080 0000:06:00.0 10354032640	[0] 5090 34190917632, [1] 3080 10737418240
0	[0] 5090 0000:01:00.0	[0] 5090, [1] 3080
1	[0] 3080 0000:06:00.0	[0] 5090, [1] 3080
1,0	[0] 3080, [1] 5090	[0] 5090, [1] 3080
0,1	[0] 5090, [1] 3080	[0] 5090, [1] 3080

NVML ignores the variable in every case, libcuda honours it in every case, and the two lists never agree on order once visibility is restricted.

So match_nvidia_device resolved the 3080's slot correctly by PCI address. merge_nvml_device_infos had already poisoned that slot: it walks NVML positionally, and under =1 NVML index 0 is the hidden 5090, so the 5090's 34190917632 bytes and its UUID were written into the 3080's entry before matching ran.

The arithmetic that pins it: this issue reports reserved_bytes: 514850816, and round_up_to_mib(514719744) is exactly that, where 514719744 is the 5090's NVML reserved figure read off this host. A rounding artifact could not produce that.

Worth knowing for anyone reproducing: mesh-llm gpus never shows this. With the native runtime unloaded, query() falls through to hydrate_gpu_facts, which reads NVML directly and never calls the enrichers, so it lists both cards under every setting including CUDA_VISIBLE_DEVICES=9. Only the serve path reaches the bug.

The structural gap behind it: the CUDA list carried PCI addresses and no UUIDs, the NVML list carried UUIDs and no PCI addresses, so the two enumerations had no identity key in common at all. Position was the only thing left to join on. The fix reads nvmlDeviceGetPciInfo_v3 so both sides carry an address.

Phase 4: /api/status on real hardware

Same host, same procedure, serve with a release host build. Pre-fix is the v0.76.0 build at c84846cbd, post-fix is bugfix/1755-gpu-vram-identity-match.

CUDA_VISIBLE_DEVICES	Reported devices	vram_bytes	my_vram_gb
unset	5090, 3080	34190917632, 10737418240	44.029706240
0	5090	34190917632	33.676066816
1	3080	10737418240	10.353639424
1,0	3080, 5090	10737418240, 34190917632	44.029706240
0,1	5090, 3080	34190917632, 10737418240	44.029706240

Direct A/B under CUDA_VISIBLE_DEVICES=1:

• pre-fix: my_vram_gb: 33.676066816, vram_bytes: 34190917632 on the RTX 3080. Reproduces this issue exactly.
• post-fix: my_vram_gb: 10.353639424, vram_bytes: 10737418240.

Every device gets its own total, correctly attributed, in every ordering. The over-advertisement is gone.

The model-load criterion is not met, and not for the original reason

serve --model unsloth/Qwen3.5-4B-GGUF:UD-Q4_K_XL under CUDA_VISIBLE_DEVICES=1, ninfer stopped, 3080 at 239 MiB used out of 10240:

• pre-fix: llama_ready: false, my_vram_gb: 33.676066816
• post-fix: llama_ready: false, my_vram_gb: 10.353639424

Both fail with RuntimeError: failed to create llama context. Neither log contains a cudaMalloc failed: out of memory line or the allocating 8704.00 MiB on device 0 this issue recorded, so this is a different failure from the one the inflated budget caused, and correcting the budget does not clear it. I am not going to claim that acceptance box.

Two candidates worth separating out: the context planner may not be bounding itself by the corrected budget, or the 0.76.0-rc3 native runtime in this host's cache behaves differently from the one in the original report. That wants its own issue rather than being folded into this fix, since the reporting bug and the planning bug are independent.

Acceptance status

• Phase 0 table, with the PCI-BDF question answered (it matched; the merge was the problem)
• Tests 1-6 exist and fail for the stated reasons; 6 of 10 fail against the pre-fix logic, and pci_bdf_match_wins_over_index_position passes both ways by design
• All pass after the fix, suite stays green
• Phase 4 table verified on two-GPU hardware
• 2.8 GB model loads under CUDA_VISIBLE_DEVICES=1 without --max-vram — still fails, different cause, see above
• Peer-visible vram_gb matches the corrected local value — not run, needs a second node
• No device is silently enriched from an unidentified source; unmatched devices warn