Split planner weights layer allocation by hold-capacity (`vram_bytes`), not throughput — RAM-heavy nodes get the largest shard

[ccwriter369-beep]

Summary

skippy-topology's split planner weights per-node layer allocation by vram_bytes, which is (intentionally, per #219) a blended weight-holding capacity, not a measure of compute throughput. Because capacity and throughput are different quantities, a node with large system RAM and weak compute is allocated a larger share of the pipeline than a node with a fast GPU.

This is not a restatement of #219 — that issue established the number is intentional and fixed the display labels (PR #222). This is about a consumer of that number: using hold-capacity as an allocation weight.

The premise (from #219, quoting @Bortlesboat)

The number is intentional but the label is wrong. vram_bytes is computed as gpu_vram + 0.75 * (system_ram - gpu_vram). llama.cpp can offload layers to system RAM, so the value represents how large a model the node can actually serve — not how much GPU memory you have.

Agreed, and that's the correct semantic for a fit question. model_target_reconciliation_local_fit (crates/mesh-llm-host-runtime/src/runtime/mod.rs:4018-4026) asks "can this node hold the model?" and compares against vram_bytes — correct usage.

The problem

crates/skippy-topology/src/lib.rs:937 asks a different question with the same number:

(((request.layers.len() as u128) * (node.vram_bytes as u128)) / (total_weight as u128))

Layers allocated ∝ hold-capacity. But layers allocated should be ∝ how fast the node can execute them. NodeSpec (lib.rs:31-37) carries only node_id, cached_slice_bytes, vram_bytes — there is no throughput term. NodePlacementSignal adds rtt_ms/availability_score/missing_artifact_bytes, but those are tie-break adjustments in node_package_score (lib.rs:1158-1174), never multipliers on the span.

Pipeline parallelism is serial per request, so an over-allocated slow node isn't a partial contributor — it's a stall on every token.

Worked example (CPU-only node)

crates/mesh-llm-system/src/hardware/mod.rs:191-195:

fn apply_cpu_only_runtime_budget(survey: &mut HardwareSurvey, metrics: &[Metric], system_ram: u64) {
    if metrics.contains(&Metric::VramBytes) && system_ram > 0 {
        survey.vram_bytes = (system_ram as f64 * 0.75) as u64;
    }
}

and the not(feature = "skippy-devices") path at mod.rs:537 uses * 0.90.

So a 128 GB CPU-only box advertises ~115 GB capacity and outranks a 24 GB RTX 4090 by ~4.8x, receiving ~4.8x the layers — to execute on CPU. There is no zero-capacity guard: validate_request (lib.rs:1500-1528) checks only EmptyLayers/EmptyNodes and contiguity, and span = span.max(1) (lib.rs:941) floors every node at >=1 layer.

The same directionality applies (more mildly) to any GPU node with large RAM offload, since mod.rs:330 does vram_bytes = total + (ram_offload as f64 * 0.90).

The data already exists on the wire

crates/mesh-llm-protocol/src/proto/node.rs:92-97 already carries mem_bandwidth_gbps, compute_tflops_fp32, compute_tflops_fp16; node.rs:44 carries advertised_model_throughput. As far as I can trace, their only consumers are /status and gpu-bench display (api/status.rs:258-344, runtime/mod.rs:5864-6012) — none reach plan_weighted_contiguous.

Notably, request routing already reasons about throughput (network/metrics.rs:479-489 tps_for_model, consumed at network/openai/transport.rs:3468 and ingress.rs:247), while the split planner does not — mesh-llm-routing has no skippy dependency.

One caveat on any fix

crates/mesh-llm-gpu-bench/src/runner.rs:31-34 returns None when gpu_count == 0, so bandwidth/TFLOPs are never measured on precisely the CPU-only nodes this affects (base detection hardcodes None at hardware/mod.rs:1125-1127). A throughput term would be inert for those nodes until some CPU-side measurement (STREAM-style, or a conservative device-class default) exists. Measurement likely has to land before the weight.

A min(vram_bytes, required_bytes) clamp would separately address a node with far more capacity than the model needs still absorbing proportionally more layers.

Possibly related

ROADMAP.md:60-63 notes MoE sharding "most results show this doesn't perform as well as one would hope, more research is needed." Speculative, but capacity-weighted placement across heterogeneous nodes seems like it could contribute.

Question

Is capacity-as-allocation-weight intended (e.g. deliberately co-locating layers where the weights fit, accepting the throughput cost), or is a throughput term a gap worth filing properly? Happy to work up a PR if the latter, but this looks like a design call that should be yours first.

Verified against 2c2e808 (2026-07-15). skippy-topology is a pure crate (serde-only, 33 unit tests); note weighted_contiguous_plan_uses_node_vram_for_layer_spans (tests.rs:231-255) pins the current behavior deliberately, which is partly why I'm asking rather than assuming.

[Bortlesboat]

Since my #219 comment is the premise here: agreed on both halves. That number was always a fit quantity — "how large a model can this node serve" — and model_target_reconciliation_local_fit is the right kind of consumer. Using it as the span weight inverts the intent: the planner answers "who can hold the most" when the question is "who executes fastest," and with serial pipeline stages that's a per-token stall, not a graceful degradation.

Two observations that might help whoever picks this up:

• The min(vram_bytes, required_bytes) clamp is worth landing on its own regardless of the design call. It needs no new measurement, and it collapses the worst case: a 128 GB CPU box drops from ~4.8x a 4090's weight to ~1.25x for a 30 GB model. Doesn't fix the directionality, but it bounds the damage while the throughput question gets settled.

• For the throughput term itself, mem_bandwidth_gbps is probably the right first-order proxy rather than TFLOPs — decode is bandwidth-bound, and tokens/s ≈ bandwidth / active bytes gets the CPU-vs-GPU ratio (~50–100 GB/s vs ~1 TB/s) about right even without a bench run. That also softens the CPU-only measurement gap: a conservative device-class bandwidth default is much easier to defend than a synthetic TFLOPs number, and the field already exists on the wire per the issue.