VINDEX3: c9 all-layer container, and a real model running from it - #202
Merged
Conversation
`load_vindex_config` has refused non-VINDEX2 directories since dual-generation
support landed, and its docstring says why: without the check a VINDEX3
directory deserialises into `VindexConfig` on the strength of its shared field
names, and the v1 loader proceeds against a layout whose weights live somewhere
else — a served model with wrong weights rather than an error.
Two entry points read `index.json` for themselves and skipped that gate:
load_vindex_embeddings reached first by walk/run — `walk_cmd` loads
embeddings at :270, before the gated config at :415
load_vindex_with_range reached by `larql serve` via bootstrap
Both refused a VINDEX3 container only by accident. VINDEX3's `index.json`
happens to omit `intermediate_size`, so serde rejected it with a field-level
message naming nothing about generations — `larql run` reported
parse error: missing field `intermediate_size` at line 45 column 1
Any `#[serde(default)]` added for schema-1 compatibility would have opened the
path silently. Schema 1 is already read "by filling absent fields with
defaults" (see format::generation), so that is not a hypothetical edit.
Both now gate, and the regression feeds all three entry points a VINDEX3 index
carrying every field `VindexConfig` needs — removing the accident so the test
measures the gate rather than the field overlap.
`larql run` and `larql dev walk` now report:
this is a VINDEX3 index; that path requires the VINDEX2 loader
c9 is not c8 in a loop. `import_one_layer` returns a `ContainerSpec` whose
segments own their bytes; at ~421 MB per layer, describing thirty would hold
~12 GB of already-mmapped weights a second time before writing any of them.
`ContainerBuilder` streams instead — each layer is written straight to its
final path and forgotten — so peak memory is one expert's slice whatever the
model's size. `index.json` is still written last: a multi-gigabyte import is
long enough to actually be interrupted, and a crash must leave a directory that
is not yet a container rather than one announcing itself as VINDEX3 with two
thirds of its banks missing.
Measured on gemma4-26b-a4b: 30 routed layers, 12.33 GiB, reopens and verifies
clean, 7680/7680 regions byte-identical to their VINDEX2 source.
Also fixes a mis-declaration this rung's arithmetic exposed. The importer
described both regions with one stored width, but they differ:
gate_up [2*inter, hidden] never padded — hidden is already a
256-multiple, so Q4_K quantises cleanly
down [hidden, inter_padded] inter padded to the next super-block
(704 -> 768 on gemma4-26b-a4b)
So gate_up was declared 1536 rows where the bytes hold 1408. The segment file
is 441,192,576 B; gate_up@704 + down@768 over 128 experts is 441,188,352 B plus
4,224 B of header and tables. gate_up@768 would need 467,140,608 B — 26 MB more
than the file is.
Nothing caught it, and the reasons compound: regions are copied verbatim so the
declared shape never enters the byte length; `verify` checks structure rather
than shape-against-length; and `Lyrw2Writer::write_region` records `bytes.len()`
and ignores the schema. That last one is deliberate — `RegionFormat::Unknown`
must round-trip — which means length-vs-shape can only ever be an invariant for
known formats. It is not one yet; worth a fifth verifier invariant alongside
the group_width four.
This matters more than a wrong number in a manifest: a container that states
what an operation will read was overstating it, in the region K3's grouped
dispatch will read most.
`MoeLayerSource` now carries both widths, the fixtures encode the asymmetry
(they previously padded both equally and so could not have caught this), and a
test pins each region to its own width.
Every VINDEX3 execution result until now bound its operands out of a VINDEX2
file, and `Vindex3Container` appeared nowhere in larql-inference — nothing on
the execution path could read a container to compute with. c8/c9 proved the
bytes could be written; this is the first time anything reads them.
larql run <vindex2> --routed-from <vindex3> "The capital of France is"
composed run: VINDEX2 spine ... + VINDEX3 routed banks ...
Paris.
Exactly one operand source is replaced — spec §4 classes 4 and 5. Tokenizer,
config, embeddings, attention, norms, routers, dense/shared FFN and LM head all
come from the VINDEX2 model unchanged, so a comparison against the same prompt
without the flag is a statement about the routed bytes and nothing else.
Teacher-forced parity, both arms through the identical loop, on the templated
prompt the deployed path actually serves:
hidden state 92928 / 92928 bit-identical max |diff| 0e0
final logits max |diff| 0e0
top-1 control 50429 ("Paris") == container 50429 ("Paris")
Zero, not close. The regions are byte-identical to source (c9), so the same
bytes through the same kernels must give the same numbers; 1e-7 would have been
a finding about the binding, not a pass. A greedy continuation match is not the
gate — argmax is stable long after a distribution has moved.
This is a composed run, not a VINDEX3 model. A routed-only container has no
spine and no tokenizer, and `larql run <vindex3-dir>` is still refused.
Regions are reached only through Vindex3Container -> segment -> region_bytes,
never by parsing LYRW descriptors here, so the pending 24 B -> 28 B descriptor
change lands in the parser and leaves this stable.
--- A refusing MoE route no longer zeroes a layer and continues
Wiring the flag exposed a semantic defect at the injection point. On a backend
error `moe_ffn_block_cpu` printed to stderr and left `h2` at zeros, so a broken
layer produced a fluent continuation from a model that had lost that layer's
entire expert contribution — not a fallback to VINDEX2 bytes, a fallback to
nothing. `--routed-from` refusing correctly is worthless if the caller swallows
the refusal.
Failure handling is now the operation's property, not the backend's:
MoeFailurePolicy::Fatal run, scoring, parity, benchmarks, serving
MoeFailurePolicy::RecordRefusal analysis only, declares its incompleteness
A backend cannot distinguish those callers, and asking it would imply a false
taxonomy where some backends' failures are tolerable. They are not: a remote
shard failing mid-generation and contributing zero is exactly as invalid as a
missing container region. Remote generation is therefore not grandfathered —
`generate_kquant_cpu_remote` is now fatal and aborts the request. The
ffn_adapter Strict/BestEffort contract is untouched: it needs the refusal
captured so Strict can raise it, so it keeps RecordRefusal.
`predict_kquant_hidden` keeps its signature — 39 of its 41 callers pass None,
where no refusal is reachable — and gains a `_checked` sibling taking a route.
Where RecordRefusal is used the unreachable-error path is an explicit expect(),
so a future policy change cannot quietly restore the old behaviour.
A test named generate_kquant_cpu_remote_runs_against_disconnected_backend
asserted generation "still picks tokens off the dense path" with the backend
disconnected. It was encoding the defect; it now asserts the abort. No log in
the repo contains "dispatch error L", so no recorded result is invalidated.
--- Known cost
The composed path costs ~1.6 s/token more in the expert stage (1.246x wall),
from resolving 7680 regions per token through the entry table. Measured with a
discarded warm-up round and alternating arm order — a first attempt without
those reported the container as *faster*, which was warm-up landing on whichever
arm ran second.
Separately: 83% of a token on this full-recompute path is in no stage counter
at all (attn and lm_head both read zero), so the existing split accounts for a
minority of the runtime. Both are follow-ups; output is bit-identical.
Two CI failures, both mine and neither caught locally.
**Format check** (5 jobs, all platforms): I never ran `cargo fmt`. Whitespace
and line-wrapping only; no logic changed, and the workspace builds and tests
identically after it.
**Coverage policy**: `format/vindex3/import.rs` fell to 89.01% against the
90% per-file floor. The lines were ones this branch added and left untested —
`region_format_for`'s refusal arm, the second iteration of the per-region width
check, and the parent-directory creation in `write_segment_file`.
Rather than pad, the gaps are now tested for what they are:
- a representable format maps to its region format
- an unrepresentable one is refused, not relabelled — the failure this whole
module exists to prevent, since a mislabelled region surfaces as wrong
numbers at execution with nothing pointing back to the import
- a view wider than the *down* region is refused naming `down`, which is the
only case that proves the second arm is reachable at all (the gate_up arm
is checked first and would otherwise mask it)
- a segment key's directories are created from the key, which is what the c9
builder relies on instead of pre-creating a tree it would have to keep in
step with the key format
import.rs 89.01% -> 93.96% (171/182); `check_coverage_policy.py` passes:
total 94.16% lines, 236 files checked, 193 at the 90% default.
Third CI gate missed in as many pushes, and the one that made the pattern
obvious: a green test suite, clean fmt and a passing coverage policy say
nothing about clippy either.
Two errors, both introduced by this branch:
hidden.rs `type_complexity` — wrapping `Option<(Array2, Option<SharedKV>)>`
in a `Result` to make a refused route fatal pushed the return
type past the threshold. Now a named `MoeLayerOutcome`, which is
a better shape anyway: the outer Option is an *absence* (no layer
to run) and the Result around it a *failure* (a refused route),
and collapsing those two is precisely how a missing operand
becomes a zeroed layer.
load.rs `map_identity` — `.err().map(|e| e)` in the generation-gate test
I added last push.
Also note for anyone reproducing locally: larql-vindex lints
`--all-targets`, which covers examples, while larql-inference lints
`--lib --tests --benches`. Checking with the latter's flags misses every
example — including the three this branch adds.
Verified with CI's exact invocations per crate, plus fmt, tests and the
coverage policy script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three findings, one new capability.
extractstill writes VINDEX2 and the ABIremains unfrozen; nothing here changes a default.
A real model runs with its experts served from VINDEX3
Teacher-forced parity, both arms through the identical loop, on the templated
prompt the deployed path serves:
Zero, not close — the regions are byte-identical to source, so the same bytes
through the same kernels must give the same numbers. A greedy continuation
match would not have been the gate; argmax is stable long after a distribution
has moved.
This is a composed run, not a VINDEX3 model. A routed-only container has no
spine and no tokenizer;
larql run <vindex3-dir>is still correctly refused.Container completeness is the next rung.
c9 — every routed layer, 12.33 GiB, byte-identical
30 layers, 7680/7680 regions identical to source, verifies clean. Streaming
builder: peak memory is one expert's slice rather than ~12 GB, and
index.jsonis written last so an interrupted import leaves a directory that is not yet a
container.
Three defects fixed
The importer over-declared the fused region.
gate_upis never padded,downis; describing both with one width claimed 1536 rows where the byteshold 1408. Byte-identity,
verifyandwrite_regionall pass it — regionsare copied verbatim so the declared shape never enters the length. A
container that states what an operation will read was overstating it, in
the region K3's grouped dispatch reads most.
Two v1 loader entry points skipped the generation gate.
larql runhitload_vindex_embeddingsbefore the gated config and reportedmissing field 'intermediate_size'. It refused only by accident of fieldoverlap; a
#[serde(default)]for schema-1 compatibility would have openedit silently — and schema 1 is already read by filling absent fields with
defaults.
A refusing MoE route zeroed the layer and continued. Generation produced
a fluent continuation from a model missing an entire expert layer, with one
stderr line as evidence. Now
MoeFailurePolicy::{Fatal,RecordRefusal},selected by the operation rather than declared by the backend.
Behaviour change worth calling out
Remote-shard generation is no longer best-effort by default. A shard
failure aborts the request instead of contributing zeros. This is a
serving-visible change riding in on a VINDEX3 branch. The
ffn_adapterStrict/BestEffort contract is untouched. No log in the repo contains
dispatch error L, so no recorded result is invalidated.Known, not fixed
per token. Measured with a discarded warm-up and alternating arm order — a
first attempt without those reported the container as faster, which was
warm-up landing on whichever arm ran second.
attnand
lm_headboth read zero), so the existing split accounts for a minorityof runtime.
it needs regenerating after the lyrw2 28-byte change. The builder survives —
it never parses descriptors — only the bytes on disk are stale.
Verification
cargo test -p larql-inference -p larql-kv -p larql-vindex -p larql-cli— 63 suites, 5664 tests, 0 failures. Both intermediate commits build.