Repository navigation
Bind every node a slot address names when the model reaches a module by several paths - #146
Closed
ShivanshuPurohit wants to merge 29 commits into
Closed
ShivanshuPurohit wants to merge 29 commits into
ShivanshuPurohit wants to merge 29 commits into
Conversation
The prefix-cache state audit still carried a recurrent (GDN) checkpoint branch and an int8 codec expression that no commissioned arena can reach: the Qwen arena seals disable_radix_cache=true, which the cache seam refuses for hybrid engines before any guard is built, and its deployed d9bd034 source predates the cache seam; GLM and the planned DeepSeek-V4.1-Flash arena have no recurrent pool. A hybrid pool that carries recurrent state is now refused at bind time instead of being served unchecked. Consumer: the `tree_cache` binding in cacheon/integrations/sglang_cache.py, which builds PrefixStateAudit on every hybrid engine. Casualty: the recurrent checkpoint audit (`_recurrent`, mamba fixtures), the int8 codec expression, the Qwen recurrent-results section, and the Qwen-specific "disables radix caching" wording across the miner guide. Net lines: production +35/-99, tests +8/-159, docs +44/-78, scripts +3/-3. surface-change: none (surface_snapshot --check matches all 10 sections). Docs: slot contract, prefix-cache results, fidelity, miner guide and the manifest example updated in the same change; loc_baseline lowered to lock the deletion in. CPU suite: 2838 passed, 11 skipped; 3 dashboard health tests fail on macOS for the missing /proc only (not touched here).
…fs, inputs Off-GPU preparation for the third arena, read from the SGLang 0.5.21 source; the GPU proof on the commissioned image follows. Node audit: DeepSeekV4TokenToKVPool stores whole 256-token pages, so the existing slot-addressed capture raised on its first call. The new interpreter captures, restores and grades the window pages, a kv-source layer's compressed and FP4 index pages, and its per-request pending-pair ring, decoded by the layout the pool declares; unrecognized layouts raise rather than pass unchecked. Consumer: sglang_nodes._cache_rows. Supersedes nothing: GLM's DSA and the slot-addressed paths are unchanged. Prefix cache: 0.5.21 hands a finished request through insert_req then on_release instead of cache_finished_req; the guard now seals and checks both shapes, hashes the shorter DeepSeek-V4 index pages per full page, checks a window pool that stores whole pages by page, snapshots request-held compressor state across a handoff, and reads window eviction through the runtime's component API. Consumer: the tree_cache binding on every hybrid engine. Arena inputs for dsv41flash (Dockerfile on the v0.5.21 list digest, TP4/EP4 engine options verified as ServerArgs fields, 24 development requests in the GLM layout, README naming which node addresses execute). Dashboard sources declare their URL slug in the operator config; the hardcoded arena slug table is deleted. Net lines: production +238/-49, tests +256/-21, docs +23/-11, dashboard +14/-10, examples +144/-0; loc_baseline raised accordingly, the node adapter test file kept under 900 lines by moving the V4.1 tests to their own file. surface-change: none (10 sections match). Docs updated in the same change: slot contract, prefix-cache results, SGLang tracking, dashboard README. CPU suite: 2854 passed, 11 skipped; the same 3 dashboard health tests fail on macOS for the missing /proc only.
…h-cell display and Qwen support retired Consumer: the sealed EngineSessionConfig.engine_env reaches every rank through oci_session_worker._engine_session -> engine_worker.build_session_environment, and cacheon check --engine-config exports the same object; the dashboard, link previews and cacheon explain read the replay result through baseline_band.replay_measurements. Casualty: the batch-cell lane readers (_phase_measurements, the B/C/B' tok/s, prefill and TTFT/TPOT renderers, the machine-noise headline), the GDN ReplaySSM node-audit rows, the Qwen engine options, the Qwen input kit and its docs. Net lines: scripts +11/-10; tests +179/-403; docs +35/-42; cacheon +112/-276; dashboard +34/-159; examples +24/-335; surface-change: modules (baseline_band exports replay_measurements; sglang_dsv4_state registered).
The prefix-cache audit rounded read starts up and skipped the entire 128-token DSV4.1 window with 256-token pages. Include the first partially covered page while preserving record boundaries. The existing test now covers windows shorter than a page and nonmultiples of the page size. Consumer: existing prefix-cache match audit. Casualty: the ceiling-rounded read boundary. Production +4/-1, tests +6/-3, docs +2/-1; directory ceilings rise by 3/3/1 for this regression fix. 108 focused tests, docs checks and strict build, LOC/assertion ratchets and surface snapshot pass. GPU negative and honest-control confirmation remains pending.
The identity control ran 3.5-8% slower than stock on DeepSeek-V4.1 (B300, 2026-10-09). Each dispatched call cost 85 us in the commissioned image, 40 calls per eager step: a failed per-call import of the removed legacy capture module (28 us), resolving the receipt root (35 us), and rebuilding the call descriptor and matching every variant (12 us). Import each capture authority at most once per process, resolve an absolute receipt root once, and memoize the registry selection per (graph mode, dtype, width, device). Measured in the image: 85 -> 9-12 us per dispatched call. Consumer: the node dispatcher in every timed lane. Supersedes the per-call import and resolve it replaces. Net +70/-29 lines; the cacheon ceiling rises by 41 because the dispatcher path offered no further deletion beyond the stale lookup-API docstring removed here.
The 20 GB dependency relocation never changes between source revisions, but it ran in the same layer as the Cacheon install, below COPY src, so every source-only rebuild of the DeepSeek-V4.1-Flash worker image paid it again: 468 s for the copy-up and 917 s for the layer export on the B300 host on 2026-10-09. The relocation now sits above the copy and the install plus the bootstrap hook follow it, so a source-only rebuild reuses the cached layer. Final paths and the runtime check are unchanged. examples ceiling +3 for the comment that records the measurement.
The eager audit only ever ran after a speed PASS, so on an arena whose sealed replay fixes token counts at temperature zero under speculative decoding no wrong-output control could reach it, and the production audit had never been seen rejecting a wrong answer (fidelity.md already requires that canary). The sealed policy gains one optional field, audit_control_delta_digests: the builder gives a listed delta the existing calibration-observation disposition, intake and the deployment check accept that disposition only for a listed delta, and the continuation runner now stops after the audit whenever speed did not pass. An audit FAIL exits at the audit stage as before; an audit PASS is retained on the speed exit, whose decision stays the speed grade, so a control can neither qualify nor settle. Pristine T never runs after a speed non-PASS. The field is absent from the policy payload when empty, so every policy sealed before it keeps its digest; it is bound through the sealed authority configuration, not the per-request declaration digest. Supersedes the continue-into-T-after-a-failed-speed semantics (removed with its test), the report loop's second regrade of the same witness, and two unreachable None checks in the deployment plan check. Production +76/-40, tests +116/-40, docs +34/-6 plus the reboot and retention paragraphs of the arena-service page. Ceilings: cacheon +36, tests +76, docs +154 for these pages and the results page that follows; file ceilings lowered for qualification_runner.py and its test file. surface-change: none (snapshot unchanged).
Add docs/results/deepseek-v41-flash-b300.md: the production-path wall clock of the first complete qualification on this model (75.3 min, speed 55.0, eager audit 11.4, pristine T 8.0; futility FAIL 28 min), the dispatcher cost the 3f41699 image charged the candidate lane and the 51a359d fix, the 0.02 noise seal and its pass lines, and the controls run on the commissioned image: identity on both pairs, skip-MLP at the futility rule, candidate engine killed mid-window, pod service restarted mid-run, and skip-MLP under the registered-control audit policy of 9cbce87. The page records the cross-pair orientation-two skew as the next noise item. Nav entry under Results. Docs ceiling +15 (the new page); no production code. Consumer: the Results navigation section and the validator guide's qualification page, which link to it. Supersedes nothing.
Carry main's reviewed ceilings for the weight-reveal monitor change (docs +7, tests +49) onto the branch's baseline; cacheon stays within the branch ceiling. No production code in the merge itself.
The merge of main brought the weight-reveal monitor change (+33 cacheon lines); the per-directory ceiling now matches the merged tree. No production code.
…ashboard Every intake listener records every arrival it observed at its own configured fee, so each arena's dashboard summed the shared payment table: on 2026-10-11 the GLM page showed 165.0 τ, Qwen 28.7 τ and DeepSeek 158.5 τ from the same rows. The live scoped totals are GLM+MiniMax 154.0 τ (18 x 1.0 + 272 x 0.5), Qwen 16.3 τ (3 x 0.5 + 74 x 0.2) and DeepSeek 9.0 τ (18 x 0.5): 179.3 τ. - /api/payments joins the arena's scoped reservations, so a page lists only the fees its own reservations consumed. - New global /api/revenue reports each arena's fee payments grouped by amount and the total received across every arena, counting each on-chain payment once and naming unavailable sources. The Overview shows this block and takes the arena's own figure from it. - Casualty: the Overview's unscoped payment sum (totals.payments_count and totals.payments_tao) is deleted; dashboard/app.py shrinks to 932 lines and its ceiling is lowered to match. Consumer: the Overview tab and /api/revenue of the running dashboard. Lines: dashboard +76/-11 (app.py +1/-4, sources.py +50/-1, index.html +13/-3, README +12/-3); tests +135/-0 (new tests/test_dashboard_revenue.py). Directory ceilings raised by exactly those counts. Surface snapshot unchanged.
…a fixture The deletion audit on this branch judged the fee fix (f4f9c48) the PR's largest cuttable addition: a third route and a 135-line test file for a number the arena list already loads on every page refresh. - /api/arenas now carries each arena's fees (count, total, grouped by the fee charged; null while its database is unavailable) plus fees_count and fees_total_tao across every arena, counting an on-chain payment once under the first configured arena that owns it. The Overview reads the arena list it already holds instead of fetching a third endpoint. - Casualty: the /api/revenue route and tests/test_dashboard_revenue.py. The behavior is covered by one test on the existing two-arena planes fixture in tests/test_dashboard_sources.py. Consumer: the Overview tab and /api/arenas of the running dashboard. Lines: dashboard +43/-57 (sources.py +27/-43, index.html +11/-13, README +5/-11); tests +36/-136. Directory ceilings lowered to lock the deletion in.
…r-service test bundles/dp_output_projection targeted collective.dp_output_projection_norm, an op-slot the catalog no longer registers (f0fd527 retired op slots; the projection role is served by bundles/glm53_champion_nodes). Nothing in the tree reads it: no code, test, doc, packaging entry or surface hash. tests/test_remote_worker_service.py was an untracked 849-line file pinned by a per-file .gitignore entry; it imported a dispatcher symbol deleted in af5f47d and failed collection. The file and its ignore line are gone. Consumer: none; pure deletion. Casualty: the two artifacts above. Lines: bundles -587 (manifest 16, README 49, projection.py 98, dp_projection_native.cu 410, metadata 13, rebuild.json 1), .gitignore -1, baseline lowered by 587. No production Python path changes.
…ests backfill_reservation_baseline_segments bound pre-upgrade queue rows to the stack active when they arrived. Commit 3685d41 replaced its only caller with the commissioned-incumbent rebind in commission_boundary and named "crown-derived queue binding" as the casualty, but left the function and its FinalizedIntakeStore delegator behind. Nothing in cacheon/, dashboard/ or scripts/ calls either; the two tests that did now bind through the production path. tests/test_chain_evaluation_leases.py loses eight tests of paths the live validator cannot take: lease claim, expiry and accept on the plain FinalizedIntakeStore (every production store is the recoverable one: standing_cpu_supervisor, mainnet_screen_dispatcher, evaluation_lease_operator), reproduction_pending rows set by raw SQL (store open converts them), a schema-migration test that equals creating a fresh database, a duplicate of the worker-pool exclusive-claim test, a settlement fence built by raw SQL (covered by test_fixed_incumbent_rewards and test_chain_intake with the production fixture), and an event-reader tamper test that needs DROP TRIGGER on the validator's own SQLite. The deferral, fence, exact-cohort and infrastructure-never-FAIL tests stay. No live code path changes: the deleted function had no caller. Consumer: commission_boundary via the recoverable dispatcher. Casualty: the crown-derived backfill retired in 3685d41 and the plain-store lease tests. Lines: cacheon -113 (baseline_segments -104, intake -9); tests -214 (leases -213, chain_intake -4, +3 explicit bindings). Baselines lowered. surface-change: modules drops cacheon.chain.baseline_segments.backfill_reservation_baseline_segments (no consumer since 3685d41).
…routing surface Production symbols with no caller outside their own tests, and the tests that existed only to exercise them: - intake.py: release_manifest_compatibility_failure (an operator recovery seam nothing invokes) and settlement_blockers (no production reader). - device_state.py: _capture_verdict, before_capture, after_capture and the _drain envelope switch; their consumer left in eae4e8a. - stack_plan.py: CohortPlan and its entropy execution order. Production prepares one MarginalArmPlan at a time (b300_registered_qualification), so marginal_runtime loses the cohort branch. plan_marginal_arm digests are unchanged, compared against HEAD on the same inputs. - test_kernel_variants.py: registry routing tests that duplicate test_capabilities.py; registration-time duplicate-variant and inclusive range-overlap refusal moves there as one test. - test_chain_validator_loop.py: the in-loop qualification lane test (production runs chain-validate --intake-only) and the once-mode fault test duplicated in test_chain_archive.py. Consumer: none added; every kept symbol's callers are unchanged. Casualty: the symbols and tests above. +45 / -865 lines. surface-change: modules.json drops CohortPlan from cacheon.stack_plan. surface-change: digest_domains.json drops cacheon.stack.cohort-plan and cacheon.stack.cohort-execution-key.
… they carried - tests/support/evaluation.py owns the intake-row, arena-service and CPU coordinator fixtures that test_evaluation_coordinator and test_remote_evaluation_dispatcher each carried, and that test_parallel_arenas and test_evaluation_worker_pool imported from a test module. - tests/support/b300.py gains the executor, factory-builder, deployment authority and arena manifest fixtures. test_b300_arena_provider.py is deleted; its six non-duplicated contracts (plan order, runtime/policy mismatch, lane overlap and drift, declared-only refusal, closed provider, factory exception) move into test_b300_mainnet_worker.py. - test_remote_worker_pod_service.py: the two fake-process adapter tests fold into the real-subprocess test (frame schema, process reuse, durable start count); the publish test's republish idempotency moves into the recovery suite; the parser test becomes a real poll-bound check (the old one returned 2 because the registration file was missing, not because the bound fired). The 2026-08-10 cooldown counter test stays. - test_qualification_runner.py: the hidden-judge memoize and relabel tests (the commission check makes that branch unreachable), the duplicate audit-fail test, the fixture-restating calibration test, the empty-receipt witness test, the publish round-trip and two vacuous asserts. The exclusive-transaction refusal moves into test_oci_backend's real-manager lock test. Consumer: the suites above, unchanged in behavior. Casualty: the duplicated fixtures and tests. +596 / -1443 lines, no production code.
The single-PASS to two-PASS migration (schema 1/2 -> 3) last ran on a live database on 2026-07-13; every intake database since then opens at stamp 3 or later, and nothing in the tree writes the schema3_reproduction_required reason any more. Local read-only copies of the GLM (stamp 5) and Qwen (stamp 3) databases hold zero such rows. Removed: the migration branch in the store opener (an older stamp now fails loudly as unsupported instead of being rewritten), the hold-reason constants, archive_schema3_migration_hold and the schema-3 exclusions in automatic expiry, expire and release_hold, the chain-archive-schema3-hold CLI subcommand, the four tests that exercised them and the doc passages that described them. The expiry predicate loses one always-true term; with no such rows it selects the same set. Consumer: none added; the opener, expiry and hold paths keep their callers. Casualty: the migration path and its operator command. +20 / -384 lines. surface-change: cli.json drops the chain-archive-schema3-hold subcommand from the usage line and its help section. surface-change: modules.json drops cmd_chain_archive_schema3_hold from cacheon.cli.
…ct model docs/architecture/overview.md repeated the proposal/crown table, the evaluation-stack/reference table, the 3%-on-7% example, the trust model and the end-to-end flow that concepts.md, product-model.md, pipeline.md, stacks.md, threat-model.md and codebase-map.md already carry; its only unique section, the design acceptance tests, restated product-model.md's architectural acceptance test. docs/engine/integration.md repeated the crown/ship boundary that product-model.md states normatively. Both pages are deleted and their nine inbound links retargeted (README, CLAUDE.md, the docs home, concepts, pipeline, stacks and the nav). concepts.md loses the mental-model diagram and the two tables it shared with product-model.md; the docs home loses the B/C/A/T recap that the miner guide and glossary keep; product-model.md's manifest table becomes one sentence, and its source map no longer names the retired integration and release-stack identities. No code or test changes. Consumer: the surviving pages. Casualty: the two pages and their duplicated passages. +37 / -270 lines.
`chain-validate` has only ever run as `--intake-only` in production; the standing supervisor claims qualification and settlement from the same store. The in-process lane (`_apply_qualification`, an injected `ArenaServiceRegistry`, `AdmissionDecision`/`ArenaQueueSnapshot` admission, `retained_only` passes, `--arena-id`) had no production caller. Consumer: the live `chain-validate --intake-only` argv. Its intake path is unchanged apart from dedent; `cmd_chain_validate` now refuses any other mode and `run_pass`/`run_validator` take no arena or mode parameters. Casualty: `_qualification_reservations`, `_validate_work`, `_apply_qualification`, `QualificationWork`, the inline closed-target park, the in-pass settlement call, `ArenaServiceRegistry`, `ArenaService.admit_qualification`, `AdmissionDecision`, `ArenaQueueSnapshot`, `arena_queue_snapshot`, `qualification_cohort`, and the operator docs that told deployments to inject a registry. The closed-target test now exercises the supervisor's `prepare_qualification_queue` directly; `test_chain_intake` asserts the same cohort through `preview_evaluation_claim`. Lines: +113 / -639 (production +44/-435, tests +11/-100, docs +45/-91, baselines +13/-13). `PassResult.decisions`/`settlements` stay (always empty) so the audit-log schema the dashboard reads is unchanged. surface-change: cli.json chain-validate drops --arena-id; --intake-only help now states it is required surface-change: modules.json cacheon.arena_service drops AdmissionDecision, ArenaQueueSnapshot, ArenaServiceRegistry; cacheon.chain.validator_loop drops QualificationWork surface-change: digest_domains.json cacheon/arena_service.py drops cacheon.arena.service-registry
Restore the previously authorized 8fd25ac admission fix in the active source branch. The qualification dispatcher consumes it; the first-crown arrival cutoff is deleted. Measurement baseline and the 1.5 percent reward rule remain unchanged. Validation: 73 focused CPU tests in isolated worktree; docs checks and strict build; LOC, assertion, reachability, and surface checks. Production +5/-15 (net -10), tests +24/-24 (net 0), docs +27/-37 (net -10). Exact deployed-source candidate also passed admission against a same-host snapshot of the rejected submission without changing retained evidence or payment state.
The /api/arenas consumer now resolves allocation aliases through their intake database and shows unallocated arenas at zero. This supersedes the exact-key/all-sources assumption that hid the DeepSeek-only allocation. Validation: 26 focused tests passed during the live fix; LOC and assertion ratchets passed. Production +14/-1 (net +13); tests +12/-0 (net +12). Internal display correction; no public command or schema change.
…by several paths The DeepSeek-V4.1 fork image (sha256:19871e2a..., 2026-10-12) registers each decoder layer's successor as a `_next_layer` submodule for its layer-boundary fusion. The deduplicating `named_modules()` listing then names layer 3 only as `model.layers.1._next_layer._next_layer`, so `bind()` matched `model.layers.*.mlp` and `model.layers.*.self_attn` against 3 of 40 layers (0, 1 and 14) on both TP ranks of a served engine; every arena bundle on that image would have bound 3 layers. `bind()` now lists every path (`remove_duplicate=False`), binds a module once per slot, and treats a second slot reaching an already bound module by any path as the existing overlap error. The one-segment address semantics are unchanged, so no documentation changes. Regression test: a two-layer model with `layers[0]._next_layer = layers[1]` binds both `layers.N.mlp` nodes; it failed before the fix. Net lines: production +21/-3, tests +19/-0. No deletion was available in this change.
…ten the listing The regression case (a layer the model reaches first as its predecessor's `_next_layer`) now lives in the existing every-matching-node test, which fails on main's bind() and passes here; the separate test is deleted. bind() lists every path with one dict and keeps the per-slot name match; a second path to an already bound module is the existing overlap error, loud rather than a silent double bind. Net lines against main: production +7/-5 (the incident comment and the identity clash term), tests +3/-0. The listing supersedes no path, so the cacheon and tests ceilings in scripts/loc_baseline.txt rise by exactly 2 and 3. No docs change: the one-segment address semantics are unchanged.
Collaborator
Author
|
Closed: opened from the stale dsv4flash branch after the squash merge of #144, so the diff showed the whole branch history instead of the two-file fix. Reopened clean from main as a fresh branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
bind()incacheon/integrations/sglang_nodes.pymatched slot addresses against the deduplicatingnamed_modules()listing. The DeepSeek-V4.1 fork image (sha256:19871e2a…, 2026-10-12) registers each decoder layer's successor as a_next_layersubmodule for its layer-boundary fusion, so that listing names layer 3 only asmodel.layers.1._next_layer._next_layer. On a served enginemodel.layers.*.mlpandmodel.layers.*.self_attnbound 3 of 40 layers (0, 1 and 14) on both TP ranks. Every arena bundle evaluated on that image would have run on 3 layers and been scored as such.bind()now lists every path (remove_duplicate=False), keeps the one-segment address match, and treats a second path to an already bound module, under any slot, as the existing overlap error (loud, never a silent double bind). The one-segment address semantics are unchanged, so no documentation changes.Evidence
validator-style(deduplicated)=3 … all-paths=40for both slots on both ranks (engine logsclaude-router-pf0114:08Z); the same engine bindsx40per slot with the all-paths listing.test_one_row_binds_every_matching_node_and_a_do_nothing_candidate_passesnow gives the served model alayers[0]._next_layer = layers[1]alias. It fails on main'sbind()(bindslayers.0.mlponly) and passes here.Checks
tests/: 2843 passed, 11 skipped, 3 failed. The 3 aretest_dashboard_metrics.py::test_health_and_queue_use_worker_observation_not_cpu_pulse[*]raisingFileNotFoundError: '/proc'on macOS; they fail identically on main.check_islands.py0 islands,check_assert_ratchet.py6883 ≥ 6437,surface_snapshot.py --check10 sections match (nosurface-change:lines),ruffclean.check_loc_ratchet.py: thecacheonandtestsceilings rise by exactly 2 and 3 inscripts/loc_baseline.txt; the all-paths listing supersedes no path, so no deletion was available.Net lines against main
production +7 / −5, tests +3 / −0, baseline +2 / −2.
Rotation note
The validator must run a commit carrying this before any bundle is qualified on the fork image. Supersedes nothing.