Repository navigation
Add mde_soko evaluation domain pack, BetaEdit compiler, and S0-S3 benchmark - #62
Merged
Merged
Conversation
…d LINT_PASS (#53 A3/A4/A6/A7)
Open
25 tasks
zero-aska
requested changes
Nov 19, 2026
zero-aska
left a comment
Collaborator
There was a problem hiding this comment.
Request changes at 3ad8686 (cloud review bc-2e58906b plus a box check).
- The PR says it makes no TCB edits, but it does. The diff changes
flat_oracle.py,cpp/soko_engine.cppandSHA256SUMS. Either drop those changes or describe them and justify them as TCB changes. - The live receipt is stale. It is pinned to 382e9da, and the II-4 mismatch it reports is already fixed at this head. Re-run it at the final head, or mark it in the receipt and PR body as historical (382e9da, superseded) so it isn't read as current evidence.
- The branch conflicts with main. Main is now 3ea918f. The conflicts are in
SHA256SUMSandcpp/soko_engine.cpp, which #60 also changes. Rebase or merge main after #60 lands, regenerateSHA256SUMSonce, and keep #60's corpse-render parity fix.
#60 is approved on substance and is waiting on the same conflict. It goes first.
Regenerate lineage.json for updated SHA256SUMS, shorten README prose for prose_gate limits, and re-derive P2-B binding/GGUF metadata after TCB digest change. Co-authored-by: ahraz <ahrazzle@users.noreply.github.com>
zero-aska
approved these changes
Nov 19, 2026
zero-aska
left a comment
Collaborator
There was a problem hiding this comment.
Approve at 6f891c2. Cloud review bc-2e58906b, run 2. My changes request 5985052037 is resolved.
- TCB edits. The PR body now states the TCB edits. Against #60's head 537e59e, this PR only adds
mde_soko/andtests/test_mde_soko.py; its TCB changes are #60's. - Stale receipt. The live receipt is marked
evidence_status: historical_archive, pinned to 382e9da, from before the II-4 corpse fix. The live tables are labelled historical, and no new numbers were invented. - Conflicts. They were resolved by merging #60, with SHA256SUMS regenerated.
| Check | Result |
|---|---|
integrity |
PASS 21/21 |
mde_soko verify |
ok, 9/9 re-derivation |
| Targeted pytest | 63/63 |
unittest |
316 ran |
The unittest run shows the same 3 C++ compile failures as main.
A live S0–S3 rerun still needs the Q8_0 GGUF and llama-server; that is open follow-up work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
mde_soko/, a Sokoban evaluation domain pack (mde:soko-eval), deterministic CPUfloat64adapter compiler, and four-mode benchmark runner, with 8unittestandpytestunit tests. None of the 21 checker and prover source files locked in rootSHA256SUMSare modified.What it includes
mde_soko/spec/):plane1_soko_spec.md,zone-manifest.soko_eval.v0.1.json, andverify_soko_lineageencode reversible 1-key probing, PuzzleScript rule order (all-switch doors and Wizard ray-first swap), bare JSON, 4.0s zero-import stdlib solvers, 1-step-consistent lower bounds, and dual-oracle trajectory checks.mde_soko/datasets/): splits all 66 trilogy rooms with zero overlap acrosscompile.jsonl(32rooms),validation.jsonl(16rooms), andbenchmark_18_rooms.json(18rooms, zero spoiler routes), plushybrid_k0.jsonl(512noise draws +24y = 0anchors) andSHA256SUMS.DomainOracleand 4.0-second sandbox (mde_soko/domain_oracle.py):SokoDomainOracleseparates inference checking (grade_candidate, zero hidden labels) from post-hoc scoring (grade_item).execute_sandbox_cellbanssoko_mcp/os/sys/openand kills timeouts at 4.0s.audit_soko_certificatechecks failure modesF01throughF08.float64BetaEditcompiler (mde_soko/compiler.py): compiles rank-8blk.31.ffn_up.weightadapters fromharvested_soko_activations.npz(908,726bytes) intoGGUF v3F16 LoRAs andweft-slab/v0.1slabs (f16,int4/int8,int4/f16), checked by a pre-registered integerGatePolicyand strict integer JSON parser.mde_soko/prose_gate.py,mde_soko/CROSS_POLLINATION.md): checks markdown and duplicate blocks without home-directory dependencies, and documents transfers acrossmcproof,weft,siliquant,AITrafficControlBench,integer-mult-bounds,stdhttp-verify, andexoplanets.Certificate and re-derivation results
All 9
P2-Bartifacts re-derive with0-byte diff in a fresh temporary directory usingnumpyonly (verify_zero_byte_rederivation). Under the fixed pre-registeredGatePolicy(max_frob_sq = 25.0,max_spectral_sq = 18.0,max_leakage = 35.0,max_safety_dot_sq = 5e-4):||dW||_F^2K0leakage (f16)K0leakage (int4/int8)K0leakage (int4/f16)P2-B(BetaEditnull-space)0.2288315.29e-54.12e-35.16e-55.96e-83/3slabs)P2-A(unprojected, norm-matched)0.228833330.871< 1e-6leakage_energy_exceeds_cap)Sabotage(low-norm rank-4 control)5.062952447248.180.300256leakage+safety_dot)Asymmetric
int4/f16factor quantization reducesK0leakage by80xrelative to symmetricint4/int8while shrinking slab size from189,083bytes (f16) to78,511bytes.Live
Qwen3.5-4BevaluationMeasured locally on
Qwen3.5-4B-Q8_0.gguf(llama-server,temperature = 0.0,top_p = 1.0, per-slot erase across Gated DeltaNet layers, full dynamic provenance inmde_soko/results/soko_eval_live_receipt.json):step0off-register retention (24y = 0anchors):P2-Bkeeps24/24(100.0%) top-1 agreement withB0(mean_abs_top1_logprob_delta = 0.000533), compared with20/24(83.33%,mean_abs_top1_logprob_delta = 0.500661,939xhigher shift) for norm-matched unprojectedP2-A.validation.jsonl(16rooms,1-shot open-loop):B0scores4/16(25.0%,4/4onH_RULE), whileP2-AandP2-Beach score1/16(6.25%,1/4onH_RULEand0/12onH_ROUTE/H_PROBE/H_CELL).240-key unsolved horizon cap):S0(B0open-loop)0/18(0.0%)11/18bare JSON,11GOALS_NOT_MET4320(240.0mean)[1, 0, -4320]S1(P2-Bopen-loop)0/18(0.0%)11/18bare JSON,11GOALS_NOT_MET4320(240.0mean)[1, 0, -4320]S2(P2-Bstateful probe loop)1/18(5.6%,I-1)18/18(100%) probed,5fatal steps rolled back4088(227.1mean)[1, 1, -4088]S3(P2-B4.0s Python cell,k <= 4)0/18(0.0%)2PREMATURE_WIN(I-1),24.0stimeouts,1AST import block4320(240.0mean)[1, 0, -4320]II-4("RRR"):S3Turn 1 triggeredDUAL_ORACLE_MISMATCHwhen FighterFdied to GoblinO. BothPuzzleScriptOracleandFlatBitmaskOraclereportedalive == Falseand identicaltraj_fnv64, whilePuzzleScriptOraclerendered corpse'X'(#....XO.#...#) andFlatBitmaskOraclerendered'F'(#....FO.#...#).verify_route_dual_strictrejected the route fail-closed without modifying lockedsoko_mcp/flat_oracle.py.Validation
python3 -m soko_mcp.integrity:source_integrity_verified: true(21/21files verified).python3 -O -B -m unittest discover -s tests -p 'test_mde_soko.py' -v:8testsOK(7pure-stdlib tests run withoutnumpyorpytest, plus9/9float64re-derivation withnumpy).python3 -B -m mde_soko verify:ok: true(9/90-byte re-derivation,5/5dataset digests,0prose or path-leak findings).mde guard --code-audit mde_soko tests/test_mde_soko.py:0findings.What Is Not Proved
FixedRadixCertificatebounds post-quantization adapter Frobenius norm, spectral norm,K0covariance leakage, and safety-direction inner products. It does not prove puzzle route optimality or replace dual-oracle trajectory replay.4Bparameter scale, single-pass open-loop generation (S0,S1) solves only short opening rooms without external search or environment feedback. Solving multi-step Sokoban rooms requires closed-loop board interaction (S2) or bounded code-execution synthesis (S3) backed by Tier 2 dual-oracle verification.Adds
mde_soko/, a Sokoban evaluation domain pack (mde:soko-eval), deterministic CPUfloat64adapter compiler, and four-mode benchmark runner, withtests/test_mde_soko.py(9 tests) plus locked-TCB integration described below.TCB edits (intentional): this branch updates three locked manifest entries so dual-oracle fatal-step replay matches across Python and C++ (
soko_mcp/flat_oracle.pycorpseXrendering,cpp/soko_engine.cppOBJ_CORPSEparity and portable C++20 headers, and regenerated rootSHA256SUMSviapython3 -m soko_mcp.integrity --write). These align with #60 verification discipline and are required for honestmde_sokoverify_route_dual_strict/S2–S3grading.#60is merged into this branch.What it includes
mde_soko/spec/):plane1_soko_spec.md,zone-manifest.soko_eval.v0.1.json, andverify_soko_lineageencode reversible 1-key probing, PuzzleScript rule order (all-switch doors and Wizard ray-first swap), bare JSON, 4.0s zero-import stdlib solvers, 1-step-consistent lower bounds, and dual-oracle trajectory checks.mde_soko/datasets/): splits all 66 trilogy rooms with zero overlap acrosscompile.jsonl(32rooms),validation.jsonl(16rooms), andbenchmark_18_rooms.json(18rooms, zero spoiler routes), plushybrid_k0.jsonl(512noise draws +24y = 0anchors) andSHA256SUMS.DomainOracleand 4.0-second sandbox (mde_soko/domain_oracle.py):SokoDomainOracleseparates inference checking (grade_candidate, zero hidden labels) from post-hoc scoring (grade_item).execute_sandbox_cellbanssoko_mcp/os/sys/openand kills timeouts at 4.0s.audit_soko_certificatechecks failure modesF01throughF08.float64BetaEditcompiler (mde_soko/compiler.py): compiles rank-8blk.31.ffn_up.weightadapters fromharvested_soko_activations.npz(908,726bytes) intoGGUF v3F16 LoRAs andweft-slab/v0.1slabs (f16,int4/int8,int4/f16), checked by a pre-registered integerGatePolicyand strict integer JSON parser.mde_soko/prose_gate.py,mde_soko/CROSS_POLLINATION.md): checks markdown and duplicate blocks without home-directory dependencies, and documents transfers acrossmcproof,weft,siliquant,AITrafficControlBench,integer-mult-bounds,stdhttp-verify, andexoplanets.Certificate and re-derivation results
All 9
P2-Bartifacts re-derive with0-byte diff in a fresh temporary directory usingnumpyonly (verify_zero_byte_rederivation). Under the fixed pre-registeredGatePolicy(max_frob_sq = 25.0,max_spectral_sq = 18.0,max_leakage = 35.0,max_safety_dot_sq = 5e-4):||dW||_F^2K0leakage (f16)K0leakage (int4/int8)K0leakage (int4/f16)P2-B(BetaEditnull-space)0.2288315.29e-54.12e-35.16e-55.96e-83/3slabs)P2-A(unprojected, norm-matched)0.228833330.871< 1e-6leakage_energy_exceeds_cap)Sabotage(low-norm rank-4 control)5.062952447248.180.300256leakage+safety_dot)Historical live
Qwen3.5-4Bevaluation (not current head evidence)mde_soko/results/soko_eval_live_receipt.jsonis an historical archive (evidence_status: historical_archive,provenance.git_head_sha=382e9dafbe4ce500f454feafcae1a7e7c859333a). It is not automatically current evidence for PR head: refresh withpython3 -B -m mde_soko eval --base-gguf <Qwen3.5-4B-Q8_0.gguf>andllama-serverwhen hardware is available. The tables below quote that archive only; numbers are unchanged from the original capture.step0off-register retention (24y = 0anchors):P2-Bkeeps24/24(100.0%) top-1 agreement withB0(mean_abs_top1_logprob_delta = 0.000533), compared with20/24(83.33%,mean_abs_top1_logprob_delta = 0.500661,939xhigher shift) for norm-matched unprojectedP2-A.validation.jsonl(16rooms,1-shot open-loop):B0scores4/16(25.0%,4/4onH_RULE), whileP2-AandP2-Beach score1/16(6.25%,1/4onH_RULEand0/12onH_ROUTE/H_PROBE/H_CELL).240-key unsolved horizon cap) (historical receipt):S0(B0open-loop)0/18(0.0%)11/18bare JSON,11GOALS_NOT_MET4320(240.0mean)[1, 0, -4320]S1(P2-Bopen-loop)0/18(0.0%)11/18bare JSON,11GOALS_NOT_MET4320(240.0mean)[1, 0, -4320]S2(P2-Bstateful probe loop)1/18(5.6%,I-1)18/18(100%) probed,5fatal steps rolled back4088(227.1mean)[1, 1, -4088]S3(P2-B4.0s Python cell,k <= 4)0/18(0.0%)2PREMATURE_WIN(I-1),24.0stimeouts,1AST import block4320(240.0mean)[1, 0, -4320]II-4"RRR", pre-corpse TCB at382e9daf): the archived receipt loggedDUAL_ORACLE_MISMATCHbecauseFlatBitmaskOraclestill rendered'F'on fatal steps. After the TCB corpse fix (now on this branch, merged from Enforce state budgets, post-death run_late parity, must_fail gate, and LINT_PASS (#53 A3/A4/A6/A7) #60), both oracles render'X'and agree (tests/test_flat_oracle_corpse.py).Validation
python3 -m soko_mcp.integrity:source_integrity_verified: true(21/21files verified).python3 -O -B -m pytest -p no:cacheprovider tests/test_mde_soko.py:9testsOK(CPUfloat649/9re-derivation withnumpy).python3 -B -m mde_soko verify:ok: true(9/90-byte re-derivation,5/5dataset digests,0prose or path-leak findings).What Is Not Proved
FixedRadixCertificatebounds post-quantization adapter Frobenius norm, spectral norm,K0covariance leakage, and safety-direction inner products. It does not prove puzzle route optimality or replace dual-oracle trajectory replay.4Bparameter scale, single-pass open-loop generation (S0,S1) solves only short opening rooms without external search or environment feedback. Solving multi-step Sokoban rooms requires closed-loop board interaction (S2) or bounded code-execution synthesis (S3) backed by Tier 2 dual-oracle verification.