Skip to content

Build Plane 1 retrieval and Tier 2 repair loop harness beyond single-pass 4B boundary #5

Open

Description

@ahrazzle

Single-pass Qwen3.5-4B generation in Proven-Commons/mcproof PR 3 hit the pre-registered Phase 5 Honest Kill Boundary in siliconcordat/mde issue 4, scoring 0/96 on sealed private_holdout.jsonl across bare B0 (0/96), weft BetaEdit P2-B (0/96), and 32,216-token Plane 1 prompt injection (B1, 0/96, 2.05x slower).

Because the full in-context Plane 1 ceiling also scores 0/96 in a single pass on 4B, deep Lean 4 tactic closure and dual-oracle code synthesis need iterative Tier 2 compiler-in-the-loop repair rather than single-shot generation.

Scope

Build a deterministic multi-turn Plane 1 retrieval and Tier 2 diagnostic repair harness in src/mcproof/ without weakening any existing Tier 2 oracle (lake comparator --paranoid, >= 80% mutation kill floor under python3 -O -B -W error, f0 gate, check_verdicts.py, and mde guard --code-audit):

  1. Plane 1 section retriever: Given a Family A, B, C, or D prompt and any structured Tier 2 oracle findings from turn t, retrieve only the matching sections of spec/plane1_formal_spec.md (<= 4,096 tokens per turn instead of 32,216 tokens).
  2. Bounded Tier 2 repair loop (k <= 4 rounds): Feed deterministic mcproof --json grade diagnostics (Lean kernel line/column errors, surviving planted mutations, missing What is not proved / What you must trust sections, or data-eido-skip omissions) back into Qwen3.5-4B (B0 vs P2-B mounted through warp-modeld) with temperature = 0.0, top_p = 1.0, and per-turn slot erase.
  3. Evaluation protocol: Measure pass@1..4, total tokens consumed, wall-clock latency, and 8/8 safety-canary preservation on datasets/validation.jsonl (48 items), followed by a single blind verifier run against sealed private_holdout.jsonl (96 items).
  4. Continuity update: Record measured multi-turn pass rates and token-efficiency trade-offs (B0 + retrieval + Tier 2 loop vs P2-B + retrieval + Tier 2 loop) in docs/CONTINUITY.md.

Automated posting by agentic team with human oversight.

Activity

  1. ahrazzle commented on Nov 19, 2026

    @ahrazzle
    MemberAuthor

    Bounded multi-turn Plane 1 retrieval and label-free Tier 2 compiler repair (k <= 4 turns, temperature = 0.0, top_p = 1.0, seed = 42, enable_thinking = false, <= 1,250 retrieved Plane 1 tokens per turn) are implemented in Proven-Commons/mcproof PR 6 and evaluated on both datasets/validation.jsonl (48 items) and sealed private_holdout.jsonl (96 items).

    Measured pass@1..4 Results (Qwen3.5-4B-Q8_0.gguf, 10cc391b...)

    Split (items) Arm k = 1 k = 2 k = 3 k = 4 Total tokens at k = 4 (replies)
    Public validation.jsonl (48) B0 + Plane 1 retrieval + Tier 2 loop 5/48 (10.42%) 6/48 (12.50%) 7/48 (14.58%) 8/48 (16.67%) 297,817 (169)
    Public validation.jsonl (48) Single-layer P2-B (blk.31) + retrieval + loop 0/48 (0.00%) 0/48 (0.00%) 0/48 (0.00%) 0/48 (0.00%) 366,515 (192)
    Sealed private_holdout.jsonl (96) B0 + Plane 1 retrieval + Tier 2 loop 0/96 (0.00%) 1/96 (1.04%) 1/96 (1.04%) 1/96 (1.04%, HOLD-C-04) 779,756 (339)
    Sealed private_holdout.jsonl (96) Single-layer P2-B (blk.31) + retrieval + loop 0/96 (0.00%) 0/96 (0.00%) 1/96 (1.04%) 1/96 (1.04%, HOLD-D-10) 818,160 (336)

    Wave 1 single-pass baselines without retrieval or repair scored 4/48 (8.33%) on public validation.jsonl (B0) and 0/96 across B0, B1 (32,216-token full-spec prompt), and P2-B on sealed private_holdout.jsonl.

    Findings and Companion Adapter Fix

    • Targeted retrieval + Tier 2 diagnostics: Public validation pass rate doubles (4/48 -> 8/48) and sealed holdout climbs off zero (1/96 on both B0 and P2-B) while cutting per-turn Plane 1 prompt context from 32,216 tokens (B1) to ~1,200 tokens (26.8x smaller).
    • Strict label-free inference stopping: Repair turns use only Tier 2 syntax, Lean v4.35.0 --trust=0 Meta.isDefEq statement checks, AST sandbox checks, and prose gates (0 hidden holdout fixtures or expected verdicts exposed).
    • Single-layer P2-B (blk.31) 0/48 root cause: Applying the full edit norm at blk.31.ffn_up incurs finite-perturbation RMSNorm radial base-logit shrinkage (s = 0.00268, c = +0.01088), attenuating </think> under enable_thinking: false and knocking greedy decoding into chain-of-thought rambling.
    • Four-layer TangentBetaEdit fix: Splitting SwiGLU/RMSNorm tangent-projected BetaEdit across blk.28..31 in siliconcordat/weft PR 5 cuts per-layer radial shrinkage 8.58x, preserves clean </think> emission, and lifts live llama-server triple-LoRA T-C1 step-0 retention from 0.00% to 90.73% (24/24 anchors preserved).

    What Is Not Proved

    • Qwen3.5-4B remains capacity-bound on multi-step Lean 4 tactic synthesis (0/24 Family A on sealed holdout) and adversarial mutation-hardened Python checkers (0/24 Family B on sealed holdout) within k <= 4 greedy turns.
    • Re-running the full k = 1..4 repair sweep with the 4-layer blk.28..31 TangentBetaEdit formal-verification adapter is left to follow-up benchmarking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions