docs: autonomous iteration & self-correction methodology (No Silent Degradation) + Mac launcher - #145
Merged
FluffyAIcode merged 3 commits intoJun 17, 2026
Conversation
…egradation) + Mac launcher Root-causes the month-long verifier-only dead end (silent fallback/simplification: proposer/f_θ bypassed while keeping the 'fused' label) and gives an automatable, self-correcting iteration loop to make it impossible to repeat: - §0 one rule: No Silent Degradation — system under test is the intended system or the run is INVALID. - §4 machine-checkable liveness contract (blocks>0 proves proposer; f_theta_ran proves f_θ; fallbacks_taken==[]; recall floor; KV bounded) asserted by a fail-loud gate (generalize k3_report_gate). - §3 run→gate→diagnose→re-run loop with only PASS/FAIL/BLOCKED (no 'simplified-and-done'); §5 agent rules; §6 wiring; §7 living summary. Also restores scripts/run_kakeya_mac.sh + mlx-kakeya-launcher-smoke (left behind by the #144 squash). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…adation (proposer/f_θ/fallback) Turns the methodology's §4 from doc into an executable defense: - k3_report_gate.assert_liveness(): asserts proposer ran (blocks>0), f_θ ran when intended (f_theta_ran on every turn), and fallbacks_taken==[] — from RUNTIME signals, not from flags. Missing liveness field = violation. New codes: PROPOSER_NEVER_RAN, FTHETA_NOT_RUN, SILENT_FALLBACK, MISSING_LIVENESS. - validate_report() dispatches liveness reports; validate_k3_reports.py (CI + the Mac-bridge on-device gate) now gates kind=mac_gemma4_kakeya_fused_chat. - harness emits f_theta_intended + fallbacks_taken in the chat report. - fused-chat presets set validate_reports=True → the Mac runner FAILS if the engine silently degraded to verifier-only. - 100% coverage on k3_report_gate + manifest; walker verified to FAIL a degraded report (blocks=0 / f_θ bypassed) and pass a live one. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
… liveness gate cannot The full Mac engine produced GARBAGE + throughput collapse (0.31 tok/s) on a long answer (PoW, 780 tok) while the §4 liveness gate PASSED (blocks=340>0, f_θ ran) — liveness is necessary but NOT sufficient. Root cause (confirmed in code comment): restoration is prefill-amortized for the prompt, covering only <= window (64) decode tokens; longer generations evict unrestored positions → degenerate. assert_quality() adds: - RESTORATION_COVERAGE: restored run with tokens > window (the structural cause). - OUTPUT_DEGENERATE: runaway-repeat text (the symptom). validate_report dispatches liveness reports through assert_liveness + assert_quality. Verified: a PoW-style report (tokens=780 > window=64, repeated '* * *') now FAILS the walker (CI + on-device) with both codes. 100% coverage. Doc §1/§4b/§7 updated; the engine fix (continuous decode-time restoration) scoped as open work. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
FluffyAIcode
marked this pull request as ready for review
June 17, 2026 11:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Root-causes the ~month-long failure where dev ran on an effectively verifier-only config (proposer/f_θ silently bypassed), AND a newly-surfaced failure: the full engine produces garbage + throughput collapse on long generation. Delivers the methodology and turns its §4/§4b into an executable, fail-loud gate so neither can recur silently.
1. Methodology —
docs/kakeya-autonomous-iteration-and-self-correction.md§0 No Silent Degradation. §1 anti-pattern catalog (A–E) +
git -Sforensics. §2 North Star invariants. §3 self-correcting loop (PASS/FAIL/BLOCKED). §4 liveness contract. §4b quality contract. §5 agent rules. §6 wiring. §7 living summary + known limitation.2. §4 liveness gate (proposer/f_θ/fallback) — implemented, wired, verified
assert_liveness(): proposer ran (blocks>0), f_θ ran when intended (f_theta_ran),fallbacks_taken==[], from runtime signals. Verified: local walker fails a degraded report (exit 1) & passes a live one; Mac M4 on-device gate PASSED a live run (evidence_gate_exit_code=0,f_theta_ran=True,blocks=2/4).3. §4b quality gate (the long-decode failure liveness can't see) — NEW
The full engine produced garbage +
0.31 tok/son a 780-token answer while the liveness gate passed (blocks=340>0, f_θ ran). Root cause confirmed in code: restoration is prefill-amortized for the prompt, covering only ≤ window (64) decode tokens; longer generations evict unrestored positions → degenerate.assert_quality()adds:RESTORATION_COVERAGE— restored run withtokens > window(the structural cause).OUTPUT_DEGENERATE— runaway-repeat text (the symptom).Verified: a PoW-style report (
tokens=780 > window=64, repeated* * *) now FAILS the walker (CI + on-device) with both codes (exit 1).kakeya_quality_gate_catches_degeneration.log
4. Also (completes #144)
scripts/run_kakeya_mac.shlauncher +mlx-kakeya-launcher-smokepreset.Testing
pytest tests/inference_engine/bench/test_k3_report_gate.py tests/inference_engine/bridge/test_manifest.py— 81 passed; 100% coverage on both modules.pytest -m integration on Mac M4red — pre-existing self-hosted-runner infra issue, unrelated.To show artifacts inline, enable in settings.