Skip to content

Commit 83ac391

Browse files
cursoragentFluffyAIcode
cursoragent
and
FluffyAIcode
committed
docs: autonomous iteration & self-correction methodology (No Silent Degradation) + Mac launcher
Root-causes the month-long verifier-only dead end (silent fallback/simplification: proposer/f_θ bypassed while keeping the 'fused' label) and gives an automatable, self-correcting iteration loop to make it impossible to repeat: - §0 one rule: No Silent Degradation — system under test is the intended system or the run is INVALID. - §4 machine-checkable liveness contract (blocks>0 proves proposer; f_theta_ran proves f_θ; fallbacks_taken==[]; recall floor; KV bounded) asserted by a fail-loud gate (generalize k3_report_gate). - §3 run→gate→diagnose→re-run loop with only PASS/FAIL/BLOCKED (no 'simplified-and-done'); §5 agent rules; §6 wiring; §7 living summary. Also restores scripts/run_kakeya_mac.sh + mlx-kakeya-launcher-smoke (left behind by the #144 squash). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent 22847e0 commit 83ac391

4 files changed

Lines changed: 335 additions & 0 deletions

File tree

‎docs/kakeya-autonomous-iteration-and-self-correction.md‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
# Kakeya — Autonomous Iteration & Self-Correction Methodology
2+
3+
**Status:** living charter + methodology. Maintained continuously as the project
4+
evolves. This document exists because of a concrete, expensive failure (see §1)
5+
and its single job is to make that failure **impossible to repeat**.
6+
7+
---
8+
9+
## 0. The one rule
10+
11+
> **No Silent Degradation.** The system under test is the *intended* system, or
12+
> the run is **INVALID** — never "passing with a simpler thing." Every run must
13+
> **prove** which components actually executed, and a gate must **fail loud** if
14+
> any of them silently degraded to a fallback, baseline, mock, or proxy.
15+
16+
Everything below operationalizes this one rule.
17+
18+
---
19+
20+
## 1. The failure this prevents (why this document exists)
21+
22+
The Kakeya engine is a **verifier + proposer + f_θ** architecture whose purpose is
23+
**bounded memory with no sacrifice to intelligence (recall) or token throughput**.
24+
Over ~a month, development silently ran on a degraded configuration: the proposer
25+
and/or f_θ were **bypassed** while the run kept the original "fused" label, so the
26+
effective system was **verifier-only**. The work looked like progress; it was on a
27+
dead branch.
28+
29+
How it slipped through — the **silent-fallback anti-pattern**, in its observed forms:
30+
31+
| # | Degradation | What was claimed | The tell (ignored) |
32+
| --- | --- | --- | --- |
33+
| A | proposer bypassed → native AR | "fused spec-decode" | `blocks=0` on every sample |
34+
| B | f_θ bypassed under S5 ("free lunch" smoke opt) | "restoration engine" | `build_restoration` returns `{}`; no f_θ forward |
35+
| C | a proxy/plumbing run | "engine validated" | wrong model (Qwen3-4B), no trained f_θ/proposer, prompt inside window |
36+
| D | a simpler component shipped | "the engine" | verifier-only AR chat presented as the product |
37+
38+
Common root cause: an agent (or optimization) chose the **easy/robust path** and
39+
**relabeled it as the hard one**, and no automated check asserted the intended
40+
components actually ran. The numbers (latency, even partial correctness) looked
41+
fine, so the substitution went unnoticed.
42+
43+
**Forensic note (how to find when degradation entered):** `git log -S "<symbol>"`
44+
on the bypass markers pinpoints it. (Here: f_θ S5-bypass entered 2026-06-12 in
45+
`b3a04d0` *"Optimize MLX adaptive S5 native smoke path"*; the proposer `blocks=0`
46+
silent bypass was caught later by `0a6fb19` *"Evidence gate"* which added
47+
`--force-fused-specdecode`.) Always run this when behavior "feels" too easy.
48+
49+
---
50+
51+
## 2. Development goals (the North Star — the invariants that define "real")
52+
53+
The engine is "real" only if **all** of these hold simultaneously:
54+
55+
1. **Bounded KV** — resident KV footprint does not grow with conversation length
56+
(sink+window resident; evicted context reconstructed on demand).
57+
2. **Proposer live** — the dLLM proposer (DFlash) drafts blocks the verifier
58+
accepts (speculative decode), not native AR.
59+
3. **f_θ live (where load-bearing)** — f_θ projects proposer hidden → verifier
60+
K/V for the restored layers. On gemma-4 it is recall-irrelevant (the 5 exact
61+
layers carry recall — "S5 free lunch") but must still **execute** when the full
62+
pipeline is the system under test; on **full-attention models it is the only
63+
way to bound memory at full recall**.
64+
4. **No intelligence loss** — recall preserved (NIAH / task recall ≥ baseline).
65+
5. **No throughput loss** — token throughput meets the platform target
66+
(CUDA: spec-decode > AR; Mac: ≈AR is the honest ceiling, memory is the win).
67+
68+
A run that achieves (1) by dropping (2)/(3), or (4)/(5) by dropping (1), is **not
69+
the engine** — it is a degraded baseline and must be labeled and gated as such.
70+
71+
---
72+
73+
## 3. The self-correcting autonomous iteration loop
74+
75+
```
76+
┌────────────────────────────────────────────────────────────┐
77+
│ 0. DECLARE the liveness contract for this run (intended │
78+
│ components + invariant thresholds). §4. │
79+
└───────────────────────────┬────────────────────────────────┘
80+
▼
81+
┌────────────────────────────────────────────────────────────┐
82+
│ 1. RUN — and emit a machine-checkable EXECUTION MANIFEST: │
83+
│ not just outputs, but liveness flags for every component │
84+
│ (did the proposer run? did f_θ run? is it a baseline?). │
85+
└───────────────────────────┬────────────────────────────────┘
86+
▼
87+
┌────────────────────────────────────────────────────────────┐
88+
│ 2. GATE — assert the contract against the manifest. │
89+
│ ANY degraded/missing component → run is INVALID (fail │
90+
│ loud), NOT "passing with caveats". §4. │
91+
└───────────────┬───────────────────────────┬──────────────────┘
92+
PASS ▼ FAIL ▼ (or INCONCLUSIVE)
93+
┌──────────────────────┐ ┌──────────────────────────────────────┐
94+
│ 3a. RECORD evidence + │ │ 3b. DIAGNOSE: which invariant failed, │
95+
│ honest scope; advance │ │ which component degraded, why. Form a │
96+
│ the milestone (PR). │ │ hypothesis. Instrument. Re-run (→1). │
97+
└──────────────────────┘ │ Repeat until contract holds OR ... │
98+
│ ... escalate with status = BLOCKED │
99+
│ (never substitute a simpler system). │
100+
└──────────────────────────────────────┘
101+
```
102+
103+
### Status vocabulary (only these three; no fourth "simplified-and-done")
104+
- **PASS** — contract fully satisfied on the *intended* system; evidence attached.
105+
- **FAIL** — a contract invariant is violated → diagnose + iterate.
106+
- **BLOCKED** — cannot run the intended system (env/dep/training missing). Say so
107+
explicitly; do **not** swap in a simpler system and call it progress.
108+
109+
---
110+
111+
## 4. The liveness contract (machine-checkable; the heart of self-correction)
112+
113+
Every run emits an **execution manifest** — a JSON of *what actually executed* —
114+
and a gate asserts it. For the Kakeya engine the contract is:
115+
116+
| Invariant | Manifest field (emit it) | Gate assertion | Already emitted? |
117+
| --- | --- | --- | --- |
118+
| system_under_test is intended | `system_under_test` | `== intended` (not `native_ar_baseline`) | yes (`adaptive_mode`/label) |
119+
| proposer ran | `blocks`, `mean_accept_len` | `blocks > 0 and mean_accept_len > 0` | **yes** (fused res) |
120+
| f_θ ran (when intended) | `f_theta_ran`, `f_theta_layers` | `f_theta_ran == True and len(layers) > 0` | **yes** (chat `_gen_turn`) |
121+
| restoration active | `restoration_active` | `== True` (unless explicitly native baseline) | yes (eval rows) |
122+
| recall preserved | `recall` | `>= recall_floor` | yes (NIAH) |
123+
| KV bounded | `resident_kv_bytes`, `kv_grows_with_ctx` | resident ≈ const across turns/ctx | partial — emit `kv_grows_with_ctx` |
124+
| no fallback/mock taken | `fallbacks_taken` (list) | `== []` | **ADD** — components log any fallback |
125+
126+
Rules for the manifest:
127+
- **Liveness is asserted from runtime signals, not from flags passed in.** "I
128+
passed `--fused-specdecode`" is not evidence; `blocks>0` is.
129+
- **A missing liveness field is a FAIL, not a skip.** Absence = "we don't know it
130+
ran" = invalid.
131+
- **Any component that falls back MUST record it** in `fallbacks_taken`; a
132+
non-empty list with `allow_fallback=False` fails the gate. This is the direct
133+
antidote to silent simplification.
134+
135+
The existing evidence gate (`inference_engine/bench/k3_report_gate.py`,
136+
`--force-fused-specdecode`) is the seed of this — generalize it to assert the full
137+
contract above and reject degraded runs in CI **and** in the agent loop.
138+
139+
---
140+
141+
## 5. Agent operating rules (behavioral — for any agent, incl. me)
142+
143+
1. **Never fallback/simplify/mock silently.** If the intended system can't run,
144+
report **BLOCKED** with the exact blocker — do not substitute a simpler system
145+
and present it as the deliverable.
146+
2. **Every claim cites runtime evidence.** "Validated/works/done" requires the
147+
execution manifest + the gate verdict, not "it compiled" or "it ran" or "the
148+
homepage loaded." Plumbing/smoke ≠ engine validation — label it precisely.
149+
3. **Verify against the liveness contract, not against "it produced output."** A
150+
correct-looking answer from a degraded system is the most dangerous outcome.
151+
4. **Test the intended config on the intended model.** A proxy (smaller/different
152+
model, untrained component) proves the proxy, not the engine — state the gap.
153+
5. **Detect your own degradation.** Before claiming progress that "felt easy," run
154+
the forensic check (`git -S` on liveness markers) and the liveness gate.
155+
6. **Proactively reconcile with the repo.** Check `main` / PR / branch state
156+
yourself; don't make the user tell you what merged.
157+
7. **One status, honestly.** PASS / FAIL / BLOCKED (§3). Never invent a fourth.
158+
159+
---
160+
161+
## 6. How to automate it (wiring)
162+
163+
- **Emit:** each run path writes the §4 execution manifest (the fused engine
164+
already emits `blocks`/`mean_accept_len`/`f_theta_ran`/`f_theta_layers`/
165+
`resident_kv_bytes`; add `fallbacks_taken` + `kv_grows_with_ctx`).
166+
- **Gate:** extend `k3_report_gate.validate_report` to assert the full liveness
167+
contract; wire into CI and the Mac-bridge `validate_reports` path so a degraded
168+
run **fails the job**, not silently passes.
169+
- **Loop driver:** a thin runner does `run → gate → (diagnose → instrument →
170+
re-run | record-PASS | escalate-BLOCKED)`. On Mac, "run" = a bridge preset whose
171+
report is gate-checked on-device; on CUDA, the Vast harness + gate.
172+
- **Regression tripwire:** a CI check that fails if a liveness field that was
173+
`True` flips to `False`/absent between commits (catches a future "S5 free lunch
174+
smoke opt" before it merges).
175+
176+
---
177+
178+
## 7. Living summary (updated each iteration)
179+
180+
**Goal:** verifier(gemma-4) + DFlash proposer + f_θ + S5 bounded KV → bounded
181+
memory, full recall, platform-appropriate throughput. Differentiator = bounded-KV
182+
(memory/concurrency density), load-bearing via proposer+f_θ on full-attention
183+
models.
184+
185+
**Process:** milestone = one stacked PR; ADR + report per milestone; Mac via the
186+
git-bus bridge (allowlisted presets, on-device evidence gate), CUDA via Vast;
187+
every milestone gated by §4.
188+
189+
**Current verified state (Mac M4):** full fused engine runs in interactive chat —
190+
proposer live (`blocks=2/4`, `accept_len=4.0/3.5`), f_θ live by default
191+
(`f_theta_ran=TRUE`, 25 sliding layers), correct answers, bounded KV, natural EOS
192+
stop. One-command launcher: `scripts/run_kakeya_mac.sh`. (PR #144 + this PR.)
193+
194+
**Open / next:** generalize the liveness gate (§4/§6) so the engine cannot
195+
silently regress to verifier-only again; full-attention model (Qwen/Llama) where
196+
f_θ is load-bearing for the large memory win.
197+
198+
> Maintenance: append to §7 every iteration; update §4 if new components/
199+
> invariants appear; never delete the §1 failure record — it is the reason for §0.

‎inference_engine/bridge/manifest.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -749,6 +749,28 @@ def _harness_preset(
749749
},
750750
validate_reports=False,
751751
),
752+
Preset(
753+
name="mlx-kakeya-launcher-smoke",
754+
description="Verify the one-command local launcher "
755+
"scripts/run_kakeya_mac.sh runs the engine end-to-end on "
756+
"the Mac: invokes it in --fast scripted mode (all-MLX "
757+
"proposer path) with a fixed prompt and writes a "
758+
"transcript. Proves launcher → harness → engine wiring + "
759+
"env resolution + preflight on the real machine.",
760+
command_templates=(
761+
(
762+
"bash", "scripts/run_kakeya_mac.sh", "--fast",
763+
"--max-new-tokens", "{max_new_tokens}",
764+
"--chat-scripted",
765+
"What is the capital of France? Answer in one short sentence.",
766+
"--output",
767+
"results/research/k3_mac_bridge_launcher_smoke.json",
768+
),
769+
),
770+
timeout_minutes=45,
771+
params={"max_new_tokens": ("int:max_new_tokens", "64")},
772+
validate_reports=False,
773+
),
752774
)
753775
}
754776

‎scripts/run_kakeya_mac.sh‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
#!/usr/bin/env bash
2+
# Run the FULL Kakeya Inference Engine locally on a Mac (Apple Silicon).
3+
#
4+
# Launches an interactive chat on the complete engine:
5+
# gemma-4 verifier (MLX) + DFlash proposer (fused spec-decode)
6+
# + f_θ K/V restoration + S5 bounded KV.
7+
# f_θ runs by DEFAULT (the full verifier/proposer/f_θ pipeline). Use --fast for
8+
# the all-MLX proposer path (f_θ bypassed via S5 native prefill — much faster on
9+
# Mac, but the f_θ projection does not execute).
10+
#
11+
# Model facts come from env vars (set on the kakeya-mac-m4 runner), with sane
12+
# fallbacks; override on the CLI if needed:
13+
# KAKEYA_MAC_VERIFIER_PATH local MLX gemma-4 dir
14+
# KAKEYA_MAC_DRAFTER_ID DFlash drafter repo/dir
15+
# KAKEYA_MAC_FTHETA_DIR trained f_θ projection dir
16+
#
17+
# Usage:
18+
# bash scripts/run_kakeya_mac.sh # full engine (f_θ on), interactive
19+
# bash scripts/run_kakeya_mac.sh --fast # proposer-only (f_θ bypassed), faster
20+
# bash scripts/run_kakeya_mac.sh --max-new-tokens 2048 --window 128
21+
# bash scripts/run_kakeya_mac.sh --dry-run # print the command, run nothing
22+
# echo 'Explain proof-of-work.' | bash scripts/run_kakeya_mac.sh # one-shot via stdin
23+
set -euo pipefail
24+
25+
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
26+
cd "$repo_root"
27+
28+
VERIFIER="${KAKEYA_MAC_VERIFIER_PATH:-$HOME/kakeya-models/gemma-4-26B-A4B-it-mlx-4bit}"
29+
DRAFTER="${KAKEYA_MAC_DRAFTER_ID:-z-lab/gemma-4-26B-A4B-it-DFlash}"
30+
FTHETA="${KAKEYA_MAC_FTHETA_DIR:-results/research/f_theta_v5_s5_sliding}"
31+
SINK="${KAKEYA_SINK:-4}"
32+
WINDOW="${KAKEYA_WINDOW:-64}"
33+
BLOCK="${KAKEYA_BLOCK_SIZE:-4}"
34+
MAX_NEW="${KAKEYA_MAX_NEW_TOKENS:-1024}"
35+
36+
FAST=0
37+
DRY_RUN=0
38+
EXTRA=()
39+
while [[ $# -gt 0 ]]; do
40+
case "$1" in
41+
--fast) FAST=1 ;;
42+
--dry-run) DRY_RUN=1 ;;
43+
--verifier-path) shift; VERIFIER="${1:?}" ;;
44+
--drafter-id) shift; DRAFTER="${1:?}" ;;
45+
--f-theta-dir) shift; FTHETA="${1:?}" ;;
46+
--max-new-tokens) shift; MAX_NEW="${1:?}" ;;
47+
--window) shift; WINDOW="${1:?}" ;;
48+
--sink) shift; SINK="${1:?}" ;;
49+
--block-size) shift; BLOCK="${1:?}" ;;
50+
-h|--help) sed -n '2,28p' "$0"; exit 0 ;;
51+
*) EXTRA+=("$1") ;; # pass-through (e.g. --chat-scripted ...)
52+
esac
53+
shift
54+
done
55+
56+
log() { echo "[run-kakeya-mac] $*" >&2; }
57+
58+
# ---- argv for the full-engine harness chat ----
59+
args=(
60+
--verifier-path "$VERIFIER"
61+
--drafter-id "$DRAFTER"
62+
--f-theta-dir "$FTHETA"
63+
--s5-exact-full-attn --fused-specdecode
64+
--sink-size "$SINK" --window-size "$WINDOW" --block-size "$BLOCK"
65+
--max-new-tokens "$MAX_NEW" --chat
66+
)
67+
if [[ "$FAST" == "1" ]]; then
68+
# all-MLX proposer + bounded trim: faster, but f_θ is bypassed (S5 free lunch).
69+
args+=( --all-mlx-drafter --cuda-trim )
70+
MODE="FAST (verifier + proposer + S5 bounded KV; f_θ BYPASSED)"
71+
else
72+
# torch drafter + f_θ: the harness auto-enables --force-f-theta in --chat, so
73+
# f_θ projection ACTUALLY RUNS each turn (the full pipeline).
74+
MODE="FULL (verifier + proposer + f_θ + S5 bounded KV; f_θ runs)"
75+
fi
76+
77+
log "mode : $MODE"
78+
log "verifier: $VERIFIER"
79+
log "drafter : $DRAFTER"
80+
log "f_theta : $FTHETA"
81+
log "params : sink=$SINK window=$WINDOW block=$BLOCK max_new=$MAX_NEW"
82+
83+
cmd=( python3 scripts/research/k3_integrated_niah_eval_mac.py "${args[@]}" "${EXTRA[@]}" )
84+
85+
if [[ "$DRY_RUN" == "1" ]]; then
86+
echo "PYTHONPATH=.:sdks/python ${cmd[*]}"
87+
exit 0
88+
fi
89+
90+
# ---- preflight (Apple Silicon + MLX + model) ----
91+
command -v python3 >/dev/null || { log "python3 not found"; exit 1; }
92+
python3 -c "import mlx.core" 2>/dev/null \
93+
|| { log "MLX not importable — this needs Apple Silicon + 'pip install mlx mlx-lm'"; exit 2; }
94+
[[ -d "$VERIFIER" ]] \
95+
|| { log "verifier model dir not found: $VERIFIER (set KAKEYA_MAC_VERIFIER_PATH)"; exit 3; }
96+
if [[ "$FAST" != "1" && ! -e "$FTHETA" ]]; then
97+
log "f_θ dir not found: $FTHETA — set KAKEYA_MAC_FTHETA_DIR, or use --fast (f_θ bypassed)"
98+
exit 4
99+
fi
100+
101+
log "starting... (type a message, blank line / Ctrl-D to quit)"
102+
PYTHONPATH=".:sdks/python" exec "${cmd[@]}"

‎tests/inference_engine/bridge/test_manifest.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ def test_allowlist_contains_exactly_the_documented_presets():
8383
"mlx-kakeya-chat-smoke",
8484
"mlx-kakeya-fused-chat-ftheta",
8585
"mlx-kakeya-fused-chat-smoke",
86+
"mlx-kakeya-launcher-smoke",
8687
"mlx-multitenant-pressure",
8788
"mlx-upgrade",
8889
"mlx-upstream-batch-probe",
@@ -150,6 +151,17 @@ def test_mlx_kakeya_chat_smoke_preset_resolves():
150151
assert not [t for t in argv if t.startswith("{") and t.endswith("}")]
151152

152153

154+
def test_mlx_kakeya_launcher_smoke_preset_invokes_launcher():
155+
request = parse_manifest(_manifest(
156+
preset="mlx-kakeya-launcher-smoke", params={"max_new_tokens": "64"}))
157+
(argv,) = build_commands(request, {})
158+
assert argv[0] == "bash"
159+
assert argv[1].endswith("run_kakeya_mac.sh")
160+
assert "--fast" in argv
161+
assert "--chat-scripted" in argv
162+
assert argv[argv.index("--max-new-tokens") + 1] == "64"
163+
164+
153165
def test_mlx_kakeya_fused_chat_ftheta_preset_runs_f_theta_path():
154166
request = parse_manifest(_manifest(
155167
preset="mlx-kakeya-fused-chat-ftheta",

0 commit comments

Comments
 (0)