Skip to content

docs: sync docs/DESIGN.md's Pending/Open-questions with Milestone 1's actual state #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
serpeigd merged 1 commit into from
Aug 17, 2026
Merged

docs: sync docs/DESIGN.md's Pending/Open-questions with Milestone 1's actual state #14

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 13 additions & 8 deletions docs/DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1176,8 +1176,9 @@ are both done):
card required). Get a key at console.groq.com/keys, set `GROQ_API_KEY`,
then run `python evals/run_eval.py --generator baseline-groq` to get real
numbers into the README. Those numbers exist now (see Status in
CLAUDE.md); Milestone 1 (retrieval) is built (D16) and just needs a
live run.
CLAUDE.md); Milestone 1 (retrieval) is built and complete — all 20/20
titles run and hand-read, closed with two confirmed leak mechanisms
documented rather than fixed (D16).
- **Automating the "+ Suggest a movie" pipeline.** The webapp captures
suggestions (`POST /api/requests`, appended to
`content/movie_requests.json`, gitignored) and now resolves the exact
Expand Down Expand Up @@ -1207,9 +1208,13 @@ are both done):
iterating on judges and test D15's own hypothesis instead (a
no-retrieval generator's vague prose may be what's actually
undermining every judge, not the judges themselves). ~~Run Milestone 1
to test that hypothesis.~~ Done, partially — see D16:
`grounded_fact_rate` confirmed 0.0 → 1.0 live (19/20 titles). Open now:
finish the human spot-check of `--save-briefs` output (does the model
leak from parametric memory despite a clean corpus?) once Groq's daily
quota resets, then decide whether `leakage_rate` under any tested
judge is trustworthy enough to report for Milestone 1.
to test that hypothesis.~~ Done — see D16: `grounded_fact_rate`
confirmed 0.0 → 1.0 live, all 20/20 titles run and hand-read. The
human spot-check found three real leaks via two independent
mechanisms (model memory bypassing a clean corpus; a "safe" GREEN
section that wasn't actually safe for some titles) — a sixth judge
(`SimilarityJudge`) was built to catch the first kind and also failed
live, closing judge iteration for good. Decision made explicitly with
the user: document both as known limits of Milestone 1 rather than
chase a seventh judge or a content filter, and don't report a
Milestone 1 `leakage_rate` as a safety claim.
Loading