From c595e29b2d44abced7a9983b54edadb983185445 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 20:25:14 +0000 Subject: [PATCH] docs: sync README/DESIGN.md against actual repo state - Researched catalogue: 8/20 -> 23 entries (18/20 measurement titles + 5 more); update every count/list that referenced the old 8. - Disclose the "+ Suggest a movie" auto-publish path (no review step, a deliberate risk per app.py's own docstring) -- README implied every path into content/researched/ still went through human review. - CI is currently failing on every push since 2026-08-19 (test_auto_publish.py/ test_groq_retry.py import fastapi/groq/httpx without importorskip, tests.yml never installs them) -- was claiming 18/26 passing. - Correct test count (26 -> 35, verified by running the suite locally). - Note Groq's 2026-08-18 model decommission next to the LLMJudge comparison numbers that used the now-gone model names. - docs/DESIGN.md: add D17 for the auto-publish decision, close the stale Pending bullet claiming it wasn't built yet. Co-Authored-By: Claude Sonnet 5 --- README.md | 147 ++++++++++++++++++++++++++++++++----------------- docs/DESIGN.md | 55 ++++++++++++++---- 2 files changed, 139 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 7795a16..cc8faec 100644 --- a/README.md +++ b/README.md @@ -40,12 +40,13 @@ python webapp/app.py # open http://127.0.0.1:8000 ``` -Pick a researched movie (Sixth Sense, Fight Club, Get Out, Parasite, The -Prestige, Se7en, Arrival, or Gone Girl), read the spoiler-free entry, and -when you're ready, open the curtain. The other 12 titles in the catalogue -show a live TMDB poster/synopsis instead (see [Browse tier](#browse-tier-tmdb)) — no -API key needed for any of this, `TMDB_READ_ACCESS_TOKEN` just upgrades -those placeholders to real posters. +Pick a researched movie — 23 now (Sixth Sense, Fight Club, Get Out, +Parasite, The Prestige, Se7en, Arrival, Gone Girl, and 15 more), read the +spoiler-free entry, and when you're ready, open the curtain. The two +remaining measurement titles show a live TMDB poster/synopsis instead +(see [Browse tier](#browse-tier-tmdb)) — no API key needed for any of +this, `TMDB_READ_ACCESS_TOKEN` just upgrades those placeholders to real +posters. ## What makes this different from "another movie CRUD" @@ -72,8 +73,9 @@ those placeholders to real posters. [Research-assist tool](#research-assist-tool-scaling-the-researched-catalogue)) drafts a new researched entry from real Wikipedia + TMDB retrieval, but every claim still needs a real source URL — a code-level sanitizer strips - any citation the model invents, and a human review gate still sits - between a draft and `content/researched/`. + any citation the model invents, and a human review gate sits between a + draft and `content/researched/` for the CLI workflow (one deliberate + exception exists — see [Limitations](#limitations)). ## Stack @@ -98,10 +100,15 @@ retrieval) · MyMemory (free — on-the-fly Spanish translation, cached) · Upstash Redis free REST API (optional — durable comments/movie-requests on a redeploy; falls back to local files without it). -**CI:** GitHub Actions runs 18 of the 26 tests on every push (see badge -above) — the other 8 (`SimilarityJudge`, `TrainedClassifierJudge`) need -optional `sentence-transformers`/`scikit-learn` and are skipped by design -(`pytest.importorskip`) when those aren't installed, which CI doesn't. +**CI:** currently broken on every push (see badge above, and +[Limitations](#limitations)) — `tests.yml` only installs +`pydantic`/`pytest`/`pyyaml`, but two newer test files import +`fastapi`/`groq`/`httpx` directly instead of through +`pytest.importorskip` the way `test_similarity_judge.py`/ +`test_trained_classifier_judge.py` correctly do, so collection is +interrupted before any test runs. Locally, with every dependency +installed except `sentence-transformers`, 31 of the suite's 35 tests +pass (the rest need that optional package too). ## How it's built @@ -111,23 +118,23 @@ optional `sentence-transformers`/`scikit-learn` and are skipped by design | `webapp/research_assist.py` | Drafts a new researched entry from real Wikipedia + TMDB retrieval (never LLM memory) — see [Research-assist tool](#research-assist-tool-scaling-the-researched-catalogue). | | `webapp/prewarm_translations.py` | One-time build step: caches Spanish translations of researched + browse-tier content so the live deploy never calls the translation API on a visitor's request. | | `webapp/resolve_tmdb_ids.py` | One-time helper: resolves a `tmdb_id` for every title in `evals/dataset/titles.yaml` so the browse tier can show a poster for all 20. | -| `content/researched/*.json` | 8 hand-researched entries (with cited sources: Wikipedia, Hollywood Reporter, No Film School…), not generated by an unverified LLM. | +| `content/researched/*.json` | 23 cited entries: 8 hand-researched (Wikipedia, Hollywood Reporter, No Film School…) plus 15 drafted by `research_assist.py` — most human-reviewed before publishing, but one path isn't (see [Limitations](#limitations)). | | `src/preshow/` | Data contracts (Pydantic) for both the researched content and the measurement harness, plus the TMDB/Wikipedia/translation/KV-store clients. | | `evals/` | The real experiment: leakage/grounding/richness metrics, calibrated judge, 20-title stratified dataset, external calibration scripts. | -| `docs/DESIGN.md` | Every non-trivial design decision (D1–D16) with its trade-off, written as it was made. | +| `docs/DESIGN.md` | Every non-trivial design decision (D1–D17) with its trade-off, written as it was made. | ## Status | What | Status | |---|---| -| Twistify app (catalogue, spoiler gate, filters, comments) | ✅ 8/20 entries researched | -| Browse catalogue (TMDB posters, live search, ES/EN) | ✅ 20/20 have posters, search reaches all of TMDB | -| Offline evals harness | ✅ 18/18 passing in CI, 26/26 with optional `scikit-learn`/`sentence-transformers` installed | +| Twistify app (catalogue, spoiler gate, filters, comments) | ✅ 23 entries researched — 18/20 measurement titles plus 5 beyond that set | +| Browse catalogue (TMDB posters, live search, ES/EN) | ✅ posters for every title in the catalogue, search reaches all of TMDB | +| Offline evals harness | ⛔ CI currently failing (collection error — see [Limitations](#limitations)); 31/35 tests pass locally, 35/35 with optional `sentence-transformers` too | | Spoiler ground truth (20 titles) | ✅ 20/20, LLM-researched with cited sources (never hand-labeled — see [Ground truth, precisely](#ground-truth-precisely) below) | | Baseline generator (no retrieval) | ✅ two providers — Anthropic (paid) and Groq (free tier, no card) | | Judge calibration (offline + real spoiler reviews) | ⛔ **closed, unsolved** — six judges built and tested against real generator output; none clears the bar to trust a `leakage_rate`. `SubstringJudge` (recall=0.0) stays the default because its failure mode is bounded and known (see [Limitations](#limitations)) | | Measure the baseline over the 20 titles | ✅ done — see numbers and caveats below | -| Research-assist tool (D14) | ✅ drafts a researched entry from Wikipedia + TMDB; tested end-to-end on one title (Citizen Kane) | +| Research-assist tool (D14) | ✅ drafted 15 of the 23 researched entries so far (batch-promoted, human-reviewed); a separate, un-reviewed auto-publish path also exists — see [Limitations](#limitations) | | Retrieval (Wikipedia, GREEN-only corpus) + `--generator retrieval-groq` | ✅ `grounded_fact_rate` 0.0→1.0 confirmed live, all 20/20 titles hand-read (D16) | ## Ground truth, precisely @@ -155,7 +162,7 @@ a system that **measures**, instead of promising, three things per entry: this one) ```bash -python -m pytest tests/ -q # 18/18 (26/26 with scikit-learn + sentence-transformers), no network, no API key +python -m pytest tests/ -q # 31/35 (35/35 with sentence-transformers too), no network, no API key -- CI itself is currently broken, see Limitations python evals/run_eval.py --generator baseline-groq # free tier, no card python evals/run_eval.py --generator baseline # or the paid Anthropic version ``` @@ -269,11 +276,14 @@ Recall is essentially flat — a model roughly 9x larger doesn't catch meaningfully more real spoiler reveals. Precision looks better but the CIs still overlap substantially. **Combined with the truncation result above, this is a clean answer**: giving the judge more text mattered a -lot; paying for a bigger model didn't. `llama-3.1-8b-instant` is the -better default going forward — no documented daily token cap, and it's -what survived four attempts of testing without a single free-tier -failure. `LLMJudge`'s ceiling at reasonable free-tier settings looks to -be around recall≈0.35–0.4. +lot; paying for a bigger model didn't. `LLMJudge`'s ceiling at +reasonable free-tier settings looks to be around recall≈0.35–0.4 — a +conclusion about truncation vs. model size, not about a specific model +name: Groq decommissioned both `llama-3.3-70b-versatile` and +`llama-3.1-8b-instant` on 2026-08-18, so these two are cited here as the +models that actually produced the numbers above, not as what to run +today (current default: `openai/gpt-oss-120b`, `openai/gpt-oss-20b` for +the judge). ### A trained classifier beats every judge above @@ -470,16 +480,31 @@ python webapp/research_assist.py "Citizen Kane" 1941 `content/researched/` — a human still has to read and promote a draft before it's published, same bar as the existing 8 entries. -**Status:** tested end-to-end on one title (Citizen Kane, 3/3 candidates -succeeded: 4, 4, and 5 grounded claims, correctly picked the 5, zero -fabricated citations reached the output). Needs `GROQ_API_KEY` (free). -See [Roadmap](#roadmap) for what's next. +**Status:** used to draft 15 of the app's 23 researched entries so far, +batch-promoted from `content/_drafts/` after human review — the same +gate the original 8 hand-researched entries went through. That review +caught a real problem, not just typos: 4 of 13 candidates in the first +batch stated a core/major spoiler directly in the spoiler-free `story` +field, traced to the script giving one LLM call both the plot text and +an instruction not to leak it — a genuine partition (a separate call +that's only ever given non-plot text) fixed 2 of the 4; the other 2 +(Los cronocrímenes, Tetsuo) are still held back, and Tetsuo's own +Wikipedia overview states its premise-is-the-spoiler directly, which no +retrieval partition can fix. Needs `GROQ_API_KEY` (free). + +**A second, separate path skips that gate entirely.** The live app's +"+ Suggest a movie" flow auto-publishes a `research_assist.py` draft +straight to `content/researched/` with no human or AI review step at +all — a deliberate, disclosed, knowingly-risky decision (see +[Limitations](#limitations)), not an oversight. See +[Roadmap](#roadmap) for what's next. ## Browse tier: TMDB -The 12 titles in the 20-title measurement set that aren't hand-researched -yet still show a real poster and synopsis instead of an empty -placeholder, and `/api/search` reaches effectively all of TMDB — this is +The 2 titles in the 20-title measurement set that aren't researched yet +(Los cronocrímenes, Tetsuo: The Iron Man) still show a real poster and +synopsis instead of an empty placeholder, and `/api/search` reaches +effectively all of TMDB — this is a deliberately separate, lower tier: it never claims to be spoiler-safe or cited the way `content/researched/*.json` is (see D10 in `docs/DESIGN.md`). Needs `TMDB_READ_ACCESS_TOKEN`; without it, the app @@ -497,7 +522,7 @@ root (`src/preshow/env.py`, falls back to the real environment too — no | Variable | Unlocks | Cost | |---|---|---| -| `TMDB_READ_ACCESS_TOKEN` | Browse-tier posters/search for the 12 not-yet-researched titles (D10) | Free, [themoviedb.org](https://www.themoviedb.org/settings/api) | +| `TMDB_READ_ACCESS_TOKEN` | Browse-tier posters/search for the 2 not-yet-researched measurement titles (D10) | Free, [themoviedb.org](https://www.themoviedb.org/settings/api) | | `GROQ_API_KEY` | `--generator baseline-groq`, `LLMJudge` external calibration, `research_assist.py` | Free tier, no card, [console.groq.com/keys](https://console.groq.com/keys) | | `ANTHROPIC_API_KEY` | `--generator baseline` (Claude instead of Groq for the same baseline generator) | Paid — the *only* piece of this project that costs money, and it's opt-in | | `UPSTASH_REDIS_REST_URL` + `UPSTASH_REDIS_REST_TOKEN` | Comments/movie-requests survive a redeploy on a free host with an ephemeral filesystem (D11) | Free, no card, [console.upstash.com](https://console.upstash.com) | @@ -505,8 +530,8 @@ root (`src/preshow/env.py`, falls back to the real environment too — no Full install, including the optional pieces: ```bash -pip install fastapi "uvicorn[standard]" pydantic pyyaml pytest # core app + harness + tests -pip install groq # optional: baseline-groq, LLMJudge calibration, research_assist.py +pip install fastapi "uvicorn[standard]" pydantic pyyaml pytest # core app + harness +pip install groq httpx # needed for the full test suite too (see Limitations) — also unlocks baseline-groq, LLMJudge calibration, research_assist.py pip install anthropic # optional: paid baseline generator only ``` @@ -548,14 +573,14 @@ evals/ the real experiment (measurement track) dataset/titles.yaml 20-title stratified spoiler ground truth content/ - researched/*.json 8 hand-researched, cited entries + researched/*.json 23 cited entries (8 hand-researched + 15 via research_assist.py) _translations/ cached ES translations (committed — see D9) _drafts/ research_assist.py output (gitignored, pre-review) _tmdb_cache/ (gitignored) _wikipedia_cache/ research_assist.py's fetched articles (gitignored) -tests/ offline pytest suite (26 tests; 18 run in CI, 8 need optional scikit-learn/sentence-transformers), no network, no API key -docs/DESIGN.md every design decision (D1–D16) with its trade-off +tests/ offline pytest suite (35 tests, no network, no API key); needs fastapi/groq/httpx installed too, or collection fails outright — CI doesn't install those, so it currently collects zero tests, see Limitations +docs/DESIGN.md every design decision (D1–D17) with its trade-off docs/screenshots/ the two screenshots at the top of this README .github/workflows/tests.yml CI: installs core deps, runs pytest on every push/PR ``` @@ -578,14 +603,21 @@ and `CLAUDE.md`'s "Next task": iteration was closed: it would carry exactly the false-negative risk every judge attempt already demonstrated. What comes after this milestone is an open choice, not a queued fix. -- **Research the remaining 12 measurement titles** the same way Gone Girl - was (cited sources, no invented facts) — `research_assist.py` is the - start of automating this, once its output quality is more consistent - than a single Citizen Kane run has proven so far. -- **Automate the "+ Suggest a movie" pipeline.** It already resolves a - `tmdb_id` per suggestion via TMDB autocomplete, but doesn't research or - add anything automatically yet — that's exactly what - `research_assist.py` is the first step toward. +- **Research the remaining 2 measurement titles.** Los cronocrímenes is + a normal case — `research_assist.py`'s output improved once its plot + text was fully partitioned out of the pre-viewing call, just held back + pending another review pass. Tetsuo: The Iron Man is not: its own + Wikipedia overview states the film's premise-is-the-spoiler directly, + so no retrieval partition can produce a synopsis that's both honest + and safe — it needs a hand-written entry, the same way the original 8 + were done, or it stays out of the demo. +- **"+ Suggest a movie" is now automated, not pending.** It resolves a + `tmdb_id` via TMDB autocomplete and auto-publishes a + `research_assist.py` draft with no review step — a deliberate, + disclosed risk rather than the missing piece it used to be (see + [Limitations](#limitations)). Whether to add a review gate back for + this specific path is the open question now, not whether to automate + it. - **Upstash on the live Render deploy** — deliberately deferred (the project owner's call, not a blocker): comments/movie-requests on the live demo reset on every idle spin-down until an Upstash account is @@ -630,13 +662,24 @@ own stated goal applied to itself: - **The mainstream vs. long-tail hypothesis is unconfirmed.** Both strata scored identically in the Milestone 0 run — a judge with ~0 real recall can't reveal a gap that might genuinely exist. -- **`research_assist.py` is tested on one title.** Best-of-3 generation - fixed the run-to-run inconsistency seen on that one title (Citizen - Kane); it hasn't been run across a larger, varied sample yet, so - "consistent output quality" is a hypothesis, not yet a measured claim. - It also can't cite Rotten Tomatoes/Metacritic directly (no simple free - API for either) — a real, disclosed gap against the 8 hand-researched - entries, which do cite those sites directly. +- **`research_assist.py` has been run across many titles now, not just + one** (15 of the 23 researched entries), but it still can't cite Rotten + Tomatoes/Metacritic directly (no simple free API for either) — a real, + disclosed gap against the 8 hand-researched entries, which do cite + those sites directly. +- **"+ Suggest a movie" auto-publishes with no review step, unlike every + other path into `content/researched/`** (see + [Research-assist tool](#research-assist-tool-scaling-the-researched-catalogue) + above for the leak that review step actually caught). An explicit, + disclosed, knowingly-risky decision, not an oversight — see + `webapp/app.py`'s module docstring. +- **CI is currently failing on every push (since 2026-08-19).** + `tests.yml` only installs `pydantic`/`pytest`/`pyyaml`, but two newer + test files (`test_auto_publish.py`, `test_groq_retry.py`) import + `fastapi`/`groq`/`httpx` directly instead of through + `pytest.importorskip` — collection is interrupted before a single + test runs. Locally, with those installed (but not + `sentence-transformers`), 31/35 tests pass. - **The live demo's comments/movie-requests reset on idle spin-down** (Render free tier wipes the filesystem; Upstash isn't wired in yet — see [Roadmap](#roadmap)). Known and accepted, not a bug to chase. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index f9a6486..2d2744d 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -1119,7 +1119,9 @@ bar itself, and not the human review before anything is published. model to police its own citations; enforce it in code that runs after generation. - **Output goes to `content/_drafts/` (gitignored), never straight to `content/researched/`.** A human review pass is still the actual quality gate — what's automated is turning - "write from scratch" into "review and edit," not eliminating the check. + "write from scratch" into "review and edit," not eliminating the check. (True for this + CLI path specifically; D17 documents a later, separate entry point that deliberately + skips it.) **Honestly reported limitation, found by testing on one real title (Citizen Kane) across three prompt iterations**: a single generation call from a small free model is @@ -1150,6 +1152,41 @@ statement from someone who made the film (director, screenwriter) — the field' content, as seen in Gone Girl's Fincher/Flynn quotes. It still cites the retrieved Wikipedia source correctly; it's a category mismatch, not a fabrication. +## D17 — "+ Suggest a movie" auto-publishes with no review step (2026-08-19) + +D14 built one path into `content/researched/`: `research_assist.py` drafts to +`content/_drafts/`, a human reviews and promotes. That review step is what +caught a real bug the same week: diffing the first 13-candidate batch against +`evals/dataset/spoilers/` found 4 titles (Tetsuo, Come and See, Coherence, +Cronocrímenes) stating a core/major spoiler directly in the spoiler-free +`story` field — the script gave one LLM call both the plot text and an +instruction not to leak it, an instruction-level guard, not the context +PARTITION this project's whole thesis (D3) says is required. Fixed by +splitting generation into two calls: one for `story`/`context_bullets`/ +`author_voice`/etc., given only overview/production/accolades text; the +other for post-viewing fields, given everything. That fix cleared Coherence +and Come and See on re-verification (both promoted). Cronocrímenes improved +but stayed borderline, held back pending another look. Tetsuo *still* leaked +the same way after the fix — its own Wikipedia overview states the film's +premise-is-the-spoiler directly, so no retrieval partition can produce a +synopsis that's both honest and safe for it; it needs a hand-written entry +like the original 8, not a code fix. + +Against that backdrop, the user's own instruction for a second entry point — +the live app's "+ Suggest a movie" box — was to skip the review gate +entirely: `webapp/app.py`'s `_auto_publish_suggestion()` runs +`research_assist.draft_best_of()` as a background task and writes straight +to `content/researched/`, no `content/_drafts/` stop in between. Given +explicitly, after being told plainly what the 4/13 finding above means for +this specific path. Not a contradiction of D14 by accident — a deliberate, +disclosed exception to it, for one entry point, made with the failure mode +already measured rather than hypothetical. `tests/test_auto_publish.py` +covers the happy/no-op/dedup paths with everything mocked (no network, no +Groq quota spent); nothing has verified live whether a real suggestion +submitted through the deployed app has actually hit this path yet, and +Render's filesystem wipe (D11) means even a successful one wouldn't survive +the next spin-down without a redeploy baking it back into the repo. + ## Rejected - **Multi-agent (researcher / writer / critic).** No dynamic decision to @@ -1179,16 +1216,12 @@ are both done): CLAUDE.md); Milestone 1 (retrieval) is built and complete — all 20/20 titles run and hand-read, closed with two confirmed leak mechanisms documented rather than fixed (D16). -- **Automating the "+ Suggest a movie" pipeline.** The webapp captures - suggestions (`POST /api/requests`, appended to - `content/movie_requests.json`, gitignored) and now resolves the exact - film via TMDB autocomplete (D10) so a suggestion carries a `tmdb_id`, - but it still does **not** research or add anything automatically. Turning - a suggestion into a `content/researched/*.json` entry has to meet the - same bar as the existing 8 — cited sources, no invented facts (D6/D7) — - which is a real research pipeline with a review step, not a one-request - LLM call. Deliberately left as a manual step until that pipeline is - designed. +- **Automating the "+ Suggest a movie" pipeline — done, not pending.** The + webapp resolves the exact film via TMDB autocomplete (D10) and now + auto-publishes a `research_assist.py` draft straight to + `content/researched/*.json` with no review step at all — see D17. That's + a deliberate, disclosed exception to D14's review gate for this one entry + point, not the missing manual step this bullet used to describe. ## Open questions