From 3bf0d8c06abc5fecf13373d34c6bc86fb82a5fa5 Mon Sep 17 00:00:00 2001 From: Claude <125498425+serpeigd@users.noreply.github.com> Date: Thu, 13 Aug 2026 19:29:27 +0000 Subject: [PATCH] docs: sync README with closed judge search, Milestone 1, and a legal notice - Fix stale counts: the suite is 26 tests, not 8 (four places), and docs/DESIGN.md now runs D1-D16, not D1-D14 (two places). - Rewrite the Status row, Roadmap, and Limitations for the actual state: judge iteration is closed after six judges, Milestone 1 is complete at 20/20 titles, and the three confirmed leaks are documented with both mechanisms (model memory vs. Wikipedia's own overview section). - Drop the Roadmap item that still framed retrieval as not started and the judge gap as the thing blocking it; both are resolved. - Expand the project layout with retrieval.py, retrieval_groq.py, retrieval_prompts.py, stats.py, external_dataset.py, train_spoiler_classifier.py and the three newer calibration scripts, and correct judge.py's description from two judges to six. - Replace the placeholder License section with a real notice: explicit copyright reservation (no LICENSE file exists, so public visibility is not a grant), and the third-party terms it cannot relicense (TMDB, Wikipedia CC BY-SA, the IMDb Spoiler Dataset, cited quotes). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MDoFHbKZRxhUhyFQuYVFUc --- README.md | 141 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 99 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index f8f591e..84ea3a4 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,7 @@ retrieval) · MyMemory (free — on-the-fly Spanish translation, cached) · Upstash Redis free REST API (optional — durable comments/movie-requests on a redeploy; falls back to local files without it). -**CI:** GitHub Actions runs the 8 tests on every push (see badge above). +**CI:** GitHub Actions runs the 26 tests on every push (see badge above). ## How it's built @@ -111,7 +111,7 @@ on a redeploy; falls back to local files without it). | `content/researched/*.json` | 8 hand-researched entries (with cited sources: Wikipedia, Hollywood Reporter, No Film School…), not generated by an unverified LLM. | | `src/preshow/` | Data contracts (Pydantic) for both the researched content and the measurement harness, plus the TMDB/Wikipedia/translation/KV-store clients. | | `evals/` | The real experiment: leakage/grounding/richness metrics, calibrated judge, 20-title stratified dataset, external calibration scripts. | -| `docs/DESIGN.md` | Every non-trivial design decision (D1–D14) with its trade-off, written as it was made. | +| `docs/DESIGN.md` | Every non-trivial design decision (D1–D16) with its trade-off, written as it was made. | ## Status @@ -119,10 +119,10 @@ on a redeploy; falls back to local files without it). |---|---| | Twistify app (catalogue, spoiler gate, filters, comments) | ✅ 8/20 entries researched | | Browse catalogue (TMDB posters, live search, ES/EN) | ✅ 20/20 have posters, search reaches all of TMDB | -| Offline evals harness | ✅ 8 tests passing | +| Offline evals harness | ✅ 26 tests passing | | Spoiler ground truth (20 titles) | ✅ 20/20, LLM-researched with cited sources (never hand-labeled — see [Ground truth, precisely](#ground-truth-precisely) below) | | Baseline generator (no retrieval) | ✅ two providers — Anthropic (paid) and Groq (free tier, no card) | -| Judge calibration (offline + real spoiler reviews) | ✅ both judges calibrated against the same external human data — `SubstringJudge` recall=0.0, `LLMJudge` recall=0.089/precision=0.471 — **neither clears the bar to trust a `leakage_rate` yet** (see [Limitations](#limitations)) | +| Judge calibration (offline + real spoiler reviews) | ⛔ **closed, unsolved** — six judges built and tested against real generator output; none clears the bar to trust a `leakage_rate`. `SubstringJudge` (recall=0.0) stays the default because its failure mode is bounded and known (see [Limitations](#limitations)) | | Measure the baseline over the 20 titles | ✅ done — see numbers and caveats below | | Research-assist tool (D14) | ✅ drafts a researched entry from Wikipedia + TMDB; tested end-to-end on one title (Citizen Kane) | | Retrieval (Wikipedia, GREEN-only corpus) + `--generator retrieval-groq` | ✅ `grounded_fact_rate` 0.0→1.0 confirmed live, all 20/20 titles hand-read (D16) | @@ -152,7 +152,7 @@ a system that **measures**, instead of promising, three things per entry: this one) ```bash -python -m pytest tests/ -q # 8/8, no network, no API key +python -m pytest tests/ -q # 26/26, no network, no API key python evals/run_eval.py --generator baseline-groq # free tier, no card python evals/run_eval.py --generator baseline # or the paid Anthropic version ``` @@ -518,22 +518,31 @@ webapp/ FastAPI app, one-time build helpers, research-assist too resolve_tmdb_ids.py build step: resolves tmdb_id for every title src/preshow/ shared library: schemas, clients, generators - schemas.py Pydantic data contracts (Claim, PreShowBrief, ContentPack…) - baseline.py / baseline_groq.py / baseline_prompts.py no-retrieval baseline generator (2 providers, 1 prompt) + schemas.py Pydantic data contracts (Claim, SourceDoc, PreShowBrief, ContentPack…) + generator.py / demo_generator.py generator interface + the offline scripted fake + baseline.py / baseline_groq.py / baseline_prompts.py Milestone 0: no-retrieval baseline (2 providers, 1 prompt) + retrieval.py Milestone 1: builds the GREEN-only corpus (D16) + retrieval_groq.py / retrieval_prompts.py Milestone 1 generator — its own prompt, not a baseline variant content.py loads/serves content/researched/*.json - tmdb.py / wikipedia.py stdlib-only clients for the browse tier and research-assist retrieval + tmdb.py / wikipedia.py stdlib-only clients for the browse tier and retrieval translate.py free MyMemory API client, disk-cached kv_store.py Upstash Redis client with a local-file fallback env.py shared .env reader evals/ the real experiment (measurement track) - run_eval.py runs the baseline over the 20-title dataset - judge.py SubstringJudge + LLMJudge + run_eval.py runs a generator over the 20-title dataset (--generator/--judge/--titles/--save-briefs) + judge.py all six judges — Substring (default), LLM, NLI, TrainedClassifier, Hybrid, Similarity metrics.py leakage_rate / grounded_fact_rate / richness + stats.py Wilson score confidence intervals, used by every calibration script + external_dataset.py shared loader for the IMDb Spoiler Dataset (D12) + train_spoiler_classifier.py trains TrainedClassifierJudge on this project's own labels (D15) calibrate_substring.py internal calibration (LLM paraphrases) calibrate_substring_external.py SubstringJudge vs. real IMDb reviews (D12) calibrate_llm_external.py LLMJudge vs. the same real reviews (D13) - dataset/titles.yaml 20-title stratified spoiler ground truth + calibrate_nli_external.py NLIJudge, same protocol — failed, kept for the record (D15) + calibrate_trained_classifier_internal.py in-domain validation, not just IMDb reviews (D15) + calibrate_similarity.py SimilarityJudge — the sixth and final attempt (D16) + dataset/titles.yaml 20-title stratified spoiler ground truth content/ researched/*.json 8 hand-researched, cited entries @@ -542,8 +551,8 @@ content/ _tmdb_cache/ (gitignored) _wikipedia_cache/ research_assist.py's fetched articles (gitignored) -tests/ offline pytest suite (8 tests), no network, no API key -docs/DESIGN.md every design decision (D1–D14) with its trade-off +tests/ offline pytest suite (26 tests), no network, no API key +docs/DESIGN.md every design decision (D1–D16) with its trade-off docs/screenshots/ the two screenshots at the top of this README .github/workflows/tests.yml CI: installs core deps, runs pytest on every push/PR ``` @@ -553,18 +562,19 @@ docs/screenshots/ the two screenshots at the top of this README Documented as open, not started, in `docs/DESIGN.md`'s "Open questions" and `CLAUDE.md`'s "Next task": -- **Resolve the judge-trust gap before Milestone 1 (retrieval).** Neither - judge currently clears the bar to report a trustworthy `leakage_rate` - (see [Limitations](#limitations)). Next concrete step: re-run - `evals/calibrate_llm_external.py` with full, untruncated review text - and/or a stronger model (`llama-3.3-70b-versatile`) to find out whether - today's 0.089 recall is a real model-capacity ceiling or an artifact of - the 350-char truncation forced by this run's token budget. If that's - still not good enough, two genuinely different alternatives are on the - table but not started: a lightweight local NLI/entailment classifier - (no per-call cost or rate limit), or a classifier trained directly on - this project's own 2,197-positive/5,460-negative external labels with a - proper held-out split. +- **Judge iteration is closed, not solved** (D15/D16). Six judges were + built and tested; none can be trusted to report a `leakage_rate`, and + no seventh is planned. What replaced it is not another judge but a + practice: `run_eval.py --save-briefs PATH` plus a human reading the + generated text. That is what actually caught all three confirmed leaks + in this project — no automated judge caught any of them. +- **Milestone 1 is complete** (20/20 titles run and hand-read) and + **closed with two confirmed leak mechanisms documented rather than + fixed** — see [Limitations](#limitations). A content filter over the + GREEN corpus was considered and rejected for the same reason judge + iteration was closed: it would carry exactly the false-negative risk + every judge attempt already demonstrated. What comes after this + milestone is an open choice, not a queued fix. - **Research the remaining 12 measurement titles** the same way Gone Girl was (cited sources, no invented facts) — `research_assist.py` is the start of automating this, once its output quality is more consistent @@ -584,19 +594,36 @@ and `CLAUDE.md`'s "Next task": Stated plainly, not hidden behind a green badge — this is the project's own stated goal applied to itself: -- **Neither spoiler judge is trustworthy yet**, and no `leakage_rate` in - this README should be read as a real safety measurement until this - changes. `SubstringJudge` catches 0 of 2,197 real human spoiler reviews - (recall = 0.0 — it only matches verbatim planted phrases). `LLMJudge` - (Groq, `llama-3.1-8b-instant`) does better but still misses ~91 of every - 100 real spoiler reveals in its calibration sample (recall = 0.089, - precision = 0.471, n=180 — see [`LLMJudge` calibration](#llmjudge-calibrated-the-same-way-groq-free-tier)). - This is *why* Milestone 1 (retrieval) hasn't started — there's no point - measuring whether retrieval helps against a judge that can't reliably - see leaks either way. -- **`grounded_fact_rate = 0.0` on the current baseline is expected, not a - bug** — the no-retrieval baseline is given no corpus on purpose; it's - the number Milestone 1 needs to beat. +- **No spoiler judge in this project is trustworthy**, and no + `leakage_rate` anywhere in this README should be read as a real safety + measurement. Six were built and tested against actual generator output: + `SubstringJudge` (recall = 0.0 — it only matches verbatim planted + phrases), `LLMJudge`, `NLIJudge`, `TrainedClassifierJudge`, + `HybridJudge`, and `SimilarityJudge`. Several calibrated well offline + and then failed live, in the same direction every time: confidently + flagging release years, cast credits, stage directions, and marketing + copy as "core" spoilers. `SimilarityJudge` scored a film's own public + premise (0.561) *higher* than the one confirmed real leak it was built + to catch (0.525) — no threshold separates those. `SubstringJudge` + remains the default not because it is good but because its failure + mode is bounded and known: it misses things, rather than inventing + them. Six independent failures, so judge iteration is closed (D15/D16). +- **Three real leaks are confirmed in Milestone 1's output, found by + human reading and by no judge**, via two independent mechanisms: + *Los cronocrímenes* ("one man must stop his other selves") came from + the **model's own memory** — the retrieved corpus never mentions it — + while *Tetsuo* and *Hard to Be a God* leaked because **Wikipedia's own + `overview` section states those films' core spoilers verbatim**, so the + "safe" GREEN corpus wasn't safe for them. *Come and See* has the same + risk latent in its `production` section. Milestone 1's real result + stands (grounding genuinely improved; the five highest-risk + famous-twist titles all came back clean), but it is **not a leak-proof + guarantee** — a context partition bounds what the model can read, not + what it already knows. +- **`grounded_fact_rate = 0.0` on the Milestone 0 baseline is expected, + not a bug** — that generator is given no corpus on purpose. Milestone 1 + beat it: 0.0 → 1.0, and that one is judge-independent (it only checks + whether a `source_id` is populated). - **The mainstream vs. long-tail hypothesis is unconfirmed.** Both strata scored identically in the Milestone 0 run — a judge with ~0 real recall can't reveal a gap that might genuinely exist. @@ -624,7 +651,37 @@ own stated goal applied to itself: - **Wikipedia** — CC BY-SA, already in use for researched entries. - **Scraping IMDb** — forbidden by ToS, not done under any excuse. -## License - -No license defined yet — personal portfolio repo. If you want to reuse -something, ask first. +## License and legal notice + +**Copyright © 2026 Sergio Peigneux d'Egmont +([@serpeigd](https://github.com/serpeigd)). All rights reserved.** + +No open-source licence is granted for this repository. Being publicly +readable on GitHub is not a licence: absent one, copyright law reserves +copying, modification, and redistribution to the author. You're welcome +to read it, learn from it, and cite it with attribution — for anything +beyond that, ask first and you'll probably get a yes. A permissive +licence may be added later; until a `LICENSE` file exists in this repo, +this section is the operative statement. + +That reservation covers this project's own code, prose, and design +documentation. It does **not** cover, and cannot relicense, the +third-party material this project builds on: + +- **Film titles, posters, synopses, and metadata** belong to their + respective rights holders. TMDB content is used under TMDB's free + non-commercial terms with attribution shown in the app; this project is + not endorsed or certified by TMDB. +- **Wikipedia text** retrieved into the GREEN corpus and the researched + entries is CC BY-SA, and stays under that licence. +- **The IMDb Spoiler Dataset** used for judge calibration + (`evals/dataset/external/`, gitignored) carries its own terms and is + never redistributed here. Scraping IMDb directly is prohibited by its + ToS and is not done anywhere in this project. +- **Quotes attributed to directors, screenwriters, and critics** in + `content/researched/*.json` are short, cited excerpts used for + commentary, and remain their authors' work. + +Nothing here is a legal opinion, and this is a personal portfolio +project — no warranty, and no fitness-for-purpose claim, is made for any +part of it.