Skip to content

docs: sync README with closed judge search, Milestone 1, and a legal notice #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
serpeigd merged 1 commit into from
Aug 13, 2026
Merged

docs: sync README with closed judge search, Milestone 1, and a legal notice #12

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 99 additions & 42 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ retrieval) · MyMemory (free — on-the-fly Spanish translation, cached) ·
Upstash Redis free REST API (optional — durable comments/movie-requests
on a redeploy; falls back to local files without it).

**CI:** GitHub Actions runs the 8 tests on every push (see badge above).
**CI:** GitHub Actions runs the 26 tests on every push (see badge above).

## How it's built

Expand All @@ -111,18 +111,18 @@ on a redeploy; falls back to local files without it).
| `content/researched/*.json` | 8 hand-researched entries (with cited sources: Wikipedia, Hollywood Reporter, No Film School…), not generated by an unverified LLM. |
| `src/preshow/` | Data contracts (Pydantic) for both the researched content and the measurement harness, plus the TMDB/Wikipedia/translation/KV-store clients. |
| `evals/` | The real experiment: leakage/grounding/richness metrics, calibrated judge, 20-title stratified dataset, external calibration scripts. |
| `docs/DESIGN.md` | Every non-trivial design decision (D1–D14) with its trade-off, written as it was made. |
| `docs/DESIGN.md` | Every non-trivial design decision (D1–D16) with its trade-off, written as it was made. |

## Status

| What | Status |
|---|---|
| Twistify app (catalogue, spoiler gate, filters, comments) | ✅ 8/20 entries researched |
| Browse catalogue (TMDB posters, live search, ES/EN) | ✅ 20/20 have posters, search reaches all of TMDB |
| Offline evals harness | ✅ 8 tests passing |
| Offline evals harness | ✅ 26 tests passing |
| Spoiler ground truth (20 titles) | ✅ 20/20, LLM-researched with cited sources (never hand-labeled — see [Ground truth, precisely](#ground-truth-precisely) below) |
| Baseline generator (no retrieval) | ✅ two providers — Anthropic (paid) and Groq (free tier, no card) |
| Judge calibration (offline + real spoiler reviews) | ✅ both judges calibrated against the same external human data — `SubstringJudge` recall=0.0, `LLMJudge` recall=0.089/precision=0.471 — **neither clears the bar to trust a `leakage_rate` yet** (see [Limitations](#limitations)) |
| Judge calibration (offline + real spoiler reviews) | ⛔ **closed, unsolved** — six judges built and tested against real generator output; none clears the bar to trust a `leakage_rate`. `SubstringJudge` (recall=0.0) stays the default because its failure mode is bounded and known (see [Limitations](#limitations)) |
| Measure the baseline over the 20 titles | ✅ done — see numbers and caveats below |
| Research-assist tool (D14) | ✅ drafts a researched entry from Wikipedia + TMDB; tested end-to-end on one title (Citizen Kane) |
| Retrieval (Wikipedia, GREEN-only corpus) + `--generator retrieval-groq` | ✅ `grounded_fact_rate` 0.0→1.0 confirmed live, all 20/20 titles hand-read (D16) |
Expand Down Expand Up @@ -152,7 +152,7 @@ a system that **measures**, instead of promising, three things per entry:
this one)

```bash
python -m pytest tests/ -q # 8/8, no network, no API key
python -m pytest tests/ -q # 26/26, no network, no API key
python evals/run_eval.py --generator baseline-groq # free tier, no card
python evals/run_eval.py --generator baseline # or the paid Anthropic version
```
Expand Down Expand Up @@ -518,22 +518,31 @@ webapp/ FastAPI app, one-time build helpers, research-assist too
resolve_tmdb_ids.py build step: resolves tmdb_id for every title

src/preshow/ shared library: schemas, clients, generators
schemas.py Pydantic data contracts (Claim, PreShowBrief, ContentPack…)
baseline.py / baseline_groq.py / baseline_prompts.py no-retrieval baseline generator (2 providers, 1 prompt)
schemas.py Pydantic data contracts (Claim, SourceDoc, PreShowBrief, ContentPack…)
generator.py / demo_generator.py generator interface + the offline scripted fake
baseline.py / baseline_groq.py / baseline_prompts.py Milestone 0: no-retrieval baseline (2 providers, 1 prompt)
retrieval.py Milestone 1: builds the GREEN-only corpus (D16)
retrieval_groq.py / retrieval_prompts.py Milestone 1 generator — its own prompt, not a baseline variant
content.py loads/serves content/researched/*.json
tmdb.py / wikipedia.py stdlib-only clients for the browse tier and research-assist retrieval
tmdb.py / wikipedia.py stdlib-only clients for the browse tier and retrieval
translate.py free MyMemory API client, disk-cached
kv_store.py Upstash Redis client with a local-file fallback
env.py shared .env reader

evals/ the real experiment (measurement track)
run_eval.py runs the baseline over the 20-title dataset
judge.py SubstringJudge + LLMJudge
run_eval.py runs a generator over the 20-title dataset (--generator/--judge/--titles/--save-briefs)
judge.py all six judges — Substring (default), LLM, NLI, TrainedClassifier, Hybrid, Similarity
metrics.py leakage_rate / grounded_fact_rate / richness
stats.py Wilson score confidence intervals, used by every calibration script
external_dataset.py shared loader for the IMDb Spoiler Dataset (D12)
train_spoiler_classifier.py trains TrainedClassifierJudge on this project's own labels (D15)
calibrate_substring.py internal calibration (LLM paraphrases)
calibrate_substring_external.py SubstringJudge vs. real IMDb reviews (D12)
calibrate_llm_external.py LLMJudge vs. the same real reviews (D13)
dataset/titles.yaml 20-title stratified spoiler ground truth
calibrate_nli_external.py NLIJudge, same protocol — failed, kept for the record (D15)
calibrate_trained_classifier_internal.py in-domain validation, not just IMDb reviews (D15)
calibrate_similarity.py SimilarityJudge — the sixth and final attempt (D16)
dataset/titles.yaml 20-title stratified spoiler ground truth

content/
researched/*.json 8 hand-researched, cited entries
Expand All @@ -542,8 +551,8 @@ content/
_tmdb_cache/ (gitignored)
_wikipedia_cache/ research_assist.py's fetched articles (gitignored)

tests/ offline pytest suite (8 tests), no network, no API key
docs/DESIGN.md every design decision (D1–D14) with its trade-off
tests/ offline pytest suite (26 tests), no network, no API key
docs/DESIGN.md every design decision (D1–D16) with its trade-off
docs/screenshots/ the two screenshots at the top of this README
.github/workflows/tests.yml CI: installs core deps, runs pytest on every push/PR
```
Expand All @@ -553,18 +562,19 @@ docs/screenshots/ the two screenshots at the top of this README
Documented as open, not started, in `docs/DESIGN.md`'s "Open questions"
and `CLAUDE.md`'s "Next task":

- **Resolve the judge-trust gap before Milestone 1 (retrieval).** Neither
judge currently clears the bar to report a trustworthy `leakage_rate`
(see [Limitations](#limitations)). Next concrete step: re-run
`evals/calibrate_llm_external.py` with full, untruncated review text
and/or a stronger model (`llama-3.3-70b-versatile`) to find out whether
today's 0.089 recall is a real model-capacity ceiling or an artifact of
the 350-char truncation forced by this run's token budget. If that's
still not good enough, two genuinely different alternatives are on the
table but not started: a lightweight local NLI/entailment classifier
(no per-call cost or rate limit), or a classifier trained directly on
this project's own 2,197-positive/5,460-negative external labels with a
proper held-out split.
- **Judge iteration is closed, not solved** (D15/D16). Six judges were
built and tested; none can be trusted to report a `leakage_rate`, and
no seventh is planned. What replaced it is not another judge but a
practice: `run_eval.py --save-briefs PATH` plus a human reading the
generated text. That is what actually caught all three confirmed leaks
in this project — no automated judge caught any of them.
- **Milestone 1 is complete** (20/20 titles run and hand-read) and
**closed with two confirmed leak mechanisms documented rather than
fixed** — see [Limitations](#limitations). A content filter over the
GREEN corpus was considered and rejected for the same reason judge
iteration was closed: it would carry exactly the false-negative risk
every judge attempt already demonstrated. What comes after this
milestone is an open choice, not a queued fix.
- **Research the remaining 12 measurement titles** the same way Gone Girl
was (cited sources, no invented facts) — `research_assist.py` is the
start of automating this, once its output quality is more consistent
Expand All @@ -584,19 +594,36 @@ and `CLAUDE.md`'s "Next task":
Stated plainly, not hidden behind a green badge — this is the project's
own stated goal applied to itself:

- **Neither spoiler judge is trustworthy yet**, and no `leakage_rate` in
this README should be read as a real safety measurement until this
changes. `SubstringJudge` catches 0 of 2,197 real human spoiler reviews
(recall = 0.0 — it only matches verbatim planted phrases). `LLMJudge`
(Groq, `llama-3.1-8b-instant`) does better but still misses ~91 of every
100 real spoiler reveals in its calibration sample (recall = 0.089,
precision = 0.471, n=180 — see [`LLMJudge` calibration](#llmjudge-calibrated-the-same-way-groq-free-tier)).
This is *why* Milestone 1 (retrieval) hasn't started — there's no point
measuring whether retrieval helps against a judge that can't reliably
see leaks either way.
- **`grounded_fact_rate = 0.0` on the current baseline is expected, not a
bug** — the no-retrieval baseline is given no corpus on purpose; it's
the number Milestone 1 needs to beat.
- **No spoiler judge in this project is trustworthy**, and no
`leakage_rate` anywhere in this README should be read as a real safety
measurement. Six were built and tested against actual generator output:
`SubstringJudge` (recall = 0.0 — it only matches verbatim planted
phrases), `LLMJudge`, `NLIJudge`, `TrainedClassifierJudge`,
`HybridJudge`, and `SimilarityJudge`. Several calibrated well offline
and then failed live, in the same direction every time: confidently
flagging release years, cast credits, stage directions, and marketing
copy as "core" spoilers. `SimilarityJudge` scored a film's own public
premise (0.561) *higher* than the one confirmed real leak it was built
to catch (0.525) — no threshold separates those. `SubstringJudge`
remains the default not because it is good but because its failure
mode is bounded and known: it misses things, rather than inventing
them. Six independent failures, so judge iteration is closed (D15/D16).
- **Three real leaks are confirmed in Milestone 1's output, found by
human reading and by no judge**, via two independent mechanisms:
*Los cronocrímenes* ("one man must stop his other selves") came from
the **model's own memory** — the retrieved corpus never mentions it —
while *Tetsuo* and *Hard to Be a God* leaked because **Wikipedia's own
`overview` section states those films' core spoilers verbatim**, so the
"safe" GREEN corpus wasn't safe for them. *Come and See* has the same
risk latent in its `production` section. Milestone 1's real result
stands (grounding genuinely improved; the five highest-risk
famous-twist titles all came back clean), but it is **not a leak-proof
guarantee** — a context partition bounds what the model can read, not
what it already knows.
- **`grounded_fact_rate = 0.0` on the Milestone 0 baseline is expected,
not a bug** — that generator is given no corpus on purpose. Milestone 1
beat it: 0.0 → 1.0, and that one is judge-independent (it only checks
whether a `source_id` is populated).
- **The mainstream vs. long-tail hypothesis is unconfirmed.** Both strata
scored identically in the Milestone 0 run — a judge with ~0 real recall
can't reveal a gap that might genuinely exist.
Expand Down Expand Up @@ -624,7 +651,37 @@ own stated goal applied to itself:
- **Wikipedia** — CC BY-SA, already in use for researched entries.
- **Scraping IMDb** — forbidden by ToS, not done under any excuse.

## License

No license defined yet — personal portfolio repo. If you want to reuse
something, ask first.
## License and legal notice

**Copyright © 2026 Sergio Peigneux d'Egmont
([@serpeigd](https://github.com/serpeigd)). All rights reserved.**

No open-source licence is granted for this repository. Being publicly
readable on GitHub is not a licence: absent one, copyright law reserves
copying, modification, and redistribution to the author. You're welcome
to read it, learn from it, and cite it with attribution — for anything
beyond that, ask first and you'll probably get a yes. A permissive
licence may be added later; until a `LICENSE` file exists in this repo,
this section is the operative statement.

That reservation covers this project's own code, prose, and design
documentation. It does **not** cover, and cannot relicense, the
third-party material this project builds on:

- **Film titles, posters, synopses, and metadata** belong to their
respective rights holders. TMDB content is used under TMDB's free
non-commercial terms with attribution shown in the app; this project is
not endorsed or certified by TMDB.
- **Wikipedia text** retrieved into the GREEN corpus and the researched
entries is CC BY-SA, and stays under that licence.
- **The IMDb Spoiler Dataset** used for judge calibration
(`evals/dataset/external/`, gitignored) carries its own terms and is
never redistributed here. Scraping IMDb directly is prohibited by its
ToS and is not done anywhere in this project.
- **Quotes attributed to directors, screenwriters, and critics** in
`content/researched/*.json` are short, cited excerpts used for
commentary, and remain their authors' work.

Nothing here is a legal opinion, and this is a personal portfolio
project — no warranty, and no fitness-for-purpose claim, is made for any
part of it.
Loading