Add IMDb mirror (port 40015)

[hqhq1025]

Summary

• Adds sites/imdb/ — a Flask mirror of imdb.com that fits WebHarbor's 16th port slot (40000+15 = 40015)
• Built end-to-end per the .claude/skills/ six-phase pipeline (clone-website → design-tasks → evolve-env → harden-env → seed-database)
• 392 titles / 3796 persons / 4280 real posters & headshots, scraped from imdb.com with Playwright; assets shipped via the paired HF dataset PR

Catalog

Entity	Count	Source
Titles	392	Top 250 movies + Top 250 TV + Most Popular + Box Office (deduped)
Persons	3796	Cast/crew of all titles, with real bios + headshots
Credits	4852	director/writer/actor/producer/composer
Genres	19	Canonical IMDb genre buckets
Posters + headshots	4280	Real images downloaded from m.media-amazon.com
Featured reviews	12	Curated, attached to high-traffic titles
News items	12	Industry-style headlines linked to titles/people
Box office data	145 movies	Real US / WW / opening / budget figures

Routes (22)

• /, /_health
• /title/<tt>, /title/<tt>/fullcredits, /title/<tt>/reviews, /title/<tt>/review (POST)
• /title/<tt>/rate (POST), /title/<tt>/watchlist (POST)
• /name/<nm>
• /find?q=&s=all|tt|nm, /search, /search/title?genre=&year_from=&year_to=&rating_min=&sort=
• /chart/top, /chart/toptv, /chart/moviemeter, /chart/boxoffice
• /genre/<slug>
• /list/watchlist, /list/ratings
• /news
• /login, /register, /logout, /account
• Scored token-overlap search (skill rule: never strict-AND); 18 stop-word list

Benchmark users (per skill convention)

alice.j / bob.c / carol.d / david.k @ test.com, password TestPass123! — each pre-seeded with a 4-item watchlist, 3-5 ratings, and one written review.

tasks.jsonl (18 tasks, 6 functional areas × 3)

• 4 hard tasks (≥5 agent steps): hardest-rated Nolan film by cross-clicking each filmography entry, advanced-search + click-through + read multi-field, end-to-end review write, cross-title compare
• 2 disambiguation tasks: bob's SF/fantasy watchlist subset (GoT, Stranger Things); carol's crime film subset (GoodFellas, Seven, Silence of the Lambs)

Data integrity safeguards (added after live audit)

• canonical-URL guard: skip 65 title JSONs where ld.url tt_id ≠ filename tt_id (IMDb redirects unknown tt_ids to a random valid page; without this guard, e.g. tt0245429 silently became Psycho)
• garbage filter: skip 1854 name JSONs that hit IMDb's 403/error fallback during concurrent scraping
• html.unescape() on all string fields; hero h1 preferred over ld.name (which is sometimes original-language: Gisaengchung vs Parasite)
• Lowercase substring match for box-office data-testid keys (bo_grossdomestic, bo_cumulativeworldwidegross, ...)

Paired HF PR

instance_seed/imdb.db + static/images/* shipped via:
https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/23

After that HF PR merges, bump .assets-revision to the merge SHA in a follow-up commit.

Test plan

• ./scripts/build.sh succeeds (after ./scripts/fetch_assets.sh pulls HF assets)
• Container starts and /health reports all 16 sites alive
• All 22 imdb routes return < 500
• POST /reset/imdb → md5sum of instance/imdb.db matches instance_seed/imdb.db (byte-identical)
• Container restart preserves md5 byte-identity
• /reset-all finishes under 10s (measured 1.3s on 26-site test image)
• Playwright handwalk: login + write review on Interstellar → review appears on /reviews page
• Playwright handwalk: bob disambiguation watchlist has ≥2 SF/fantasy candidates
• Playwright handwalk: Nolan filmography → click Dark Knight → extract rating 9.1

[Susan571]

Review — Add IMDb WebHarbor mirror (#33)

Verdict: Request changes (critical — site crashes on every boot).

Mechanical

• ❌ CRITICAL non-idempotent seed: _bootstrap() calls seed_all, which re-inserts
  genres on every start → sqlite3.IntegrityError: UNIQUE constraint failed: genres.slug.
  In the built 16-site image, imdb never binds (15/16 ready) and /reset/imdb returns
  503. It crashes the same way on every boot and every reset. AGENTS.md requires
  idempotent seeding and byte-identical reset — both fail. Guard inserts with
  exists-checks / INSERT OR IGNORE, and reset should restore from instance_seed.db
  rather than re-running seed_all.
• ❌ Empty environment: seeds from git/dockerignored scraped_data/*.json → 0 titles
  even when it does boot. Runtime data must come from instance_seed/imdb.db.
• ❌ Shared blocker: build.sh → check_assets.sh fails (missing instance_seed); no
  imdb.tar.gz on HF; .assets-revision not bumped.

Visual fidelity: FAIL (blocked)

Templates resemble IMDb (screenshots imdb_home.png, imdb_top250.png) but with 0 titles
there's nothing to render and no posters. (Live IMDb returns 202 bot-gating — can't be the
runtime source; ship the seed DB.)

Functional depth: FAIL (blocked)

Top-250, title detail, search, ratings can't be exercised against an empty/crashing DB.

Task quality: 0/20 verifiable (blocked)

Tasks reference specific titles/people not present in the booted env → unsolvable as-is.
Also wrong port: tasks use :40019; the site registers at :40015. Re-audit after fixes.

Required before approval: (1) make seeding idempotent so boot/reset don't crash;
(2) seed from instance_seed/imdb.db not scraped_data/; (3) fix task port to 40015;
(4) upload imdb.tar.gz to HF + bump .assets-revision.

[MufanQiu]

Review — imdb (PR #33)

Verdict: REQUEST CHANGES.

We deployed the environment locally and ran the full review checklist; every finding below was independently reproduced, and fidelity was checked against the real upstream site.

Issues (summary)

• BLOCKER: .assets-revision pinned to 'main' — new-site assets unreachable, CI fail-closed
• MAJOR: tasks.jsonl web port (:40019) disagrees with the real runtime port (:40015)
• MAJOR: Tasks 9 & 10 ambiguous: 'movies' but default advanced-search returns TV series at the top (non-deterministic answer)
• MINOR: Pre-sorted-answer-at-Add TED mirror site #1: tasks 12 (and partly 7) solvable from the first row without a detail page
• MINOR: Cast data corruption on The Dark Knight (and likely other titles): wrong actor->character mapping; Heath Ledger missing
• MINOR: 'Nonem' runtime rendering bug for TV series (null runtime)

Mechanical checks: FAIL

• Byte-identical reset holds. instance/imdb.db and instance_seed/imdb.db share md5 35eaaec5aef39652c5b468f1e4879071 (re-verified). seed_all() is gated (seed_data.py:404 if Title.count()>0: return), so /reset is idempotent.

Visual fidelity: PASS

Strong IMDb match. Homepage shows the black topbar + yellow logo, "What to watch", "Top picks for you" poster grid, "Trending this week", "Recently released", and "Latest IMDb news" with real posters.

Functional depth: PASS

Driven end-to-end via Playwright. Login (alice.j@test.com) and registration (auto-login) work. Search is scored token-overlap, not strict-AND ("godfather" → 3 titles + 36 names; multi-word queries return partial matches).

Task quality: FAIL

18 tasks span charts, detail lookups, person pages, advanced search, genre browse, news, comparison, and 3 auth/CRUD flows — good breadth, all DB-backed.

Required fixes before approval

1. (BLOCKER) Re-pin .assets-revision away from main to refs/pr/23 (healthline refs/pr/36 pattern) or the merged HF SHA that contains imdb.tar.gz.
2. (MAJOR) Make the port consistent: set tasks.jsonl web to the real runtime port (currently :40015).
3. (MAJOR) Disambiguate tasks 9 & 10: add explicit Type=Movie/Feature Film wording (or reword to "titles").
4. (MINOR) Reduce answer-at-Add TED mirror site #1 leakage on tasks 7 & 12.
5. (MINOR) Fix the DK cast mapping (Heath Ledger as Joker; remove director Sidney Lumet from acted cast) and audit other titles; regenerate imdb.db and re-verify byte-identity.

[jackjin1997]

I'm taking this contribution for environment, task-quality, and verifier review. I'll preserve @hqhq1025's original work, make any necessary fixes on a separate reviewer branch, and open a linked Review: PR with reproducible validation evidence. I'll link that PR here when it is available.

[jackjin1997]

The reviewer-owned Draft is now available: #89 — Review: validate IMDb mirror, sourced seed and task grading.

It preserves @hqhq1025's original contribution and commit history. The report includes the immutable HF candidate #57, source/before/after screenshots, recorded 22-site mechanical checks, and reproducible engineering validation. Task execution/scoring, visual scope confirmation and independent frozen-run review are still in progress; this is not a ready-for-merge verdict.