Review: AccuWeather mirror + task verifiers (site by @sarendis56, verifiers by reviewer)

[JeremyJC67]

What this PR carries

• 5217874 feat(accuweather): add high-fidelity mirror — @sarendis56's site commit from feat(accuweather): add high-fidelity mirror #93, unchanged (authorship preserved).
• 5bb9889 chore(accuweather): docs/tests sweep for site 25 and asset markers — README / AGENTS / CONTRIBUTING / CLAUDE / agent_demo README moved to 25 sites and 40000-40024; walmart_careers + rotten_tomatoes registry tests accept the 25-site registry (the docs-range test now derives the range from control_server.SITES, as feat(webmd_doctor): add WebMD Doctor mirror (site 25, port 40024) #95 does); sites/accuweather/.build-generated-seed + .requires-images so fetch_assets.sh / check_assets.sh treat the site like rotten_tomatoes / walmart_careers; requirements.txt pins the packages app.py actually imports.
• feat(accuweather): deterministic verifiers, judge rubrics and task re-anchor — the reviewer grading contract (below).

Full review with evidence is posted on #93. Verdict there: request changes for release mechanics (HF bundle layout, .assets-revision pin, port slot), not for the code.

Grading contract

sites/accuweather/verify/ — verify_lib.py + verify_0.py … verify_19.py, verifier_path and judge_rubric on every tasks.jsonl row, no answer key. Deterministic-only (--no_llm True accepted for parity; LLM helpers exist for API parity and no verdict depends on them):

1. Package identity — task id, agent_done, non-empty answer, every recorded URL on the same loopback host+port as start_url, screenshots decode as PNG.
2. Navigation gates — the page(s) that render the requested facts (both for comparisons); a /search?q= visit that would surface the target when it is not linked from the homepage grid (literal 94102 for task 10); /login + typed account email + required page order for stateful tasks.
3. Answer facts — label-aware numeric matchers that catch value/label swaps, unit adjacency, whole-number matching, phrase conditions (Cloudy ≠ Mostly cloudy), clock times, day labels, comparison-winner attribution; all negation-aware.
4. SQLite snapshot contract — six tables + columns, seed row counts, pinned catalog fingerprint (salt-independent), immutable location/forecast/hourly; read-only tasks leave user/saved_location/alert identical; tasks 6/7/8/9/15/18 must show the exact allowed delta (new user's password verified with hashlib) and nothing else. Snapshots from <run_dir>/initial.db|after.db, --initial_db/--after_db, or docker cp from $WH_CONTAINER; missing/invalid input fails closed.

Task 5 was re-anchored: "UV index" is not rendered on any page (the column exists but no template shows it), so the row now asks for the postal code (search results), the pressure reading (/weather/springfield-mo) and the air-quality value (/air-quality/springfield-mo) — same three-page workflow. All other 19 task texts are unchanged.

Validation (CONTRIBUTING §C)

• verify/tests/ (stdlib-only fixture DB reproducing the frozen seed fingerprint, agent.py-format trajectories): 293 tests OK — genuine PASS, run-dir snapshot discovery, no-op, wrong task id, shortcut, wrong answers, alternative phrasings, unterminated run, mixed origin, corrupt PNG, schema/catalog/seed drift (fail closed), collateral writes / state mismatch.
• verify/tests/run_matrix.py — real Chromium against a fresh site per task, live DB snapshots, five variants each: 20/20 tasks: pass = PASS; no-op, shortcut, wrong answer(s) and (stateful) state-mismatch = FAIL, each on the intended first failing check.
• Site: byte-identical instance/ vs instance_seed/ (md5 487f7e40…) after two boots and after driving every read-only page; all 20 locations' rendered values cross-checked against the seed DB (0 mismatches).

verify/tests/ is excluded from the image by .dockerignore.

Required before merge (from the review on #93)

1. Repack accuweather.tar.gz with ./scripts/extract_assets.sh <staging> accuweather (members rooted at accuweather/, no static/fonts, no instance_seed, no ._*) and re-upload to HF Add Recreation.gov task verifiers (site by @Chubi-alt, verifiers by reviewer) #66 — the current bundle is rejected by scripts/validate_asset_archive.py, so fetch_assets.sh and a clean Docker build fail.
2. Repin .assets-revision to the merged HF sha once Add Recreation.gov task verifiers (site by @Chubi-alt, verifiers by reviewer) #66 lands.
3. Port 40024 is also claimed by feat(webmd_doctor): add WebMD Doctor mirror (site 25, port 40024) #95 (webmd_doctor) — maintainer assigns the slot at merge.

Full 25-site Docker environment built from this branch with the bundle's icons copied in by hand (the bundle itself is rejected by validate_asset_archive.py): build ok (4.69 GB), 25/25 sites 200, /health 25/25, POST /reset/accuweather byte-identical (bad5a815…, the build-generated seed) before/after a dirty request and after docker restart, /reset-all 25/25 in 1.52 s, verifier no-op matrix in the container 20/20 FAIL, registry tests green, sites/accuweather suite 292 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP

[sarendis56]

Contributor asset follow-up for the remaining archive-layout finding:

HF PR #66 is updated at 0a73c1c1ac2e47513389a8a1a67601f75c8c4150. It now contains only accuweather/static/images/ (eight SVG icons), with no unsupported fonts root, stale seed database, or AppleDouble/Finder metadata. This matches this branch's build-generated seed contract.

I downloaded the uploaded bundle and confirmed it passes validate_asset_archive.py (10 managed members). Your code, tasks, and verifiers are untouched. HF merge and the subsequent .assets-revision pin remain the release steps.

[JeremyJC67]

Independently validated, and it matches your report. Downloaded refs/pr/66 at 0a73c1c1 on the review host and ran the repo's own validator:

bytes: 5524      md5: 522284a9610b5699d3982b48878b3623
members: 10      AppleDouble/DS_Store: 0
archive roots: accuweather/        second level: accuweather/static
validate_asset_archive.py -> validated 10 managed members for accuweather

That clears the blocker. The archive is rooted at accuweather/, carries only the managed static/images root, and no longer ships the unsupported fonts root or the stale seed database — which is right, since this site generates its seed at image build time.

One thing worth knowing that is on my side, not yours: when I merged the 26-site main into this review branch, my conflict resolution took upstream's Dockerfile wholesale and silently dropped the accuweather build-time seed step. The container then died at boot on cp -a .../instance_seed before the control plane came up. I have restored that step verbatim (e9e0809) and the site is being re-verified now. Nothing in your code or assets caused it.

Remaining is release mechanics only: a maintainer merges HF #66, then I pin .assets-revision to the merge sha.

[JeremyJC67]

Rebased onto the 26-site main — this branch is now site 27 / port 40026

main merged FedEx (40024) and WebMD Doctor (40025) overnight, which put every open site
branch into conflict. This branch has been brought up to date by merging main rather than
rebasing, so every contributor and reviewer commit is preserved verbatim, and re-slotted to
index 26 / port 40026: registry, Dockerfile EXPOSE, the shared docs and every
site-local port reference follow. The slot is provisional as always — say the word at merge
time and I will re-slot in whatever order you land things.

The whole environment was then rebuilt and re-verified at the new slot:

check	result
HTTP sweep	27/27 return 200
GET /health	ok, 27/27 alive and ready
POST /reset/accuweather → md5	cbddbe8704539f93e5be0f5f87959099, identical to the seed
POST /reset-all	ok, 27/27 ready, 1.63 s
verifier no-op matrix	20/20 FAIL
registry + docs tests	4 passed, 3,343 subtests

Worth recording because it was my fault, not yours: my first attempt at this merge resolved the
Dockerfile conflict by taking upstream's copy wholesale, which dropped this site's build-time
seed step. Since accuweather ships no instance_seed, websyn_start.sh then died on
cp -a .../instance_seed and the container exited six seconds after start. The step is restored
verbatim in e9e0809 and the run above is with it in place.

Still blocked only on HF #66 being merged, after which I repin .assets-revision.