Review: 9GAG mirror + task verifiers (site by @sarendis56, verifiers by reviewer) - #97
Open
JeremyJC67 wants to merge 10 commits into
Open
Review: 9GAG mirror + task verifiers (site by @sarendis56, verifiers by reviewer)#97JeremyJC67 wants to merge 10 commits into
JeremyJC67 wants to merge 10 commits into
Conversation
The site PR registers 9gag as site index 24 (port 40024) and bumps EXPOSE, but leaves the shared docs and the two registry integration tests on the 24-site / 40000-40023 wording, which makes sites/walmart_careers/tests/test_integration.py and sites/rotten_tomatoes/tests/test_environment_quality.py fail on the branch. Apply the same sweep PR aiming-lab#95 does: README, AGENTS, CONTRIBUTING, CLAUDE, agent_demo/README, the Dockerfile comment, and derive the documented port range from the control_server SITES registry in the walmart_careers test. Port 40024 is provisional (maintainer reassigns at merge). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…tasks Reviewer grading contract for the 9GAG mirror (PR aiming-lab#92): - sites/9gag/verify/verify_lib.py: shared utilities (merriam_webster API family with walmart_careers-style hardening) - package identity, loopback origin + port check on every recorded URL, PNG screenshot decode, search/interest/detail navigation gates, login-before-action ordering, affirmative answer matchers (counts, decimals, clock times), and a SQLite snapshot contract (pinned schema hash, frozen seed counts, immutable post catalog, exact per-table row deltas, no collateral writes). The anchored LLM helpers are kept for parity but only emit advisory evidence; every verdict is decided with --no_llm True. - verify_0.py .. verify_19.py: one verifier per task with the ground truth hardcoded (never in tasks.jsonl). Read tasks accept either the original or its identical-fact "Community remix" clone unless the task says "original"; stateful tasks require the exact saved/vote/comment/ hidden/report/post/user delta and verify the registered password hash. - tasks.jsonl: add verifier_path + English judge_rubric to every row; the task 10 wording now says "original" so the remix clone (post 47) is not a valid target, matching tasks 11-13/18/19. - verify/README.md: contract summary per task and how to run the tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
- verify/tests/_support.py + test_verify_{0..19}.py + test_verify_lib.py:
244 unittest cases over copies of the frozen seed (genuine PASS,
run-dir snapshot discovery, no-op, shortcut, wrong answers per fact,
wrong task id, unterminated, mixed origin, corrupt screenshot,
read-only writes, schema/catalog tampering, stale-state preconditions,
wrong target/account/reason/text, duplicates, counter bumps, collateral
writes, real werkzeug scrypt hash verification).
- verify/tests/live_matrix.py: drives every task for real in Chromium
against a running mirror, records agent.py-shaped run dirs with real
screenshots and DB snapshots, and grades genuine / no-op / shortcut /
wrong / state-mismatch variants (CONTRIBUTING section C evidence).
Excluded from the image by .dockerignore (sites/*/verify/tests/).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
The Memes interest (sidebar entry, top-nav link and the Recents chip), all six topic chips and the #animals popular tag rendered "No posts found": the seed carries 0 memes posts, and no post carries the tags "spider man", "usa", "trump", "elon musk", "politics", "latest news" or "animals". Seeding those interests would require a Hugging Face re-upload of instance_seed/9gag.db, so the dead entry points are instead repointed at feeds that the sidebar already exposes. Nothing becomes newly reachable, so no task gains a navigation path its verifier does not expect, and every task's result set is unchanged (re-measured: science 17, animals 13, sports 5, gaming 4, news 19, lighthouse 10, rescue cat 9, sourdough 2 - identical to the first-pass review). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
At 768px the document was 857px wide on 11 routes. Two causes: the .layout grid used a fixed minmax(0,640px) track that cannot shrink, and the six-tab .feed-tabs strip needs ~628px while the main column is narrower. The grid track is now flexible under a 934px cap (preserving the intended 640px reading column at 1000px), and the tab strip tightens and scrolls itself between 681px and 1000px instead of scrolling the page. The 9GAG greys fail WCAG 1.4.3 on white: #999 at 2.849:1, #888 at 3.545:1 and #777 at 4.478:1 against a 4.5:1 floor, plus white on the brand blue at 4.068:1 and the 26px dismiss glyph at 2.548:1 against a 3:1 floor. A --muted token (#767676, 4.54:1) and an --action token (#0a74e6, 4.52:1) carry the text and the actionable controls; the brand --blue stays for decoration. Both fixes land in a layer appended at the end of the file: the stylesheet is written as successive override layers, so the same rules placed earlier are defeated by the later .feed-tabs / .stats / .actions redefinitions. Measured with Playwright plus pixel sampling (contributor CSS vs this commit, same instrument): overflow 11 -> 0 failing route/width cells, contrast 51 -> 0 failing elements, across 1440/768/390/320 x 25 routes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
Each item below was reproduced against a running mirror before the change and re-run after it. - Invalid input was accepted silently, which is the one item here that can corrupt a task grade: POST /submit with section=NOT_AN_INTEREST stored that section, POST /report with an arbitrary reason stored it, POST /vote with value=garbage was coerced into a DOWNVOTE, and a 5000-character title went into a String(300) column. Interests, report reasons and preview images are now one tuple each, rendered by the template and validated by the handler so the two cannot drift, and bounded_text rejects over-long fields. All four now return 400 and write nothing. - SECRET_KEY was the tracked literal "webharbor-9gag-deterministic-key". A cookie minted with it returned 200 on /account rendering the seeded account and a forged POST wrote a saved_post row. Now os.environ NINEGAG_SECRET_KEY or secrets.token_hex(32), as on 7 already-merged sites. - No CSRF protection (9gag and ikea were the only 2 of 25 sites without it). A cross-origin POST to /save wrote a row; now CSRFProtect plus a token in all 13 POST forms, and that request returns 400. - GET /logout logged the user out; now POST-only, GET returns 405, and Settings posts a form. - No MAX_CONTENT_LENGTH: a 60 MB multipart POST was accepted in full; now 256 KB, returning 413. - No error handlers: /nope rendered the bare Werkzeug page, breaking mirror fidelity. 400/404/413/500 now render a 9GAG-shell error page, and the 500 handler rolls the session back. - source_id used COUNT+1, which collides after any delete; now MAX(numeric suffix)+1. - Flask-WTF added to requirements.txt (the site now imports it; 8 merged sites already declare it). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
The seed duplicates 35 curated posts as "Community remix N: ...", copying the description verbatim, so a clone shows exactly the same facts as its original. Tasks 6 and 8 are read-only fact questions, yet their verifiers accepted only the original slug: an agent that read the right number off the clone and answered correctly was graded FAIL on a rule stated nowhere the agent could see. The other seven read-only clone tasks (0-5, 7) already accept either page, so this was also an internal inconsistency. Provenance is now enforced only where it changes the graded outcome: - read-only fact tasks 0-8 accept the original or its clone (identical facts); - task 9 keeps the original (it compares point counts, which differ between original and clone); - stateful tasks 10-13, 18, 19 keep the original (the clone is a different row to save/vote/hide). Every task whose grade still depends on provenance now says so in its ques text, so the requirement is visible to the agent rather than hidden in the verifier. Widening the gate is mutation-proofed: each of tasks 6 and 8 gains a case asserting the clone now passes and a case asserting an unrelated detail page still fails on visited_post_detail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
agent_demo/agent.py never writes final_url (ast over its trajectory dict: task, task_id, start_url, model, max_steps, steps, terminated, termination_reason, final_answer, judge_rubric, verifier_path, plus success_self_report on done). Both the unit fixtures and the live matrix wrote one, so the verifiers were only ever proven against a shape the recorder does not emit. The live matrix now omits it: its Recorder.done() already appends a done step carrying the landing page url exactly as agent.py does, so the evidence it produces is shape-identical to a real run. The unit fixtures keep the field, and every read task gains a case that nulls it, so both paths are covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
main gained fedex (40024) and webmd_doctor (40025). Merged rather than rebased so every contributor and reviewer commit on this branch is preserved verbatim. Shared registry, Dockerfile EXPOSE, docs and every site-local port reference move to index 26 / port 40026; the slot stays provisional and the maintainer reassigns it at merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
Contributor
|
Contributor asset follow-up for the remaining packaging finding: HF PR #65 is updated at I downloaded the uploaded bundle and confirmed it passes |
Author
|
Independently validated, and it matches your report. Downloaded That closes required fix 2 from the audit comment. The 73 AppleDouble members and the unused Remaining on this PR is release mechanics only, and it is not yours: a maintainer merges HF #65, after which I pin |
Author
Rebased onto the 26-site
|
| check | result |
|---|---|
| HTTP sweep | 27/27 return 200 |
GET /health |
ok, 27/27 alive and ready |
POST /reset/9gag → md5 |
44768940829bf4a8bec07d7bdf072d97, identical to the seed |
POST /reset-all |
ok, 27/27 ready, 1.67 s |
| verifier no-op matrix | 20/20 FAIL |
sites/9gag suite in the image |
258 passed, 130 skipped |
The blocker fix holds in the image: POST /gag/<slug>/vote with value=NOT_AN_OPTION returns
400 instead of silently writing a downvote. Also GET /logout 405, save without a token 400,
branded 404 (1,175 bytes), 300 KB POST /register 413, and seven read-only page loads leave
the DB byte-identical.
Your repacked bundle is validated and reported separately above; the only thing left is a
maintainer merging HF #65.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Review vehicle for #92 (
9gagmirror by @sarendis56). This PR keeps the site commit intact (authorship preserved) and adds the reviewer deliverables on top:sites/9gag/verify/verify_lib.py+verify_0.py … verify_19.py(deterministic; ground truth hardcoded in the verifiers only),verifier_path+ Englishjudge_rubricon all 20 rows ofsites/9gag/tasks.jsonl, andsites/9gag/verify/README.mdsummarising the per-task contract.sites/9gag/verify/tests/(244 unit cases over copies of the frozen seed + a Playwright live matrix that drives every task end-to-end); excluded from the image by.dockerignore.agent_demo/README, the Dockerfile comment and the two registry integration tests updated to 25 sites /40000-40024(they fail on the feat(9gag): add high-fidelity mirror #92 head as submitted).Full review comment with mechanical / visual / functional / task-quality evidence is posted on #92.
Verifier design
agent_done, non-empty answer, every recorded URLhttp://on a loopback host and the same port asstart_url, every referenced screenshot decodes as PNG./search?q=with a task-relevant whole token or the named feed page, plus the exact/gag/<slug>detail page (T19: all three compared posts; T14: the/reportform), login-before-action ordering and the typed account identity for stateful tasks.4.2, clock-time variants.--no_llm True.Validation (carcinoma, standalone Flask on
127.0.0.1:45001, agent_demo Playwright env)Live matrix (
verify/tests/live_matrix.py, real Chromium drive per task, real screenshots, seed vs live DB snapshots):Unit matrix:
uv run python -m unittest discover -s sites/9gag/verify/tests -p 'test_*.py'→ 246 tests OK (10 skipped = abstract base class).agent_demo/eval_judge.py --run_dir <run> --verifier Trueinvokes the verifiers end-to-end (checked on a genuine and a shortcut run dir).Site checks (standalone): byte-identical seed md5
44768940829bf4a8bec07d7bdf072d97after boot 1, boot 2, a full read-only Playwright drive and a fresh reset; all auth/CRUD/form flows persist; 20/20 tasks solvable via the UI with no answer leaks on results cards. Full 25-site Docker environment built from this branch withrefs/pr/65assets: build ok (4.72 GB), 25/25 sites 200,/health25/25,POST /reset/9gagbyte-identical (44768940…) before/after a dirty request and afterdocker restart,/reset-all25/25 in 1.65 s, verifier no-op matrix in the container 20/20 FAIL, registry tests green,sites/9gagsuite 246 passed.Still required on the site side (tracked on #92)
.assets-revisionto the merged HF sha once discussion Add TED task verifiers (site by @shanjiaming, verifiers by reviewer) #65 lands.._*members from9gag.tar.gz.site.css@media(max-width:1000px)grid).#animals, which all land on "No posts found".Port 40024 is provisional; #95 (webmd_doctor) claims the same slot — the maintainer reassigns at merge.
🤖 Generated with Claude Code
https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP