Skip to content

Review: 4shared mirror + task verifiers (site by @sarendis56, verifiers by reviewer) - #96

Open
JeremyJC67 wants to merge 11 commits into
aiming-lab:mainfrom
JeremyJC67:review/pr-90-4shared
Open

Review: 4shared mirror + task verifiers (site by @sarendis56, verifiers by reviewer)#96
JeremyJC67 wants to merge 11 commits into
aiming-lab:mainfrom
JeremyJC67:review/pr-90-4shared

Conversation

@JeremyJC67

Copy link
Copy Markdown

Summary

Review vehicle for #90 (feat(4shared): add 4shared mirror by @sarendis56). It carries the contributor's two commits unchanged in authorship, rebased onto current main, plus the reviewer's integration and grading-contract commits:

commit author what
feat(4shared): add 4shared mirror @sarendis56 site (rebased; the registration hunk that conflicted with main was dropped from this commit and redone below)
fix(4shared): harden mirror and replace placeholder assets @sarendis56 contributor's hardening pass
chore(4shared): integrate as site 25 — port 40024, docs sweep, registry tests reviewer websyn_start.sh / control_server.py append 4shared as the LAST entry (index 24 → 40024, provisional); Dockerfile 25 sites / EXPOSE 8101 40000-40024; README / AGENTS / CONTRIBUTING / CLAUDE / agent_demo README sweep; walmart_careers + rotten_tomatoes registry tests bumped to 25; tasks.jsonl web → 40024; app.py SITE_PORT → 40024
fix(4shared): re-anchor task 2 off the first result; disambiguate task 16's plan reviewer two task rewordings (see review comment)
test(4shared): deterministic verifiers, judge rubrics and LLM-free validation matrix reviewer sites/4shared/verify/ (verify_lib + 20 verifiers + README), verifier_path + judge_rubric on all 20 rows, verify/tests/ harness

Full review (mechanical / visual / functional / per-task audit, evidence and screenshots) is posted on #90.

What was verified (standalone on the review server, Playwright, no LLM)

  • Asset tarball from HF discussion feat(webmd): add WebMD mirror site (20 tasks, hardened) #62 (refs/pr/62 = f426447ee209071110a5de80026a72d049bbd97f): validate_asset_archive.py OK (30 members, no ._*/.DS_Store), sha256 f50985a7… matches the PR body; seed DB md5 b577adc216900a6f0e3974a80e51c04c (matches the contributor's claim).
  • Byte-identical instance/ vs instance_seed/ after first boot, after a full read-only Playwright sweep, and after two further boots from seed.
  • 122 public files across 7 categories, 16/16 image records render real photographs (no broken/placeholder images), every footer route 200, 404 page, private-file access control (non-owner → 404).
  • Write paths driven end to end: login / wrong password / register + re-login / duplicate email, favorites add+remove, save, account edit, folder create + duplicate guard, upload (auto-category, validation), rename, move, trash → restore (confirm dialog), share link (+ public /shared/<token> page), comments, download → activity, premium checkout (100 GB / 500 GB / 1 TB pages, card validation).
  • Registry tests on this branch: test_exact_25_site_registry_and_port, test_docker_preserves…, test_shared_documentation_uses_25_site_range, test_no_merge_conflict_markers…, rotten_tomatoes test_task_manifest_and_registry — all pass. bash -n websyn_start.sh, py_compile of control_server/site_runner/site files OK, no cross-site imports.

Grading contract validation

verify/tests/drive_tasks.py drove all 20 tasks with a real Chromium against a freshly seeded standalone instance (agent.py trajectory format, initial.db/after.db snapshots); verify/tests/run_matrix.py then derived the negative variants and ran every verifier with --no_llm True:

variant expectation result
no-op (homepage only, empty answer, clean DB) 20 × FAIL 20/20
genuine Playwright run 20 × PASS 20/20
shortcut (correct answer + correct DB, every URL = homepage) 20 × FAIL 20/20
wrong answer / wrong persisted row 20 × FAIL 20/20
state-mismatch (stateful tasks 6–19, after.db = seed) 14 × FAIL 14/14

agent_demo/eval_judge.py --run_dir … --verifier True runs the verifier and exits 0 on the genuine run, 1 on the no-op.

Pending before merge

  1. Merge HF discussion feat(webmd): add WebMD mirror site (20 tasks, hardened) #62 and repin .assets-revision to the resulting immutable commit — neither the PR-era pin (070123d7…, 26 files) nor current main's pin (65c479f8…, 27 files) contains 4shared.tar.gz; refs/pr/62 does (14,209,943 bytes).
  2. Docker mechanical checks — done from this branch with refs/pr/62 assets: build ok (4.72 GB), 25/25 sites 200, /health 25/25, POST /reset/4shared byte-identical (b577adc2…) before/after a dirty request and after docker restart, /reset-all 25/25 in 1.76 s, verifier no-op matrix in the container 20/20 FAIL, registry tests green.
  3. Port 40024 is provisional (last SITES entry); the maintainer may reassign at merge — tasks.jsonl web and app.py SITE_PORT follow.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP

sarendis56 and others added 11 commits September 11, 2026 14:37
…ry tests

Rebased aiming-lab#90 onto upstream/main (24 sites, walmart_careers = 40023) and
registered 4shared as the LAST SITES entry (index 24 -> port 40024;
provisional, maintainer reassigns at merge):

- websyn_start.sh / control_server.py: append 4shared
- Dockerfile: 25 Flask mirror sites, EXPOSE 8101 40000-40024
- README / AGENTS / CONTRIBUTING / CLAUDE / agent_demo README: 25 sites,
  ports 40000-40024 (alt 41000-41024), site list adds 4shared
- walmart_careers + rotten_tomatoes registry tests: 25-site registry
- sites/4shared/tasks.jsonl: web 40023 -> 40024 (all 20 rows)
- sites/4shared/app.py: SITE_PORT 40023 -> 40024

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…k 16's plan

- 4shared--2: the original target (Pride and Prejudice) is the FIRST result
  for "classic fiction" in Books (ranking = score, then download_count, and
  it has the highest count), so the task was solvable by clicking the first
  result. Re-anchored onto The Time Machine (12 chapters, editorial notes
  after page 116), which sits at position 4 of 10 behind three near-misses.
- 4shared--16: "the annual Premium plan" was ambiguous — three annual Premium
  plans (100 GB / 500 GB / 1 TB) are offered. Named the 100 GB plan explicitly
  and asked for the plan name + storage allowance shown on My 4shared.

Both are reviewer rewordings; the contributor's other 18 tasks are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…lidation matrix

Reviewer grading contract for all 20 accepted tasks:

- sites/4shared/verify/verify_lib.py: run-package identity (task id, agent_done,
  non-empty answer, loopback+port origin check on every recorded URL, PNG
  screenshots), navigation gates (exact /file/<slug>, /search token, /category,
  login/workflow ordering), answer matchers (exact filename, standalone numbers,
  runtimes, displayed sizes, resolutions, comparison-claim reading) and SQLite
  after-state helpers with exact row deltas. No LLM in the verdict path; llm_*
  helpers kept for parity only.
- sites/4shared/verify/verify_{0..19}.py: hardcoded ground truth per task;
  read-only tasks require all 9 tables unchanged, stateful tasks the exact
  persisted delta and nothing else.
- sites/4shared/tasks.jsonl: verifier_path + English judge_rubric per row
  (rules only, no answers).
- sites/4shared/verify/tests/: drive_tasks.py (Playwright PASS runs against a
  freshly seeded standalone instance, agent.py trajectory format) and
  run_matrix.py (noop / pass / shortcut / wrong / state-mismatch). 100/100
  cells as expected with --no_llm True. Run dirs git-ignored.
- sites/4shared/verify/README.md: contract summary.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…fallbacks

Second-pass audit of the contributor's mirror. Each item was reproduced against a
standalone boot before the change and re-probed after it.

- /premium/checkout defaulted a missing or unknown ?plan to "100", so an agent that
  never chose a plan landed on the Premium 100 GB checkout — the exact target of
  task 16. A missing or unknown plan is now 404.
- The share form's permission <select> had no empty first option and the route fell
  back to "view", handing task 17 its required preview-only permission. The select
  now leads with a disabled empty option and an absent/invalid permission is 400.
- The upload form pre-filled size_kb=128 and carried an unchecked "public" checkbox,
  so an untouched form already submitted "private" (tasks 11 and 17). Visibility is
  now an explicit required choice and size_kb is required and bounded (1-4096 KB).
- A non-numeric folder_id silently fell back to the account root on upload, move and
  folder create; it is now 400, and an unowned folder stays 404.
- /file/<id>/favorite and /file/<id>/comment only checked "deleted", so any signed-in
  user could favorite or comment on another account's PRIVATE file (and its filename
  then rendered on the actor's own /favorites). Both now go through
  viewable_file_or_404, as does the share page.
- Upload slugs were "<slug>-<user>-<unix seconds>", so a second upload of the same
  name in the same second, or after the first was trashed, raised
  IntegrityError (UNIQUE files.slug) and returned a bare 500. Slugs are now derived
  from the rows that exist, the commit is rolled back on IntegrityError, and 400/500
  templates plus handlers replace Werkzeug's default pages.
- GET /logout logged an agent out on any prefetch; it is POST-only (GET -> 405) and
  the account nav submits a form.
- Premium pricing was non-monotonic (100 GB $77.88/yr vs 500 GB $29.99/yr vs
  1 TB $39.99/yr). The 500 GB and 1 TB annual prices now sit above the 100 GB one;
  the 100 GB price is unchanged, so task 16's ground truth still holds.
- The homepage computed three catalog queries that index.html never rendered, and
  the category/related listings ordered by download_count with no tiebreaker.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
The tab row hard-coded six categories and omitted Documents, while the filter
<select>, the nav and /category/documents all included it, so the Documents
catalog (16 records, including the targets of tasks 3 and 19) was unreachable by
tab. The row now renders nav_categories, the same list the filter uses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
review-reports/ holds per-PR audit markdown plus PNG evidence (340 KB today, and
it grows with every reviewed site). Nothing at runtime reads it, but it was being
COPYed into the image. Excluding it does not affect any site's behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
Tasks 11 and 17 upload a private record. Visibility is no longer implied by an
unchecked checkbox, so the drive harness selects it explicitly. All 20 tasks
re-drive green against the hardened mirror.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…trast

Checklist section 9 pass at 1440 / 768 / 390 / 320 against a standalone boot.
Every claim below was confirmed by pixel sampling, not by bounding rects or
computed styles alone.

- .file-icon was `color:#fff` unconditionally. Inside .media-preview (blue) that
  is fine, but on /preview/<id> and /shared/<token> the same span sits in a white
  .panel, so the extension label rendered white on white: an element screenshot of
  the 58x70 box contained exactly ONE distinct colour (#ffffff, 100% of pixels,
  zero glyph pixels). It now defaults to the action colour and only goes white
  over the media/thumbnail backgrounds.
- /my-files overflowed horizontally at 320px (documentElement.scrollWidth 332 vs
  clientWidth 320). Grid and flex children default to min-width:auto, so the
  new-folder input and long folder names widened the single-column track;
  min-width:0 plus wrapping on .inline-form fixes it. Overflow is now 0 at all
  four widths, with scrollable containers excluded via checkVisibility().
- WCAG: 51 distinct text/background pairs were below their threshold, led by the
  brand blue used as an interactive fill (white on #0797f6 = 3.104:1). Rather
  than restyle every rule, this introduces --action / --action-dark for
  interactive fills, links and focus targets and keeps --blue for decoration,
  retunes --muted (#8b9299 -> #6f757b, 4.53:1) and folds the one-off greys into
  it, and darkens the active tab, folder card, plan badge, plan-current,
  icon-button, footer links and the media/thumbnail backgrounds.

Six of the eight apparent residuals were the audit tool's own false positives:
a.file-thumb / a.result-visual inherit a link colour but contain no text node of
their own (their glyph is a child span), and sampling found 0 pixels painted in
the anchor's colour. The audit script now scores only elements that paint their
own text and skips 1x1 visually-hidden nodes (.sr-only).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…0026)

main gained fedex (40024) and webmd_doctor (40025). Merged rather than rebased so every
contributor and reviewer commit on this branch is preserved verbatim. Shared registry,
Dockerfile EXPOSE, docs and every site-local port reference move to index 26 / port 40026;
the slot stays provisional and the maintainer reassigns it at merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
@JeremyJC67

Copy link
Copy Markdown
Author

Rebased onto the 26-site main — this branch is now site 27 / port 40026

main merged FedEx (40024) and WebMD Doctor (40025) overnight, which put every open site
branch into conflict. This branch has been brought up to date by merging main rather than
rebasing, so every contributor and reviewer commit is preserved verbatim, and re-slotted to
index 26 / port 40026: registry, Dockerfile EXPOSE, the shared docs and every
site-local port reference follow. The slot is provisional as always — say the word at merge
time and I will re-slot in whatever order you land things.

The whole environment was then rebuilt and re-verified at the new slot:

check result
HTTP sweep 27/27 return 200
GET /health ok, 27/27 alive and ready
POST /reset/4shared → md5 b577adc216900a6f0e3974a80e51c04c, identical to the seed
POST /reset-all ok, 27/27 ready, 1.59 s
verifier no-op matrix 20/20 FAIL
registry + docs tests 4 passed, 3,343 subtests

Audit assertions hold: GET /logout 405, POST /upload without a session/token 400, unknown
route a branded 404 (3,029 bytes), anonymous /account 302, the Documents tab present in the
search category strip, and six read-only page loads leave the DB byte-identical.

Still blocked only on HF #62 being merged, after which I repin .assets-revision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants