Review: Add Versus mirror + task verifiers (site by @Sun-sunshine06, verifiers by reviewer) (#41)

[jackjin1997]

Reviewer pass over #41 (Versus mirror by @Sun-sunshine06). The site commit is kept as the author's, rebased onto current main; everything after it is reviewer work.

Draft on purpose — one deliberate deviation from the reviewer checklist is called out at the bottom for the maintainers to rule on. The environment, interface and grading work is finished and reproducible.

---

What the contributor supplied

A Versus mirror under sites/versus (Flask app, code-generated SQLite seed, 20 products across 5 categories) and 10 task definitions with the contributor keys only. No verifiers — correct for the contributor role per CONTRIBUTING.md.

@Susan571 reviewed it at source level on 2026-06-21 and asked to pair it with a Docker-capable mechanical pass. This is that pass, plus the grading contract.

Environment fixes

The generated seed was not byte-reproducible across builds. Two builds of the same commit produced different seed bytes (96d9d1dc… vs 568bfb9d…). Logical content was identical both times — 5 categories, 20 products, 4 users, 3 saved comparisons hash the same with password_hash excluded — so the only divergence was generate_password_hash() drawing a fresh scrypt salt per build. The in-image reset contract still held (the seed is copied from that same generation), which is why it never surfaced as a failing md5 check; what it broke is pinning the seed hash anywhere. PYTHONHASHSEED=0, which osu uses for its build-time seed, does not help. Frozen the hash the way osu freezes BENCHMARK_PASSWORD_HASH. Two builds now agree on 231149c4….

No CSRF protection. A POST /compare/<l>-vs-<r>/save carrying Origin/Referer of another site was accepted and wrote the row. 23 of the 25 sites on main install CSRFProtect; this one did not. Installed it, added tokens to the two posting forms, declared Flask-WTF in the site requirements. Also stopped pre-filling the benchmark email and password into the login inputs — the credentials stay in the task text, which is the repo convention, but a page that ships them in value attributes lets an agent sign in without reading anything.

Port and favicon. The branch targeted the 15-site tree and claimed 40015. versus now registers after fedex, webmd_doctor and healthline at index 27 and serves on 40027, and every task row followed the old port. The missing /favicon.ico was the only console error on every page load.

Mechanical results at this head: all 9 routes return 200 at 1440/768/390/320 (36/36), zero external requests, zero page errors, POST /reset/versus succeeds and leaves md5(instance) == md5(instance_seed), and a functional/negative sweep passes 19/19 — wrong password rejected, /account and the save endpoint gated when signed out, no cross-user leak between benchmark accounts, unknown slugs 404, hostile search string and non-numeric filters handled.

Grading contract

17 tasks, 17 deterministic verifiers, 17 rubrics.

Three of the original questions were answerable without opening anything. Versus prints Score, Price and Year on every product card, and /rankings lists all 20 products ordered with score, price and category and needs no filter — so "which of these two scores higher" (--0), "highest ranked graphics card" (--2) and "highest scoring smartphone under $1000" (--6) could all be read off a list. The fix is not to hide the ranking, which is a real page of the site, but to ask for a fact the list does not carry: every spec value (camera score, ANC score, megapixels, burst, VRAM, power, benchmark, battery, weight, display) renders only on detail and comparison pages, so each rewritten question now ends in one of those.

The one stateful task was a no-op. Versus--7 asked Alice to save the RTX 4080 Super vs RX 7900 XTX comparison, which seed_benchmark_users() already saves for her, and save_comparison() de-duplicates — the after state was identical whether or not the agent acted. It now targets a pair she does not have, and a second save task uses Bob, who starts with none.

One question had two valid answers. "A camera with 40 megapixels or more" matched both the Nikon Z8 (45.7) and the Fujifilm X-T5 (40.0). It now asks for the highest count.

Verifier design: ground truth is derived from the passed initial_db, not frozen, so the expected answer moves with the seed and a stale verifier fails loudly instead of grading against a dead value. Superlative questions use a helper that returns nothing on a tie and fails closed. Malformed or missing input produces a structured FAIL, never a traceback. Stateful tasks read the after-state DB and also assert the pair was absent before, so a task regressing into a no-op is caught by the verifier itself.

For the record, one item from the earlier review does not hold: Versus--5 putting the benchmark password in the task text is not a leak — 59 task rows across the existing sites do the same.

Verification

17 canonical UI runs, re-recorded after the interface rebuild. Each resets through the control plane, starts at the site root and clicks its way to the answer, recording a trajectory in agent_demo/agent.py's schema plus screenshots and before/after DB snapshots. All 17 complete with zero page errors and all 17 pass their verifier. Environment sweep at this head: 40/40 route×viewport responses are 200, with zero external requests and zero console errors.

Independent blind review

A separate session with no access to the verifiers, their verdicts, this branch or these PRs judged the 17 frozen executions from a hashed packet. It returned 17/17 PASS, agreeing with the deterministic verifiers on every task — and, on runs it passed, recorded two things the verifiers had missed.

A run answered from a browser error page. On Versus--10 the final answer was emitted at chrome-error://chromewebdata/ after a failed navigation. Every fact-bearing read had already happened, so the answer was right — and all 17 verifiers passed it. That is the finding: navigation checks only look for evidence that the right page was opened, so a run that crashed and then answered was indistinguishable from a clean one. Reproduced with a fixture before fixing; answered_on_site() now requires the answer-bearing step to be on this site's origin, the runner aborts on off-site navigation instead of recording it as a step, and the defective bundle was re-recorded rather than accepted.

Superlative runs opened only some candidates. On five tasks the guided run opened two of four products in the category. The reviewer checked the seed, confirmed each superlative still held, and passed them while noting it. That is a guided-run artefact, not a verifier defect — requiring exhaustive visits would reject legitimate routes such as search or the ranking view, which the alternative-route runs exercise. Fixed in the evidence: those five now open all four.

Not captured, and recorded as a gap rather than reconstructed: the blind reviewer's overall statement of model, coverage and unperformed checks. The per-task rationales establish the packet identity; the model identity is not established.

One run deserves mention. During the batch, Versus--7 came back with an after-state byte-identical to the seed — its save had been lost to container instability — while the agent's own answer claimed success. The verifier failed it, which is exactly what the DB check exists for. Re-run in isolation the save persists and it passes. Run manifests now record before_sha256/after_sha256/state_changed, and a run-integrity index asserts stateful tasks changed state and read-only ones did not: 17/17 consistent.

These are guided runs — the reviewer knew the answers, and every run manifest says so. They show each task is solvable through the UI and they exercise the verifiers on real trajectories. They are not a measurement of agent capability and should not be read as one.

Adversarial matrix: 205 cases, 0 expectation mismatches, 0 tracebacks. Before trusting any verdict the harness grades a genuinely defective bundle while deliberately asserting it should pass, and aborts if it reports a match — so the green result is not vacuous. Cases: no-op, correct-answer-without-navigation, wrong answer, negated answer, truncated trajectory, foreign origin, wrong-task replay, unparseable trajectory, empty run dir, missing after-state.

The matrix found a real hole in my own verifiers. foreign_origin initially passed all 17: every navigation check was a bare substring match on the path, so rewriting a canonical trajectory's URLs from localhost:40024 to localhost:40007 — same paths, different mirror — satisfied all of them. Navigation checks now keep only steps on this site's own origin, with the port derived from control_server.py's registry rather than frozen, so a registry reorder moves the verifier with it.

Five legitimate alternative routes pass (tasks 0/3/6/12/16: two detail pages instead of the comparison; search by name instead of category word; rankings instead of the price filter) — the counterweight to the negative cases, so a correct execution that took a different path is not failed.

A regression suite pins each of these defects; 7 of its 11 tests fail against the branch as submitted.

Reproduce

git fetch origin review/pr-41-versus && git checkout review/pr-41-versus
./scripts/build.sh
docker run -d --rm --name wh-versus -p 8101:8101 -p 40000-40027:40000-40027 webharbor:dev
curl -so /dev/null -w "%{http_code}\n" http://localhost:40027/
curl -X POST http://localhost:8101/reset/versus
docker exec wh-versus md5sum /opt/WebSyn/versus/instance/versus.db /opt/WebSyn/versus/instance_seed/versus.db
python3 -m unittest discover -s sites/versus/tests -v

Grading a run uses the standard entry point: uv run python agent_demo/eval_judge.py --run_dir <run> --verifier True.

Rebased onto current main, twice

The branch was first rebased from the 15-site tree onto 3600493. Upstream then
advanced three times while this review was open — FedEx (#82), WebMD Doctor
(#95) and Healthline (#105) — so this branch carries merges that keep the slots
those three already occupy and append versus after them: fedex 40024, webmd_doctor 40025,
healthline 40026, versus 40027. No earlier site moves.

Only the address changed, not the site — but the recorded trajectories carried
the old port and the verifiers derive their expected origin from the registry,
so all 22 runs were re-recorded and re-verified after each move rather than
graded with an origin override. Every number above is from the current port.

Data scale — a limitation, stated

The catalogue is 5 categories with 4 products each. Filtering and sorting
exist and are exercised, but they operate on four-item sets where the same
answer is readable off four cards, and no surface has enough rows to
paginate — in a mirror of a large-catalogue comparison site. The tasks are
solvable and graded correctly; the paths they exercise are thinner than the
source's.

The comparable upstream figure — products per category — could not be read,
because versus.com refuses this client, so it is recorded as unknown rather than
estimated. Widening the catalogue means sourcing real specs and prices for each
added product, which is a scope call rather than something to quietly expand.

Interface rebuilt against the source design

The mirror shared no design language with versus.com — a light blue-and-white template with a round "VS" badge, two flat sections and a one-line footer, against a near-black to deep-purple editorial site.

Rebuilt from evidence captured before the source began refusing this client: dark ground with the purple wash and wave cutout, lowercase wordmark, the oversized two-line compare everything hero with its accent underline, pill controls, score rings, four-column footer. The comparison page also carried the wrong shape — the source resolves a comparison per area with a margin per row, not as a single number — so the table now has a Margin column with the leading side marked and the ruling band reads "leads in N of M areas".

What the site declares the winner is deliberately unchanged: still the higher Versus Score. Tasks 9 and 16 are graded on that, and matching the source there would have moved the answer under the verifier rather than fixing presentation. The per-area counts are shown alongside.

Colours are read off screenshots, not sampled from the source stylesheet — versus.com started returning CloudFront 403 to this client mid-review and stayed blocked across repeated probes, and no attempt was made to work around it. The CSS header records them as estimates rather than claiming they are exact.

Not mirrored, and not pretended otherwise: the editorial/blog mosaic, the ~100 other categories, the glossary, the locale switcher.

Product art: synthetic by design, and gated

The art was previously drawn per request by a Flask route — no files, no hashes, no gate, and a placeholder that did not say it was one. It is still synthetic, deliberately, and now holds to the contract webmd_doctor established on main (it ships Pillow-drawn avatars and gradient panels "instead of photography"):

• generate_art.py is deterministic — no RNG, clock or locale, Pillow's bundled font, fixed PNG compression, no ancillary chunks
• generated_asset_inventory.json pins every tile's path, byte length and SHA-256
• check_generated_assets.py enforces coverage, size, hash and a full PNG decode, and runs in the Docker build
• tiles are build products: static/images/ stays gitignored and the inventory travels in Git
• each tile carries a visible SYNTHETIC ART label

Why not real photography: the source is blocked, and freely licensed images for these exact models could not be matched reliably. A Wikimedia Commons sweep found a freely licensed candidate for 18 of 20 products, but strict model matching showed most hits were the wrong item — a OnePlus 8 for the OnePlus 12, an A7R IV for the A7 IV, a 4070 Ti Super for the 4070 Super, a card-slot close-up for the Nikon Z8, earbuds for over-ear headphones. Shipping those would put false product facts into a benchmark built on factual navigation, which is worse than art that admits what it is.

This deviates from the reviewer checklist's "Real images" line. NOTICE.md states that plainly rather than glossing it, along with the non-affiliation notice and a removal procedure. It is the maintainers' call whether to accept the deviation.

The Versus Score is synthetic, and the site now says so

Product names, brands, release years, list prices and published specifications follow the manufacturers' figures — spot-checked against Momentum 4 60 h, WH-1000XM5 30 h / 250 g, Nikon Z8 45.7 MP, RTX 4080 Super 16 GB / 320 W, iPhone 15 Pro $999, S24 Ultra $1299. The Versus Score is not versus.com's value; it is benchmark data, as are all accounts and saved comparisons.

Rather than invent a provenance for it, the mirror discloses it: a new /about page, the footer on every page, NOTICE.md and README.md. Six of the 17 tasks derive an answer from score, so if a maintainer later re-sources it those tasks need re-running; the spec-, weight-, battery- and price-keyed ones do not.

Remaining, non-blocking

.dockerignore excludes sites/*/tests/ but not sites/*/verify/, so verifiers ship inside the image — repo-wide, matching merriam_webster, not introduced here. Product summaries are template strings rather than editorial text. /rankings sorts on score with no secondary key and the global list contains ties; no current task depends on tie order.

The anchored LLM check at the end of each verifier is unverified: this host has no OPENAI_API_KEY/JUDGE_MODEL, so everything ran deterministic-only and those checks reported [SKIP]. The deterministic checks carry every result above.

The full 27-site image build and reset-all were not run — the host is well below the disk headroom that needs. All checks ran against a single-site container.

---

Site and mirror by @Sun-sunshine06 (#41); source-level review by @Susan571. Environment fixes, interface rebuild, task rework, verifiers and verification by the reviewer. Head: 7aba94b58bf0aae083c45fec026fb5ecfcc27f74.

🤖 Generated with Claude Code