Review: PhET Interactive Simulations mirror — real catalogue, responsive repair and task verifiers (site by @ZhongyangLi0124)

[jackjin1997]

This is the review vehicle for #29. It keeps @ZhongyangLi0124's three commits intact
(authorship preserved, rebased onto current main) and adds the reviewer's repairs and
grading contract. Registered as site 29 on port 40028.

What the review found

The Flask app, auth, CSRF and reset behaviour were sound. Two things were not.

43 of the 98 seeded simulations do not exist on phet.colorado.edu. Each of the 49
placeholder thumbnails the original PR describes is a purple gradient card. Checking their
slugs upstream: 2 exist as written, 4 exist under a de-mangled slug (beer-s-law-lab →
beers-law-lab, same for Coulomb's, Faraday's and Hooke's Law), and 43 return 404 —
Cellular Respiration, Photosynthesis, Water Cycle, Predator-Prey Dynamics, Mendelian Genetics,
Punnett Squares and 37 others, invented along with their titles, descriptions, subjects, grade
bands, versions, release dates, play counts and language lists. Nine contributor tasks were
anchored on them. This could not be fixed by swapping images: there is no screenshot for a
simulation that does not exist.

Every page overflowed horizontally below ~840px. The desktop nav, search form and user menu
never collapsed and by themselves forced the document to 837px at both 768 and 390, leaving body
content in the left 390px with the rest of each page blank.

What this PR does

Commit	Author	Content
1877093, e0c1a9d, 4e6cd90	@ZhongyangLi0124	the site, unchanged but for the rebase
c7b5a41	reviewer	register the site; sweep repo docs to the new count
aa2b512	reviewer	responsive header and layout
0c6eec8	reviewer	rebuild the catalogue from real PhET data
ea5e87c	reviewer	rebuild the task set, add the grading contract
25fa7a9	reviewer	save-note field; RTL text isolation
908ef82	reviewer	bind reported counts to what they count
629f56c	reviewer	re-slot to site 29 / port 40028

The catalogue is the real HTML5 library, harvested 2026-09-13 from PhET's official metadata
service: 120 simulations with upstream titles, descriptions and sample learning goals; subject and
topic membership from the numeric facet ids, whose meaning was confirmed by intersecting each id
against the live filter (Physics 65, Math & Statistics 51, Chemistry 35, Earth & Space 19,
Biology 8 — exact on all four); real grade bands, version strings, release and last-updated dates,
per-simulation locale lists and relatedSimulations; the topic strip from each upstream
simulation page; the 131-language translation table, whose upstream HTML5 counts agree with the
seeded per-locale counts; and the official 600px screenshot for all 120, replacing every
placeholder. PhET simulations are CC-BY.

Facts PhET does not publish are removed rather than invented. Play counts, download counts and
per-simulation runtime are gone from the model. The homepage's third rail is Recently Updated and
the catalogue sorts by Recently updated / Most translated.

Also fixed: the topic facet now filters (the route read no such parameter, leaving 11 topic, 3
compatibility and 3 inclusive-feature checkboxes inert — the latter two groups, which an
HTML5-only snapshot cannot back with data, are replaced by a statement of that fact); the
Customize tab rendered an empty page and now lists the 49 simulations upstream marks as available
in PhET Studio; search tokenises, so build atom reaches Build an Atom instead of returning
nothing; the sidebar gains Heat & Thermo and uses upstream's Math Concepts / Math Applications
names; teacher activities are relabelled as benchmark fixtures with synthetic authors rather than
attributing invented lesson plans to real, named PhET contributors; the save form gained the note
field its API always accepted; and right-to-left language cards no longer reverse their English
text (simulations 119 → 119 simulations).

Grading contract

18 tasks replace the contributor's 43. Nine targeted fabricated simulations, four read the removed
play counter, one asked which scripts are right-to-left (answerable without opening the site) and
one only asked the agent to click through to a page with nothing to report. The new set is
anchored on facts that live on this site and that a model cannot recall — exact version strings,
release dates, per-language and per-facet counts, the related-simulations list, account state —
and none appears on a listing card, so each requires opening the page that carries it.

sites/phet_simulations/verify/verify_lib.py plus verify_0..verify_17, recorded as
verifier_path and a rules-only judge_rubric on every row. Ground truth is hardcoded in the
verifiers; tasks.jsonl has no answer key.

Validation, all deterministic (--no_llm True):

• 18/18 canonical runs pass, each verifier bound to that run's own frozen initial.db and
  after.db rather than reading live state;
• 90/90 adversarial cells match expectation — no-op, wrong answer, shortcut without
  navigation, an unexpected write on every read-only task, and a missing state change on both
  stateful tasks all fail on the intended check. The matrix ships at
  sites/phet_simulations/verify/tests/adversarial_matrix.py;
• 6/6 legitimate rephrasings still pass, including spelled-out numbers.

Three verifier defects surfaced in that matrix and are fixed: the two saved-simulation tasks
graded purely on the database delta, so an answer describing an unrelated simulation passed; and
a bare number check accepted version 9.9.9 as the count 9. Counts are now bound to the thing
they count.

Independent review

The frozen runs were judged in a separate session with no access to the verifiers, their verdicts,
or any reviewer conclusion. Result: 18 PASS / 0 FAIL, all 226 bundle files hash-verified, all
18 initial.db byte-identical (clean per-task reset), both stateful tasks confirmed by
before/after diff. It agrees with the deterministic run 18/18.

An earlier round on a previous bundle also returned 18/18 but flagged evidence-quality gaps.
Acting on those exposed the save-note defect above (the run harness had been issuing a raw API
call and recording it as a button click, which hid that the task was not completable through the
UI at all), the RTL defect, and batch-level state snapshots presented as per-task. All three are
fixed and the runs were re-recorded with a reset and a state snapshot around every task.

Assets

Paired asset PR: ChilleD/WebHarbor discussions/79, superseding #19 whose seed carries the
fabricated catalogue and the placeholder thumbnails.

refs/pr/79  phet_simulations.tar.gz  10,319,041 bytes
sha256      7ed4831ed1218bf95456a6156f82dbd769b909fb630434e042f6e6cd1987a65f
seed md5    6eed11be4c24be21e9c647caee573745

A site built from this branch's tracked code plus that tarball alone boots and serves every route,
thumbnail, logo and hero image.

.assets-revision is pinned to that PR commit (82c2764b…) so this branch builds standalone
rather than waiting on the merge. That commit is upstream's current pin (64264d0) plus
phet_simulations.tar.gz; every other archive is byte-identical, verified by comparing the file
OIDs of both trees. Repin to the merge commit once #79 lands.

Verification run

Check	Result
Control plane health	pass
Site routes	16/16 → 200
Dirty DB → POST /reset/phet_simulations	byte-identical to seed
POST /reset-all	2s, byte-identical after, site 200
Restart re-seed	no-op, md5 unchanged
Registry three-way sync	identical 29-entry lists, no duplicates, EXPOSE 8101 40000-40028
Responsive sweep, 20 pages × 1440/768/390	63/63 fit their viewport, from 40 overflowing
Placeholder thumbnails remaining	0
Negative/functional audit	auth, CSRF, registration validation, cross-account isolation, bad input, read-only invariant — all pass

Full 29-site environment smoke, run 34804778961 (green, artifact attached to the run):

Check	Result
Image build from this tree	./scripts/build.sh completes
Asset fetch at the pinned revision	tarball sha256 and seed md5 both asserted and matched
Control plane	29 sites, ok=true, all alive
HTTP sweep 41000–41028	29/29 → 200, before and after reset-all
Byte-identical reset of this site	seed → dirtied → back to seed
reset-all	7s, all 29 ready
Registry inside the built tree	29 sites, lists identical, this site at index 28 → 40028
Grading contract inside the built tree	18 tasks, each with a verifier file and rubric, no answer key

The workflow lives on a separate ci/phet-full-smoke branch and is not part of this PR. Its tree
differs from this PR's head only by that workflow file.

Known limitations

• The full 29-site image build was not run. Done — see the verification table above.
• The 18 runs are scripted fixed-script walks, not independent agent exploration; each
  trajectory records that in run_kind.
• sites/rotten_tomatoes/tests/test_environment_quality.py::test_task_manifest_and_registry
  fails on pristine main at line 101 — it asserts a literal site list that upstream reflowed
  across lines. Reproduced in a clean worktree of origin/main; left untouched here.
• Benchmark users are seeded with bcrypt hashes using random salts, so the seed database is not
  byte-reproducible from source alone. The shipped tarball is the baseline; reset stays
  byte-identical.
• The real PhET homepage carries no per-simulation rails. This mirror keeps Featured / New Sims /
  Recently Updated so simulations remain reachable from the homepage for the benchmark — a
  deliberate deviation, not a fidelity match.
• Teacher activities are benchmark fixtures with synthetic authors; the real activity listing sits
  behind interaction and was not reproduced.

Merge order

This PR no longer blocks on the asset PR: .assets-revision points at the HF PR commit, so a
fresh clone fetches and builds today. The preferred order is still:

1. Review and merge HF discussion review(bandcamp): add deterministic task verifiers #79;
2. repin .assets-revision to that merge commit (happy to push it, or fold it into the merge);
3. merge this PR.

Merging in the other order works too — the pin is valid either way, it just wants tidying
afterwards. Original contribution and full context: #29.

🤖 Generated with Claude Code

[jackjin1997]

Blind review — frozen runs, WHR-029 (phet_simulations)

Independent pass on the frozen bundle final-3bed3bb-20260913. No verifier source,
no verifier verdicts, no prior review material was consulted before the result was
written and hashed.

Result: verdict.json — SHA-256 a20891944c81d880f2fd794a52095699940ce86847269ae840ae5f2f7b673d5b
(frozen 2026-09-13T14:58:51Z, before any deterministic output).

What produced the runs

• model: scripted-guided-walk — run_kind: guided_fixed_script on every task.
  These are fixed-script UI walks, not independent agent exploration. Judged on whether
  the recorded actions and final answer satisfy the task, not on being scripted.
• Driver: Playwright via Chrome 152.

Commits / assets

• Code: 3bed3bb878af3df6b1565b76c53002e3ec1fd3a6 (base f20b5ee8377ba31bcb825b4dfe30ad96c416e477), no uncommitted diff.
• Asset: ChilleD/WebHarbor refs/pr/79 → phet_simulations.tar.gz
  sha256 7ed4831ed1218bf95456a6156f82dbd769b909fb630434e042f6e6cd1987a65f, seed md5 6eed11be4c24be21e9c647caee573745.
• Packet integrity: manifest hash matched; all 132 enumerated files (18 trajectories,
  78 screenshots, 36 SQLite snapshots) matched their listed SHA-256. No missing or extra files.

Coverage

• 18 / 18 tasks judged. 18 PASS, 0 FAIL.
• Every task's rubric-required page was reached (verified against trajectory URLs + screenshots),
  and every reported value matches the environment's own state.
• Both state-changing tasks match their rubric exactly:
  • task 13 → exactly one new saved_simulation row (student@phet.test → membrane-transport, non-empty note); no other account touched.
  • task 14 → exactly one new user row (test_user@phet.test, teacher) and exactly one new saved row (number-pairs).
  • task 12 required no mutation: after.db is byte-identical to initial.db.

Failures

None.

Checks NOT executed / evidence limits

1. Deterministic verifiers and their verdicts were excluded by design — this pass judges the
   frozen executions only. It does not certify the verifier implementation.
2. Source fidelity not certified. Whether the mirror faithfully reflects upstream
   phet.colorado.edu was out of scope; no upstream comparison was made.
3. Screenshots are viewport-limited (1440x900, top of page). Values below the fold are
   never visually confirmed, though each matches environment state:
   • tasks 1, 4, 17 — per-simulation translation counts (104 / 31) appear in no image.
   • task 3 — the five 2025 release dates appear in no image (each of those sims has 29-40
     languages, which pushes the SIM DETAILS block past the fold); only the 2026 date is legible.
   • task 7 — weakest case: single goto with no scroll, screenshot byte-identical to task 6's,
     so the Arabic (Morocco) card itself never appears. Passed because the answer reports the
     region-specific 103 rather than Arabic's 119.
   • task 10 — the related-simulations section is below the fold; the answer matches the page's
     stored set exactly (all five, same order).
4. Save actions not visually captured. In tasks 13 and 14, step_004 and step_005 are
   byte-identical, so no image shows the save form or a confirmation; the writes are evidenced
   solely by the before/after state diff. Note text is likewise only in the resulting row — the
   trajectory abbreviates typed input (passwords render as ***).
5. Incomplete enumeration in two walks (did not change either answer):
   task 2 opened 6 of the 8 listed Biology sims (skipped neuron, ph-scale; both carry
   university, so "exactly one" holds); task 8 opened 6 of 7 search hits (skipped rutherford-scattering).
6. Port discrepancy, unresolved. The manifest's environment reference states
   site_port: 40024 and each task's web is http://localhost:40024/, but every recorded
   trajectory URL is http://127.0.0.1:55025/. File hashes all match, so the packet is valid;
   flagging it as a manifest/recording inconsistency to reconcile, not as a task failure.

[jackjin1997]

Independent blind review — bundle final-02c982f-20260913b

Frozen executions judged in isolation, before any deterministic result.
Verdict artifact: verdict.json, SHA-256 669d58f169afcabfe1fb3932d8246131a41b4678fa1a244eda5d497768661b94.

Model

• Reviewer: Claude Opus 5 (claude-opus-5), fresh session, no inherited case context.
• Runs under review: model: scripted-guided-walk, run_kind: guided_fixed_script (fixed-script UI walks, Playwright via Chrome 152 — not independent agent exploration). Judged on whether the recorded actions and final answer satisfy each task; not marked up or down for being scripted.

Commits / assets

• Code: 02c982fe7893ec41d37ad6cb16f01eb741f98ff7 (base 145b200eab9721e4940a36a0e3d0b3b62677715b), uncommitted_diff: none.
• Asset: ChilleD/WebHarbor refs/pr/79 → phet_simulations.tar.gz, sha256 7ed4831e…7a65f, seed md5 6eed11be…73745.
• Manifest sha256 59a89d8d…3be510 — verified before any input was read.

Coverage

• 18 / 18 tasks judged. 18 PASS, 0 FAIL.
• All 226 manifest-listed files hash-verified, zero mismatches: 18 trajectories, 36 DB snapshots, 172 screenshots (86 viewport + 86 full-page).
• All 18 initial.db files are byte-identical → the per-task reset is clean and deterministic.
• Both state-changing tasks verified by before/after diff, not by self-report:
  • task 13 — exactly one new saved_simulation row (student@phet.test → membrane-transport, non-empty note); no other account's list changed; user table untouched.
  • task 14 — exactly one new user row (test_user@phet.test, role teacher) and exactly one new saved_simulation row (that user → number-pairs); nothing removed.
• All 16 read-only tasks have initial_db_sha256 == after_db_sha256, including task 12, whose rubric forbids changing the saved list.

Failures

None. No answer contradicted its screenshots or the environment state; no empty answers.

Observations (non-blocking, harness-side)

1. On tasks 3, 10 and 17 the graded content (SIM DETAILS release date / translation count, Related Sims) sits below the fold of the 1440×900 viewport capture and appears only in the -full companion. Tasks 1 and 4 include an explicit scroll step; 3/10/17 do not. The manifest says step_NNN.png "is the 1440x900 viewport capture the harness grades on" — if that is literally true, the harness grades a crop that omits the evidence. Answers are still correct; this is a capture-policy inconsistency, not a run defect.
2. Task 8 opened 6 of the 7 search hits (skipped Rutherford Scattering). The rubric only requires the 2026 result, which was opened.
3. Task 6's scroll to the "Arabic row" is a no-op — Arabic is already in the first visible row of /translations.
4. Every trajectory carries success_self_report: true; no verdict here relies on that field.

Checks NOT executed (excluded by design or out of scope for a blind pass)

• Deterministic verifiers, their verdicts, and sites/phet_simulations/verify/** — not read.
• The previous bundle (final-3bed3bb-20260913) and its blind result — not read.
• Reviewer conclusions, the case evidence/ directory, prior reviews, PR comments, repo history — not read before verdict.json was written and hashed.
• No re-execution: the environment was not booted and no task was re-run. Judgement is on the frozen recordings plus the bundle's own initial.db / after.db.
• Source fidelity not certified: no fetch of phet.colorado.edu. Task-level facts were cross-checked only against the bundle's own database snapshots, so this pass cannot say whether the mirror faithfully reproduces upstream.
• Verifier implementation not certified: this result judges the executions only.
• The seed tarball itself is not in the bundle; only the manifest's declared hash was noted, not recomputed.

[jackjin1997]

Verification update — full environment now covered, branch builds standalone

Two gaps from the original description are closed.

Full 29-site environment smoke: green

Run 34804778961, artifact
attached to the run. Reported from the artifact's own files rather than the badge:

Check	Evidence file	Result
Asset identity at the pin	candidate-sha256.txt	tarball 7ed4831e…7a65f, seed md5 asserted 6eed11be…
Registry inside the built tree	registry.txt	29 sites, both lists identical, this site at index 28 → 40028
Image build	build step	./scripts/build.sh completes
Control plane	health-before.json	29 sites, ok=true, all alive
HTTP sweep	http-before.txt / http-after.txt	29/29 ports 41000–41028 → 200, before and after reset-all
Byte-identical reset	phet-reset-md5.txt	6eed11be… → 8e44dea5… → 6eed11be…
reset-all	reset-all-seconds.txt	7s, all 29 ready
Grading contract	task-contract.txt	18 tasks, each with a verifier file and rubric, no answer key

The run takes under four minutes, which looks fast for a 29-site image. The artifact is what
backs the claim: 29 distinct port results twice over, three distinct md5 values across the reset
cycle, and hashes computed from the fetched files.

It ran on a fork branch (ci/phet-full-smoke) rather than locally because the review machine was
down to 13 GiB while the runner had 69 GB. That workflow is not part of this PR; its tree
differs from this PR's head only by the workflow file itself.

The branch no longer waits on the asset PR

.assets-revision now pins to 82c2764b6b309b7e00729d7280cc9a9a5e678d72, the commit on
ChilleD/WebHarbor discussions/79, so a fresh clone fetches and builds today.

That commit is upstream's current pin 64264d0 plus phet_simulations.tar.gz. I compared
the file OIDs of both trees: 31 archives on upstream's pin, 32 on this one, zero paths with
differing content. Pinning here cannot regress another site's assets. The CI run above fetched
through this pin and passed check_assets.sh.

It is a temporary pin and wants moving to the merge commit once #79 lands — happy to push that,
or fold it into the merge.

Still open, and not mine to fix

sites/rotten_tomatoes/tests/test_environment_quality.py::test_task_manifest_and_registry fails
on pristine main at line 101. It asserts the literal 'ted', 'osu', 'rotten_tomatoes' appears
in control_server.py, and that list was reflowed across lines upstream. Reproduced in a clean
worktree of origin/main, same line, same failure. Left untouched here since it belongs to
another site.

Head is now 4b994fa2339a2cbc4a80280f9329c7a9d1e5579a, mergeable and clean.