Review: UC Berkeley mirror + task verifiers (site by @richard-peng-xia, verifiers by reviewer) - #116
Open
evanz37 wants to merge 24 commits into
Open
Review: UC Berkeley mirror + task verifiers (site by @richard-peng-xia, verifiers by reviewer)#116evanz37 wants to merge 24 commits into
evanz37 wants to merge 24 commits into
Conversation
Adds a full Flask mirror of berkeley.edu as the 16th WebHarbor site. **Site features:** - 8 SQLAlchemy models: College, Department, Program, NewsArticle, Event, ResearchCenter, Faculty, Bookmark (+ User with auth) - 20+ routes: homepage, news, programs, events, research centers, departments, faculty, admissions, about, unified search - 23 Jinja2 templates styled with Berkeley Blue (#003262) / Gold (#FDB515) - 30 benchmark tasks in tasks.jsonl (WebVoyager schema) **Seed data (fully idempotent):** - 14 UC Berkeley colleges/schools (real names) - 83 degree programs (BA/BS/MA/MS/PhD/MBA/JD/MD/MEng) - 121 news articles (2023–2025, 7 categories) - 64 events (upcoming + past, 7 categories) - 25 research centers (BAIR, QB3, MSRI, …) - 82 faculty (Jennifer Doudna, Stuart Russell, Saul Perlmutter, …) - 4 benchmark users: alice/bob/carol/dave (password: test1234) **Infrastructure changes:** - control_server.py: add 'berkeley' to SITES (port 40015) - websyn_start.sh: add 'berkeley' to startup array - Dockerfile: EXPOSE 40015, generate instance_seed DB at build time (no HF assets needed — all data is code-generated via seed_data.py) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…6 / port 40026, docs sweep Merges upstream/main (26 sites @ f20b5ee) into the UC Berkeley PR branch. Berkeley is appended as site index 26 rather than inserted at the PR's original index 15: appending keeps merriam_webster..webmd_doctor on 40015-40025, so every merged site's tasks.jsonl "web" field and the fixed-port assertions in walmart_careers/rotten_tomatoes tests stay true. Inserting would have silently remapped 12 sites. - websyn_start.sh: keep upstream's SITE_COUNT form, append berkeley last - control_server.py: append 'berkeley' last (list == shell array, 27 entries) - Dockerfile: keep every upstream RUN block; EXPOSE 8101 40000-40026; header 27 sites - sites/berkeley/app.py: __main__ port from $PORT with 40026 default (was hard-coded 40015) - sites/berkeley/tasks.jsonl: all 30 rows repointed to http://localhost:40026/ - docs: README/AGENTS/CONTRIBUTING/CLAUDE/agent_demo README + 5 skills swept 40000-40025 -> 40000-40026 and "26 sites" -> "27 sites"; README mirror list gains UC Berkeley. agent_demo/README.md is included because walmart_careers' integration test asserts the current range appears in all five shared docs. - .assets-revision: keep upstream's pinned sha ad6f424f (the PR branch had left it at "main") Verification: step 1 (git status has no UU; author commit fa70f03 unchanged at the bottom of upstream/main..HEAD), step 5 (walmart_careers test_shared_documentation_uses_the_current_site_range now finds 40000-40026 in every doc it checks) and step 6 (no berkeley 40015 reference remains). Co-Authored-By: Claude Code <noreply@anthropic.com>
- .build-generated-seed: declares instance_seed/berkeley.db a build artifact (this site has no HF archive), so check_assets.sh and build.sh stop requiring one. - Dockerfile: generate the seed with the peers' exact shape — `rm -rf instance instance_seed && PYTHONHASHSEED=0 python seed_data.py && rm -rf instance` — instead of `python3 -c "from app import app"` plus a cp. - seed_data.py: __main__ writes instance_seed/berkeley.db itself (build_seed_database), so the artifact comes from the documented command rather than an import side effect. - seed_data.py: freeze the four benchmark password hashes (bcrypt salts are random, so set_password() gave every build a different DB) and pin created_at=datetime(2026, 5, 12) (the column default read the wall clock). - app.py: guarded bootstrap_site() as in walmart_careers (WEBSYN_SKIP_BOOTSTRAP=1 suppresses it), still seeding on import so site_runner.py and the generator work unchanged. - app.py: drop index=True from users.email / users.username. SQLAlchemy emits a table's named indexes in set-iteration order, so the two indexes were assigned different root pages from run to run (observed: pages 3/4 swapping) and the file was reproducible only by luck. unique=True keeps SQLite's implicit index, created in declaration order. Verification: step 2 (md5 3001bcf4bcec169f4192c08609160ab6 across 5 scratch builds — 4x PYTHONHASHSEED=0, 1x PYTHONHASHSEED=1 — and 3 further in-repo runs); step 3 bootstrap half (empty instance/ seeds to the same md5; populated instance/ is a no-op; skip flag suppresses). Co-Authored-By: Claude Code <noreply@anthropic.com>
sites/osu/app.py:27 pattern: BENCHMARK_NOW = datetime(2026, 5, 12), the same instant seed_data.py pins the seeded event calendar to. - app.py: BENCHMARK_NOW replaces every datetime.utcnow() on a request path — inject_globals, index()'s upcoming-events filter, events() (including its today_start/today_end window) and event_detail()'s related-events filter. Before this the seeded calendar (last event 2026-07-16) was already behind the wall clock: / rendered an empty Upcoming Events section and /events answered "Showing 0 of 0 events", which made the event tasks unsolvable. - app.py: the created_at / published_date column defaults now call a frozen utcnow() helper returning BENCHMARK_NOW, so no request path (register, bookmark_add) or seed path can stamp the wall clock into a row either. Verified: seed md5 unchanged at 3001bcf4bcec169f4192c08609160ab6 (no seeded row ever relied on the column default). Verification: step 3 (all six endpoints 200; instance md5 still equals the seed md5 after the request set; / renders 4 upcoming-event cards; /events default "Showing 20 of 52 events"; /events?category=Lecture "Showing 15 of 15"). Co-Authored-By: Claude Code <noreply@anthropic.com>
…rchive
A site whose seed is generated by the Dockerfile can legitimately have no archive at
all, not merely a media-only one — berkeley is the first such site. fetch_assets.sh was
the only release script still requiring an archive for every site directory (check_assets.sh,
build.sh and extract_assets.sh all honour .build-generated-seed), so a fresh clone died
before extracting anything:
[fetch] scope: 27 registered site(s)
fetch_assets: revision ad6f424f72cada9e6f5c09a58093d0ceeab9c52b has no archive for: berkeley
exit 1
- single-site branch: after the download attempt, a missing archive on a
.build-generated-seed site reports "nothing to fetch" and exits 0
- all-sites branch: such sites are skipped instead of added to `missing`
- an archive that does exist (fedex, webmd_doctor, ...) is still downloaded and validated;
nothing changes for media-carrying build-generated sites
Verified: `./scripts/fetch_assets.sh berkeley` -> exit 0, no "expected archive";
`./scripts/fetch_assets.sh` -> exit 0, 26 sites extracted; `./scripts/check_assets.sh` -> exit 0.
Verification: step 4 (berkeley-only fetch exits 0 without the "expected archive" error, and
the full fetch + check_assets.sh succeed).
Co-Authored-By: Claude Code <noreply@anthropic.com>
GET /news/<slug> bumped news_articles.view_count and committed, so every article visit wrote the DB. That broke two invariants: a read-only benchmark task could never have an after-state equal to its initial snapshot, and instance/ diverged from instance_seed as soon as an agent opened one article. The route is now a pure read. The column is kept and is still rendered as "N views" on /news and on the article page from the frozen seed values; nothing orders or filters by it. Verified: fresh-seed boot on :45011, three article detail pages all 200, md5(instance/berkeley.db) == md5(instance_seed/berkeley.db) == 3001bcf4..., news_articles rows (incl. view_count) identical before and after. Co-Authored-By: Claude Code <noreply@anthropic.com>
sites/berkeley/verify/ now carries the grading contract for the 22 accepted rows: the 20 kept ids (including the re-anchored --27) plus the new stateful --30/--31. - verify_lib.py: adapted from sites/webmd_doctor (newest merged), SITE=berkeley, no cross-site import, zero LLM calls on any verdict path. The snapshot contract is a pinned schema hash + nine-table set + seed counts + row-level catalog fingerprint; snapshots resolve from <run_dir>/initial.db|after.db, --initial_db/--after_db, or docker cp from $WH_CONTAINER, and missing or invalid input exits 1 with a structured infra_error instead of a traceback. - ground_truth.py: every target is re-derived from the run's initial.db the way the app renders it (the BENCHMARK_NOW event filter, PER_PAGE, the app's ORDER BY clauses, the unordered LIMIT 3 related-centres query). The two source-rendered rows (--11, --17) parse tracked files and fail closed if the literals move. No answer constant is frozen anywhere. - 22 verifiers: any-step URL gates plus the final action's declared target as an alternative satisfier; every multi-hop task gates each hop; set-valued answers use derived accepted sets and minimum counts; matching is negation-aware; --30/--31 bind to an exact bookmark row delta, and --31 also pins the surviving row id (2) as proof that both inserts and the removal happened. - verify/tests: 516 tests. Stdlib-only fixture DBs reproduce the pinned fingerprint; trajectories follow the agent_demo/agent.py signature. Per task: genuine PASS, no-op, wrong task id, another task's trajectory, shortcuts, 1-2 wrong answers, alternative phrasings, negated answers, truncated run, corrupt and 1x1 PNGs, missing after.db, catalog/schema drift, collateral writes and state mismatch, plus the About/Admissions source-fact assertions. Run: .venv/bin/python -m pytest sites/berkeley/verify/tests -q -> 516 passed Co-Authored-By: Claude Code <noreply@anthropic.com>
- tasks.jsonl: 22 rows — the 20 kept ids (with --27 re-anchored onto the two exact programme durations) plus the new stateful --30/--31. The 19 unchanged rows keep the contributor's exact ques text; every row gains verifier_path and a rules-only judge_rubric (shared scoring preamble + per-task checkpoints). "web" stays http://localhost:40026/. - verify/TASK_REVIEW.md: all 32 rows (30 contributor + 2 reviewer) with ACCEPT / DROP / ADDED and the workflow each accepted row must follow, plus the corrections found while deriving the targets (frozen-clock Lecture count, the AI-family allowlist for --7, BIDS' four focus areas and its rendered related-centre set, and the article-view write removal). - verify/README.md: the contract per task, the snapshot rules, the matcher semantics, and how to run the verifiers and their tests. - verify/tests/test_tasks_contract.py: validates the file (22 rows, seven keys, existing verifier paths, unique rubrics) and re-derives every target to prove no rubric carries a ground-truth value — with a self-test that the leak scan actually fires on a planted answer. The image already excludes verify/tests/ via the existing .dockerignore pattern (sites/*/verify/tests/), the same rule the merged peers rely on; no change was needed there. Run: .venv/bin/python -m pytest sites/berkeley/verify/tests -q -> 524 passed Co-Authored-By: Claude Code <noreply@anthropic.com>
verify/tests/run_matrix.py boots the mirror from a fresh seed per cell on an alt port, drives a scripted Playwright workflow, snapshots the live database and writes an agent_demo/agent.py-shaped run directory (trajectory.json with url-before-action steps and one input step per filled field, screenshots/, initial.db, after.db). It then grades every cell through `uv run python agent_demo/eval_judge.py --run_dir <dir> --verifier True` and compares the verdict with the cell's expectation. Cells per task: pass (genuine walk), no_op, shortcut (catalog-wide search with the correct answer), wrong_answer, collateral_write (one row injected straight into the live DB after the walk) and, for the two stateful rows, state_mismatch (the save skipped). Genuine answers are rendered from the derived target, so the harness carries no second copy of the ground truth. Artifacts land under sites/berkeley/scripts_dev/runs/matrix (gitignored, and self-ignored by a generated .gitignore in the output root). The matrix itself was NOT run in this window: that is the next one. What the test suite runs is the browser-free replay contract (verify/tests/test_run_matrix_contract.py), which replays each workflow as a synthetic trajectory and asserts the genuine run passes every verifier, every wrong answer is rejected and the injected collateral write fails. Run: .venv/bin/python -m pytest sites/berkeley/verify/tests -q -> 528 passed Co-Authored-By: Claude Code <noreply@anthropic.com>
- sites/berkeley/README.md: scope and routes, seed generation (build-generated from tracked source, byte-reproducible, md5 recorded), the frozen BENCHMARK_NOW clock, why the mirror ships no imagery, the seeded row counts, the demo accounts, and a pointer to the grading contract. - sites/berkeley/tests/test_integration.py (walmart_careers pattern): the registry is derived from control_server.SITES (cross-checked against websyn_start.sh) and berkeley is asserted at index 26 / port 40026; the Dockerfile exposes the current range and builds the seed from source; the .build-generated-seed marker is honoured by fetch_assets.sh; the seed's md5 is asserted when the build-generated DB is present; tasks.jsonl has 22 rows on the registered port with existing verifier paths and no answer key; app.py keeps the frozen clock and its article route neither writes view_count nor commits; the five shared docs carry the current port range. Run: .venv/bin/python -m pytest sites/berkeley/tests -q -> 7 passed Co-Authored-By: Claude Code <noreply@anthropic.com>
…der contract The first matrix run exposed four writer defects; each is fixed here, and the fix was mutation-checked (the port guard was shown to fire on a stray server, and the 30/31 pass/state_mismatch cells now grade correctly): 1. grade() launched eval_judge.py from the repo root, whose .venv has neither openai nor simpleArgParser -> ModuleNotFoundError and no eval.json, so every cell would have been reported as a mismatch with pass=None. It now runs from agent_demo/, the invocation AGENTS.md documents. 2. The page was never navigated before the first recorded step, so step 0's URL was about:blank and the verifier's all_urls_match_local_origin gate failed the genuine run of every task. It now pre-navigates to the start URL exactly as agent.py's browser.navigate_to(start_url) does. 3. Both login workflows clicked "button[type=submit]", which matches the navbar search button first: the login never happened, /account bounced to /login and the bookmark step timed out. Now form[action='/login'] button[type=submit]. 4. boot() adopted any process already listening on the port. A leftover standalone server made cells grade against a foreign instance (the 30/31 pass cells failed with bookmarks_exact_delta). boot() now refuses that port. Also the state-mismatch cell (all writes skipped) clicked the bookmark-removal form that a fresh instance never renders; the skip rule now applies to any step marked skip_in_state_mismatch, and the removal click carries the mark. Co-Authored-By: Claude Code <noreply@anthropic.com>
…by "Prof."
The C2 mutation rows found that a contradictory answer passed three verifiers:
"The chair of EECS is not Prof. James Demmel, ...", "BIDS is not directed by
Prof. David Culler, ..." and "The Economics department is not chaired by Prof.
Ulrike Malmendier, ..." were all graded as affirmative. Cause: the clause
splitter treated the period in an honorific as a sentence end, so the negation
that precedes the name landed in a previous clause and _match_is_affirmative
never saw it. The plain forms without the title ("James Demmel is not the
chair of EECS.") were already rejected, which is why the unit suite missed it.
_match_is_affirmative now computes clause boundaries over a length-preserving
mask of honorific / degree abbreviations (Prof. Dr. Mr. Mrs. Ms. Miss Mx Rev.
Fr. Sr. Jr. and "Ph.D."), so the two fixes compose: the name matcher sees the
negation in front of it wherever the rendering carries a title.
Mutation-checked: with the mask disabled the four new tests fail and only
those (4 failed, 103 passed); with it enabled the suite is 531 passed. Rows
re-run with the full variant + mutation sets; the matrix is re-run in the
same phase.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Rewrite judge_rubric for all 22 accepted tasks (rubric-only diff; every other key unchanged, no row added or removed). Rubrics open with the scoring rules (step list authoritative; a checkpoint is true unless positively contradicted; a detail page is not satisfied by a listing page; every step must be on the local mirror origin; the verifier owns exact values and DB state; an empty answer forces failure) and close with an explicit origin checkpoint. No rubric contains a derived answer value; the task-contract leak scan is clean. Rebuilt from D3/D4 evidence: 16/21 -> 19/21 verifier-judge agreement on the shim arm; D5 (tasks 24, 27) run after this rewrite leaves one reproduced divergence (verify_27.py route gate, NOT-FIXED). Co-Authored-By: Claude Code <noreply@anthropic.com>
… bounds, FK
Appendix A §6 merge blockers found by the Phase E executable probes:
- SECRET_KEY was the committed literal 'berkeley-mirror-secret-key-2024', so a
cookie signed with it read /account without a password (probe: 200). It now
comes from BERKELEY_SECRET_KEY or a per-process random key, and a wrong-key
cookie is bounced to /login.
- GET/HEAD /logout returned 302 and cleared the session (prefetcher-reachable);
the route is POST-only now (GET/HEAD -> 405) and the site chrome's Sign Out
control is a CSRF-protected form instead of a link.
- A session cookie with a non-numeric _user_id raised int() into a 500;
load_user now fails closed (anonymous).
- Huge query/form integers overflowed SQLite ('/news?page=9'*20 -> 500);
bounded_int/page_arg and the event-id range check answer 200/404 instead.
- MAX_CONTENT_LENGTH (256 KB) and SESSION_COOKIE_HTTPONLY/SAMESITE are set.
- Empty or invalid /bookmark/add submissions silently redirected; they now
answer 400 (closed item-type vocabulary) or 404 (missing row), and
/bookmark/remove is owner-scoped (another user's id is a 404) with a bounded
id.
- '?next=https://evil.example/' bounced /login and /bookmark/add off-mirror;
safe_next keeps redirect targets same-origin.
- SQLite PRAGMA foreign_keys was off (orphan bookmark rows accepted); an
Engine connect listener turns enforcement on.
- Duplicate registration now rolls back on IntegrityError.
Accessibility chrome on every page (the maintainers' b87db8f batch): skip link,
mirror/synthetic-data notice, focus-visible outlines, role=alert on flash
messages. sites/berkeley/tests/test_app_robustness.py pins all of it and was
mutation-checked (mutations archived under scripts_dev/logs/phase_e/).
Co-Authored-By: Claude Code <noreply@anthropic.com>
…ry route The Phase E leak sweep (new sites/berkeley/tests/test_answer_leaks.py, adapted from the maintainers' webmd_doctor sweep) found that listing cards rendered the exact fields the tasks must discover on a detail page: - /research, the home page and the /search results rendered "Director: X", "Founded: Y" and the first focus-area tags for every centre, so tasks 10, 23, 30 and 31 had their answers visible before /research/<slug>. - related-centre cards on another centre's page did the same for neighbouring centres. - /departments cards rendered "Chair: X" and the location, so tasks 13 and 24 had their answers visible before /departments/<slug>. The listing cards now carry name/college/description only; the detail pages still render director, founding year, focus areas, chair and location (pinned by positive controls in the test). The test enumerates each task's answer facts from verify/ground_truth.py, exempts only the routes its verifier requires, and documents every shared-value co-occurrence with its reason; it was mutation-verified by re-adding each removed field (5/5 mutations fail). Co-Authored-By: Claude Code <noreply@anthropic.com>
…ng its target /programs?page=3 lists programmes 41-60; the Data Science MS is the 24th row, i.e. on page 2 (the run then opened the detail page by URL, so the listing hop never showed the target). Replay fixtures are data-driven and unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com>
… parity Appendix A §4: - /search matched narrower field sets than the catalogue pages it duplicates: news content, event location/organizer and faculty title were reachable from the listing filters but not from the global search. All catalogue fields are now matched (superset direction only; no result set shrinks). - The related-centre / related-programme / colleague / department-programme / home-page research queries had no ORDER BY and fell back to rowid insert order; each now has a neutral key (name) and ground_truth.related_centres mirrors the same ORDER BY. /search results are ordered by the same neutral keys as their listings. - test_verify_23's fixture now names the first entry of the ORDER BY name list. Co-Authored-By: Claude Code <noreply@anthropic.com>
Appendix A §9 (4 widths x all routes, pixel-sampled contrast): - Contrast tokens: --gray-mid #6c757d -> #64696f (4.38 -> 5.17 on the off-white card ground), --light-blue #3B7EA1 -> #2E6C8B (4.48 -> 5.78 on white), .badge-green #28a745 -> #1E7E34, gold-button hover #C4820A -> #E0A50C (blue text 4.01 -> 5.85), the 404 numeral gold -> --dark-gold (1.78 -> 3.56), the hero gradient's light stop darkened and its 0.9 opacity wash removed, the card-image label made solid white, and the event/news category palettes darkened (Sports #8A5A07, Career #1E7E34, Health #0F7A5A, Arts #5A3383, Virtual #55595E; the in-badge white 0.2 wash -> black 0.3). Measured by glyph-anchored pixel sampling (scripts_dev/phase_e_contrast.py): 254 failing text elements before, 0 of 2725 after at 1440, 0 at 768/390/320. - Layout: fixed 4/2-column and 3fr/2fr grids became responsive helpers that collapse at <=768px (about, academics, admissions, department/program detail, faculty profile, research, account, home hero); long unbreakable strings get `min-width: 0` / `overflow-wrap: anywhere`; the filter-bar controls shrink; the faculty profile header wraps. 320px horizontal scroll is gone. - Heading order: footer and card headings no longer skip levels, listing pages carry an sr-only h2, detail-page section headings were re-levelled; the heading-order check is 0 jumps over 423 routes. - run_matrix/site docs: the related-centres query is ORDER BY name now (mirrored in ground_truth and TASK_REVIEW). Co-Authored-By: Claude Code <noreply@anthropic.com>
…op bar The featured-article badge is position:absolute with no positioned ancestor: .card-img was static, so the badge anchored to the page and painted over the top-bar Sign In / Create Account links (measured overlaps at 768/390/320 px). position: relative on .card-img keeps it inside the card image; verified in Chromium at 4 widths (in_card 4/4, overlapped 0). Co-Authored-By: Claude Code <noreply@anthropic.com>
…ADME claim seed_data.py now refuses to build instance_seed/berkeley.db when the app engine's URI is not the canonical BASE_DIR path, before deleting or creating anything (the maintainers' §8 wound: a redirected URI silently writes the seed elsewhere while the copy still reads DB_PATH). Baseline reproduces md5 3001bcf4...; the scratch-copy mutation with a redirected URI refuses and writes nothing. verify/README.md claimed 19/24/25/30/31 'gate each hop in order'; only 24/30/31 call check_paths_in_order — 19/25 gate both hops independently. Wording fixed. Co-Authored-By: Claude Code <noreply@anthropic.com>
…he navy band Measured: the #0b5cab focus outline is 6.7:1 on white but only 1.92:1 where it crosses the navy top bar / header (skip link, header search input, search button). A lone ring cannot contrast with both white and navy, so the ring is now two-tone: the blue outline plus a white halo (12.86:1 on navy, 6.7:1 blue on white, blue on gold 3.77:1). Verified by pixel measurement of the painted ring around each of the first six tab stops. Co-Authored-By: Claude Code <noreply@anthropic.com>
…es (§0.4) verify_27.py gated "visited_program_search" on the programme catalogue only (/programs?q~'master of engineering' OR /programs?degree~'MEng'), while the ques says "Search the Berkeley site for 'Master of Engineering'". Both task-27 runs with real answers (27shim, 27shimd5b) searched /search?q=Master+of+Engineering, opened both detail pages and gave the right department and both durations; every other check passed and only this gate failed (observed_urls=[]). PIPELINE §0.4 conditions: (a) required to accept a genuinely correct run — proved check by check; (b) the loosening keeps the anti-shortcut property — re-grading task 27's full variant and mutation sets gives the identical verdict and identical first failing check for all 13 rows (pass PASS; no_op final_answer_nonempty; shortcut visited_program_search; wrong_answer answer_has_meng_duration; collateral_write read_only_bookmarks_unchanged; tiny_png screenshots_decode; catalog_search visited_program_search; truncated trajectory_completed; other_task trajectory_task_matches; missing_after_db database_unavailable; read_only_write read_only_bookmarks_unchanged; negated_answer answer_has_department; answer_only visited_program_search); (c) logged in DECISIONS.md. The gate now also accepts /search?q~'master of engineering'. The query must name the MEng term, so a catalog-wide search still fails. Both detail-page gates and every answer check are unchanged and remain the binding anchors. 3 regression tests added (site-search accepted; catalog-wide search still rejected; search alone without both detail pages still rejected). Suites: verify/tests 534 passed; site tests 27 passed; no-op container matrix 22 rows, 0 not FAILing, 0 infra errors. Co-Authored-By: Claude Code <noreply@anthropic.com>
Two test-file docstrings pointed readers at paths that are not part of this branch: review-reports/berkeley/CHECKLIST_REPORT.md §6 and scripts_dev/logs/phase_e/ (test_app_robustness.py), and scripts_dev/REVIEW_STATUS.md §7.2 (test_integration.py). The prose keeps its meaning — probes failed on the pre-fix tree, each detector was mutation-checked — without naming artifacts that do not ship. No functional change. verify/tests/run_matrix.py was deliberately left alone: its four `scripts_dev`/`runs/` occurrences are the harness's own scratch --out default (a path it creates on demand and self-ignores), not references to review evidence. Co-Authored-By: Claude Code <noreply@anthropic.com>
… into the UC Berkeley branch Registry: append `berkeley` after `kaggle`, so Healthline keeps index 26 / port 40026 and Kaggle keeps index 27 / port 40027; UC Berkeley moves to index 28 / port 40028 (29 sites, 40000-40028). Conflict resolutions (append rule): - `websyn_start.sh`, `control_server.py`: keep all of upstream's 28 entries in order and append `berkeley` last (both registries identical, 29 entries). - `Dockerfile`: keep every upstream site block (Healthline's migrate + prune step included) and berkeley's build-generated seed block; header 28 -> 29 sites; `EXPOSE 8101 40000-40027` -> `40000-40028`. - `README.md`, `AGENTS.md`, `CONTRIBUTING.md`, `CLAUDE.md`, `agent_demo/README.md`, `.claude/skills/*`: take upstream's text, then 29 sites, port range 40000-40028, alt ports 41000-41028, and `UC Berkeley` appended to README's mirror list. - `scripts/fetch_assets.sh`, `scripts/check_assets.sh`: take upstream's registry-scoped implementations and keep berkeley's `.build-generated-seed` exemption on top. Follow-on work required by the port move: - `sites/berkeley/tasks.jsonl` (22 `web` rows), `app.py` PORT default, `README.md`, `verify/TASK_REVIEW.md`, `verify/verify_lib.py` comment, `tests/test_integration.py` (SITE_INDEX 28 / SITE_PORT 40028) and `verify/tests/test_tasks_contract.py`: port 40028. - `scripts/check_site_registry.py` (upstream's new gate) reports 29 sites consistent across both registries, `Dockerfile EXPOSE` and every `tasks.jsonl` port. Verified after the merge: registry gate green; berkeley site 27, berkeley verify 534, healthline 27; walmart_careers 50 passed and rotten_tomatoes 55 passed (+4129 subtests) each with one expected red that is pre-existing on upstream/main and reproduced there from a pristine `git archive` tree (VERIFICATION.md §re-slot); kaggle ships no pytest suite. Container rebuilt and re-run on 44000-44028: 29/29 sites 200, /health 29 alive+ready, `POST /reset/berkeley` byte-identical (`f2f0187c...`) before and after `docker restart`, no-op matrix 22/22 FAIL with 0 infra errors. Co-Authored-By: Claude Code <noreply@anthropic.com>
evanz37
marked this pull request as ready for review
September 13, 2026 15:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review + deterministic grading contract for
sites/berkeleyReviewer PR for #11. Branch
review/pr-11-berkeley-delivery. Author commitfa70f03is unchanged; all commits below are the reviewer's, on top of it.What this PR carries
Integration —
c0a3fderegister berkeley as site 26 / port40026;8e5bb7fmergeupstream/main(Healthline #105 took index 26 /40026, Kaggle #106 took index 27 /40027) and re-slot berkeley to site 28 / port40028(29 sites,40000-40028); docs sweep.Seed + clock
2f0c6c4build-generated, byte-reproducible seed (3001bcf4…underPYTHONHASHSEED=0and=1).c1cdcb8freeze the benchmark clock (BENCHMARK_NOW = 2026-05-12) so date filters are deterministic.1929b84article detail no longer writesview_count(a read path was writing the DB).Verifiers + tasks
1d17acbadd 22 deterministic task verifiers +verify_lib(targets re-derived from the snapshot).0ff43bftask selection andverifier_path/judge_rubricbackfill; 32 candidate rows triaged to 22.67ee9a5add the run-signature writer and the browser-driven matrix harness.eb19a2bsite README and integration tests.50813e7matrix writer drives the real recorder contract (grade fromagent_demo/, pre-navigate, port guard).67cff3dnegation before a titled name is no longer hidden by "Prof." (found by the mutation rows).5aca1derubric polish — all 22judge_rubricvalues rewritten (rubric-only diff).Robustness + UI
66a0c8dapp robustness: POST-only logout, per-process secret key, bounded inputs, FK enforcement.077f37fanswer facts no longer render before their discovery route (+ the answer-leak sweep).0603a3dtask 16 workflow pages to the catalogue page holding its target.678f703neutralORDER BYon list queries and/searchfield-set parity.57cb484WCAG AA contrast, 320px layout, heading order.455e5f7the Featured badge escapes its card instead of overlapping the top bar.d5d42b9seed generator self-locks its engine URI; scope a README claim.ae2bdbdtwo-tone focus ring so the indicator clears 3:1 on the navy band.a38df80task 27 search gate accepts the route the ques names (resolved under the review's §0.4).f237364drop references to untracked review artifacts from two test docstrings.Scripts exemption — droppable —
86ba06efetch_assets.shskips build-generated sites with no archive. Touchesscripts/, not the site; maintainers may drop this commit if they prefer to own the exemption.Grading contract
verify/verify_N.py) — no LLM, no key.initial_db, never frozen; ambiguity fails closed.judge_rubriccarries scoring rules only, never an answer value.Validation
state_mismatchfor 30/31; every pass PASSes, every variant FAILs on its intended first failing check.after_db, read-only write, negated answer, answer-only)./health27 alive+ready;POST /reset/berkeleybyte-identical (f2f0187c…==instance_seed), still identical after a real authenticated write + reset and afterdocker restart;reset-all27/27 in 2.6 s; 419-route smoke with the DB byte-identical afterwards; no-op matrix 22/22 FAIL, 0 errors.agent_demo/agent.py22 runs → 1 PASS / 21 FAIL, 0 verifier↔judge divergences. The endpoint returnsdonewith a top-leveltextwhile the shipped parser readsparams.text, so most stock runs recorded empty answers; a reviewer-side parser shim (recorder untouched, AST-verified byte-compatible) recovered 21 gradeable runs → verifier 10 PASS / 11 FAIL, judge agreement 20/21 (24/25 cumulative after re-runs).Required before merge
agent_demo/eval_judge.pyraisesJSONDecodeErrorbefore reaching a verifier and can leave a staleeval.json— out of scope, the site's verifier fails closed on its own CLI; (2) four filtered programme listings render exactly one row, so those filters have no same-kind distractors — needs catalogue growth; (3) the mirror ships no imagery — an author decision, not a defect; (4) the judge can false-PASS an off-mirror run — harness limit, caught by the verifier every time.40026went to Healthline (Review: Add Healthline mirror + task verifiers (site by @JeremyJC67, verifiers by reviewer) (#59) #105) and index 27 /40027to Kaggle (Review: Add Kaggle mirror site (#54) #106); this branch is merged withupstream/mainand registers berkeley at index 28 / port40028(29 sites,40000-40028), so the earlier collision with Review: validate NBA mirror and task grading (#30) #101/Review: Add Versus mirror + task verifiers (site by @Sun-sunshine06, verifiers by reviewer) (#41) #111/Review: validate BabyCenter mirror and task grading (#42) #112 is gone.Not tested