Skip to content

Review: Ohio State (osu) mirror + task verifiers (site by @richard-peng-xia, verifiers by reviewer) - #3

Merged
cursor[bot] merged 2 commits into
mainfrom
cursor/review-osu-12-3bf2
Sep 18, 2026
Merged

cursor[bot] merged 2 commits into
mainfrom
cursor/review-osu-12-3bf2

Conversation

@Lxr-max

@Lxr-max Lxr-max commented Sep 14, 2026

Copy link
Copy Markdown
Owner

This PR is the review vehicle for aiming-lab#12 (Ohio State University mirror, sites/osu, original site by @richard-peng-xia). It does not merge aiming-lab#12 itself.

Lineage

Item Detail
Contributor PR aiming-lab#12richard-peng-xia:add-osu-mirror, commit 635808d (richard-peng-xia <richard.peng.xia@gmail.com>), port 40015 / 16-site era
Prior integration aiming-lab#63 already merged an OSU integration + first grading contract onto main (site index 20 / port 40020). That merge rewrote the original commit author to StephenQSstarThomas / later hardening by raibows.
This PR Independent Lxr-max reviewer pass on current main: keep the hardened site, upgrade the grading contract to current Review-PR practice, de-leak listing cards, re-anchor task 4.

Cherry-picking 635808d onto current main would collide with the already-landed sites/osu/ tree and regress aiming-lab#63 security/image/reset work, so this branch does not rewrite those commits. Original authorship is preserved by leaving aiming-lab#12's commit intact on the contributor branch and crediting @richard-peng-xia here; reviewer commits are Lxr-max <xuanrui.li@se24.qmul.ac.uk>.

Slot after rebase onto current main

  • Site slug: osu
  • Index: 20 (after ted)
  • Container port: 40020 (http://localhost:40020/)
  • Registry already in sync: websyn_start.sh, control_server.py, Dockerfile EXPOSE 8101 40000-40027 (28 sites)

Tasks and grading contract

  • 20 / 20 tasks accepted (Ohio State University--0--19)
  • 20 deterministic verifiers: sites/osu/verify/verify_0.pyverify_19.py plus verify_lib.py
  • tasks.jsonl has verifier_path + English judge_rubric (rules/checkpoints only — no answer key, no ground-truth names in the rubric)
  • Harness: sites/osu/verify/tests/ (excluded from the image by .dockerignore)

Original aiming-lab#12 questions were almost all knowledge-shortcuts / listing-only ("What conference does Ohio State athletics compete in?", "In what year was The Ohio State University founded?", …). They were previously re-anchored onto explicit on-site workflows. This review keeps that re-anchoring and additionally:

  • Re-anchors task 4 so the question no longer names $1.3 billion; the agent must read amount + date from the article after searching research expenditures.

Reviewer fixes on this branch

  • Strip coach / record / national titles from Athletics cards (need team detail).
  • Strip research director / founding year / focus tags from Research cards (need center detail).
  • Strip program credits / duration / deadline / GRE from Programs listing (need program detail). Task 9 still sees degree-type badges on the Engineering filter.
  • Strip department chair / location from Departments cards.
  • Strip news byline/date from News listing and search result dates.
  • Drop the redundant "Dean " prefix in seed college names so Academics renders Dean: Anil Makhija rather than Dean: Dean Anil Makhija.

Validation (what was actually run)

From this branch, not a full 28-site Docker image (no Docker daemon on the review host):

python3 -m py_compile sites/osu/app.py sites/osu/seed_data.py sites/osu/verify/verify_*.py
PYTHONHASHSEED=0 python3 sites/osu/migrate_seed.py
python3 sites/osu/verify/selfcheck.py
# 30 tests, all OK (8.6s)

Offline verifier matrix (scripted trajectories, no LLM):

  • genuine PASS: 20/20
  • no-op / empty answer FAIL: 20/20
  • answer-only knowledge-shortcut FAIL: 20/20
  • listing-without-detail shortcut FAIL (detail tasks)
  • wrong answer / swapped comparison / missing filter / DB mutation FAIL as expected

Live standalone Flask on :41020 (HF osu.tar.gz images from pin 64264d0… + build-generated seed):

  • 20 representative routes HTTP 200
  • login alice@osu.edu / test1234 → account 200; GET /logout 405; POST logout without CSRF 400
  • Playwright Chromium: home → Academics (Fisher dean on that page), Athletics listing (no coach/record) → football detail (coach + record + venue), Engineering programs filter, search cancer research → article byline, login + Save Program persists on account, 390px home has no page-level overflow

Assets

  • Images: osu.tar.gz is on the current HF pin (ChilleD/WebHarbor@64264d065cdb0b7755ee99dab356be5733d9ddef); ./scripts/fetch_assets.sh osu installed 19 official WebP files. No .assets-revision bump.
  • Seed DB: still build-generated (sites/osu/.build-generated-seed); not shipped in the tarball. Byte-deterministic rebuild tested twice in the unittest.

Blockers

Commits on this branch

Commit Author Content
3d22182 Lxr-max listing de-leaks + dean prefix
858de51 Lxr-max verifiers, rubrics, verify/tests harness, task 4 re-anchor
Open in Web Open in Cursor 

Lxr-max and others added 2 commits September 14, 2026 01:57
Keep coach, records, titles, research directors, program credits/deadlines,
department chairs, and news bylines on detail pages so listing/search
shortcuts cannot complete the benchmark tasks. Drop the redundant "Dean"
prefix from college names so Academics renders "Dean: <name>".

Co-authored-by: Lxr-max <Lxr-max@users.noreply.github.com>
Independent reviewer grading contract for PR aiming-lab#12: one verify_N.py per
accepted task, shared verify_lib, verifier_path + rule-only judge_rubric
on all 20 rows, and a verify/tests harness (no-op, shortcut, wrong-answer,
listing-without-detail, swapped comparison, missing filter, DB mutation).
Re-anchor task 4 so the expenditure amount is not named in the question.

Co-authored-by: Lxr-max <Lxr-max@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant