Add CarMax mirror + task verifiers (site by @Violet24K, verifiers by reviewer) - #57
Merged
Merged
Conversation
…com. - 13 SQLAlchemy models (User / Store / Vehicle / SavedVehicle / Comparison + ComparisonItem / Reservation / TestDrive / Appraisal / FinancePreQual / Order / Review / Article) - 59 routes covering search / browse / detail / research / compare / saved / sell-my-car / pre-qual / reserve / test-drive / checkout / account / articles / FAQ / MaxCare / stores / auth - Token-overlap scored search with multi-field weighting - 141 deterministically-seeded vehicles across 31 templates - 12 real CarMax store locations - 5 benchmark users with pre-populated saved/reservation/test-drive/ appraisal/order data - 20 WebVoyager tasks in tasks.jsonl (6 Easy / 9 Medium / 5 Hard, including 2 disambiguation tasks) - Idempotent seed at function level; byte-identical reset verified
Per-task deterministic verifiers (verify_<N>.py) + shared verify_lib.py under sites/carmax/verify/, plus verifier_path + judge_rubric on each task in tasks.jsonl. Each verifier checks navigation (anti knowledge-shortcut), DB after-state for stateful tasks, and the answer against frozen ground truth. Validated against the react agent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 3, 2026
6 tasks
jackjin1997
added a commit
to jackjin1997/WebHarbor
that referenced
this pull request
Sep 12, 2026
Refresh review-reports/PR-33-IMDB.md to the fixed candidate (code 1c8a1eb, HF PR aiming-lab#57 revision f9ddfd25): drop the superseded HF pin and Owner-pending text, record the formal 20-task blind review and reconciliation, the fixed-candidate engineering and guided impact regression, Owner visual acceptance, known limitations and the maintainer handoff order. Add public summaries and current-candidate screenshots under review-reports/assets/pr33-imdb/ and mark the historical probe summaries as superseded. Docs-only: no application, task, verifier, seed or asset bytes change.
Raibows
added a commit
to jackjin1997/WebHarbor
that referenced
this pull request
Sep 17, 2026
…ex 30 / 40030 after aiming-lab#116 took index 29 Conflict resolution against current main (aiming-lab#116 merged: nvidia index 28, UC Berkeley index 29, 30 sites): - websyn_start.sh, control_server.py: keep main's 30-site order and append imdb as index 30 (container port 40030). - Dockerfile: keep main's build steps unchanged (the PR adds no build step for imdb, whose seed ships in the HF archive) and set the canonical header/EXPOSE lines: "31 Flask mirror sites" / "EXPOSE 8101 40000-40030". The merged file differs from main's by exactly those two lines. - sites/imdb/tasks.jsonl: 20 web URLs 40024 -> 40030. No task semantics changed (ques/judge_rubric/verifier_path untouched); sites/imdb/app.py keeps its PORT-env default of 5000, so no other file in the site referenced 40024. - nvidia (40028) and berkeley (40029) task URLs are unchanged from main. - .assets-revision: kept as the PR's value (revision: f9ddfd2596229f2610418d57fc88c3051e1056bb, the head commit of HF dataset PR aiming-lab#57). This pin is NOT usable after this merge: refs/pr/57 carries 26 archives and has no archive for fedex, webmd_doctor, healthline, kaggle, nvidia or berkeley, and c32018ca (the pin main uses) has no imdb.tar.gz. The pin will be replaced with the merged HF dataset commit once HF PR aiming-lab#57 is merged; this commit does not change it. - Documentation (README.md, AGENTS.md, CLAUDE.md, CONTRIBUTING.md) took main's side here; the 31-site / 40000-40030 statements follow in the next commit.
QianhuiWu
added a commit
that referenced
this pull request
Sep 17, 2026
Integrate CarMax PR #57 with reviewed fixes and harder tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the CarMax mirror + its verifiers into
mainas one PR, per the maintainer'srequest to merge via the main repo rather than the cross-fork PR #24.
Authorship is preserved — both contributors stay credited. This branch is Zihao Li's
CarMax site commits (from PR #24) with the reviewer's verifier commit on top; each commit
keeps its original author (site = @Violet24K, verifiers = the reviewer). Please merge with a
regular merge (not squash) so both signatures are kept.
Contents:
sites/carmax/site (Flask app, templates, seed, 20 tasks) — @Violet24Ksites/carmax/verify/verify_<0..19>.py+verify_lib.pyandverifier_path/judge_rubricin
tasks.jsonl— reviewer deliverable (deterministic verifier per task: navigation +DB after-state + answer; validated against
agent_demo/agent.py)Known conflict (please resolve on merge):
mainalready merged Merriam-Webster atindex 15 / port 40015, and CarMax was authored at the same slot. On merge, register CarMax as
the next site — index 16 / port 40016 — in
websyn_start.sh,control_server.py,Dockerfile(EXPOSE …-40016), and bump thewebport insites/carmax/tasks.jsonl(40015 → 40016). The verifiers are port-agnostic (they match URL paths, not the host port).
Supersedes the cross-fork PR #24 and the verifier-only PR #56.