Conversation
|
Verification date: 2026-06-02 Site name: Amtrak Upstream URLs: Contribution type: New Website Contributor: XuanRui LI / Lxr-max Safety note: deterministic demo travel and booking data only; no real payment, no real booking, no live Amtrak calls. Implementation summaryThis PR adds a self-contained local Amtrak mirror built with Flask, SQLAlchemy, Jinja2, and SQLite. The site covers route browsing, station discovery, one-way / round-trip / multi-city search, schedule-style results, fare comparison, sleeper room selection, passenger details, mock checkout, booking confirmation, trip lookup, trip management, and rewards/account flows. The implementation uses deterministic synthetic data only and does not make external runtime calls. A small follow-up commit polishes a few remaining user-visible checkout / refund labels so the mirror reads like a finished demo instead of a template. Key user flows
Seed row counts
Benchmark users
Task count and categories
Selected WebSyn port
Key routes
Verification summarySyntax and task validation
Docker build and run
Runtime smoke testsPublic route checks returned
Seed-backed flow checks passed for:
Reset and MD5 result
Screenshot pathsLocal review screenshots were refreshed on 2026-06-02 at:
Included captures:
HF asset statusHF asset packaging and upstream PR creation have now succeeded.
Per the repo docs, Known limitations
Manual reviewer notes
|
|
Taking ownership of the review for this PR (Track B). Plan: build and drive the mirror per the |
…iming-lab#93 aiming-lab#39 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
Review —
|
| Check | Result |
|---|---|
Rebase onto main |
Done in the review PR. Conflicts in 8 files; upstream versions taken for .claude/skills/*, .gitignore, AGENTS.md, CONTRIBUTING.md, README.md (the PR's edits were stale 15→16-site count sweeps), amtrak re-appended as the last SITES entry in websyn_start.sh / control_server.py, Dockerfile bumped to EXPOSE 8101 40000-40024 / "25 Flask mirror sites". Docs + registry tests swept to 25 sites in 4f18dce. |
| Syntax | python3 -m py_compile sites/amtrak/app.py seed_data.py _health.py → ok |
| Idempotent seeding | seed_database() gated on Route.query.count() > 0; seed_benchmark_users() gated on the 4 benchmark emails; the SVG ensure_asset() writes live inside the gated function (0 files touched at boot). Verified: md5 identical after 1st boot, 2nd boot. |
| Site isolation | grep -rn "from sites\.|import sites" sites/amtrak → none |
| Registration / port | websyn_start.sh and control_server.py agree, index 24 → 40024; tasks.jsonl web set to http://localhost:40024/ (the PR had 40015, which is Merriam-Webster on main). Port provisional until merge. |
tasks.jsonl |
18 rows, one JSON per line, ids Amtrak--0..17; contributor rows carried exactly web_name, id, ques, web, upstream_url; now also verifier_path, judge_rubric, no answer key. |
| Asset archive | scripts/validate_asset_archive.py amtrak.tar.gz amtrak → "validated 308 managed members"; no ._* / .DS_Store; contents = instance_seed/amtrak.db (589,824 B) + 293 generated SVGs under static/images/{alerts,deals,destinations,rooms,routes,stations,trains} |
| Seed DB | md5 cb4b6a5939b663017d9774f2d3a86884 — matches the hash the author reported. Counts match the PR body (4 users, 62 stations, 18 routes, 252 trips, 1120 segments, 1008 fare options, 336 rooms, 60 bookings, 80 tickets, 16 alerts, 18 deals, 30 help articles). |
| Boot | GET / → 200, /_health → {"db": true, "ok": true, "site": "amtrak"} |
| Byte-identical after boot | md5(instance) == md5(instance_seed) after boot #1 and after boot #2 (cb4b6a59…) |
| Byte-identical after read-only browsing | FAIL — after the 44-page read-only sweep md5 = be90a3ce…, 40 rows in search_logs: /search?q=, /help?q= and /booking/results each commit a SearchLog row (app.py:1116-1126, called at 1373 / 1395 / 1468). /reset still restores the seed (it copies instance_seed/), so the reset invariant holds, but every agent step dirties the DB → Required fix 3. |
git diff --check, bash -n websyn_start.sh |
ok / ok |
| HF revision | BLOCKER — .assets-revision is revision: main and the PR points at HF discussion #30, but refs/pr/30 contains only the 15 base tarballs (no amtrak.tar.gz). The tarball exists only on the fork commit lxr-max/WebHarbor@eac108f (116,525 B). See §5.1. |
Docker (from review/pr-39-amtrak) |
ok — amtrak.tar.gz from lxr-max/WebHarbor@eac108f validated (308 members) and extracted by hand because no ChilleD/WebHarbor revision contains it; docker build ok (4.70 GB); alt ports 44000-44024: 25/25 return 200; /health ok, 25/25 alive+ready, amtrak on internal 40024; POST /reset/amtrak → md5 cb4b6a59… == seed; GET+POST /login then reset → still cb4b6a59…; docker restart → still cb4b6a59…; POST /reset-all ok, 25/25 ready in 1.64 s; verifier no-op matrix in the container (--container, DBs via docker cp): 18/18 FAIL, 0 errors; registry tests 4 passed; sites/amtrak offline suite 13 passed |
2. Visual fidelity — PARTIAL PASS
- Layout: a coherent Amtrak-style shell (navy/teal header, primary + tools nav, alert strip, booking hero widget with trip type / stations / dates / fare class / sort, card grids, footer). It reads as a travel-booking site, not as a pixel mirror of amtrak.com. (
top_ro_home.png) - Images: all 293 images are generated SVG art (route heroes, station cards, city cards, deal cards, alert badges, sleeper rooms) — the author discloses this ("deterministic local generated assets rather than copied Amtrak-owned media"). No broken images anywhere (0
naturalWidth == 0across 44 pages), but the "real images" criterion is not met by design, and the route hero titles are cropped by the card crop ("la Express", "trak Cascades", "ifornia Zephyr" on the homepage featured cards;main.css:363-366object-fit: coveron a 1200×560 SVG whose title starts at x=72). - Homepage feature tiles "Live-style tools / Account flows / Safety note" (
index.html:20ff) render white text on a pale background — illegible. - Detail pages (routes ×3, stations ×4, destination, deals, alerts, help ×2, trip detail, confirmation) are populated and consistent; departure boards, stop timelines, amenity lists all come from the DB. (
top_ro_station_SBA.png,top_ro_service_alerts.png) - Responsive: 768 px — no overflow on home / results / station. 400 px — home and results ok, station page overflows (
scrollWidth 434 > 400, the departures table;station_detail.html:28.info-table-wrap). (top_resp_400_station.png) - 404 pages are bare Werkzeug "404 Not Found" (no site template).
- A "Benchmark users use password TestPass123!" pill sits on every page and the login page lists all four demo accounts — fine for a benchmark, but worth knowing.
3. Functional depth — PASS
Driven with Playwright (real Chromium), evidence in audit_readonly.json / audit_writes.json:
| Check | Result |
|---|---|
| Nav / links | all 215 internal links reachable from the audited pages return 200; 0 console/page errors other than the two deliberate 404 probes |
| Login | wrong password → flash "That demo email and password did not match."; alice.j@test.com / TestPass123! → /account, "Welcome back, Alice Jordan." |
| Register | new account → /account and can log back in; password mismatch → "Passwords must match."; duplicate email → "That demo account already exists." Nit: first/last name are optional — blank last name becomes "Traveler" (app.py:1866), so the header greets "Rev Traveler". |
| Search | global /search buckets routes / stations / destinations / deals / alerts / help; multi-word "Boston South Station" returns 156 (any-token scoring is loose: "denver boarding track" → 71). Help search: "checked baggage" → 4, "refund" → 1. |
| Booking funnel | results → select option (radio + "Choose this itinerary") → /booking/select-trip → fares (4 families with per-traveler price / availability / points) → /booking/passengers (empty names → "Please complete every synthetic passenger name field."; "Use saved profile" fills Alice) → review → checkout → confirmation code ZSLYNG; /account/trips goes 9 → 10 active trips; rewards 4786 → 4822 with a new "Booking ZSLYNG" activity row. Round-trip (outbound then leg=return), multi-city (3 legs, one radio per leg) and sleeper-room (/booking/rooms: Coach / Roomette +$316 / Bedroom +$592 / Family Bedroom +$788) funnels all work. |
| Trip lookup | ALGX87 + Alice's email or last name "Jordan" → /trip/ALGX87; wrong email → "No demo trip matched"; another user's booking while logged in as Alice → 403; "Record change request" appends a note; "Mark cancelled" sets status |
| Account edit | display name change persists after reload; preferred station change is reflected on /account/rewards |
| Defect | /account "Next trips" (app.py:1921-1931) orders by departure_date without filtering is_upcoming, so the card lists five Completed Apr 14–17 trips first (fx_account_dashboard.png). → Required fix 2 |
| Defect | read-only pages write search_logs (see §1) → Required fix 3 |
| Inconsistency | the option card summary says "2h 50m" (departure→arrival incl. dwell) while its segment tag says "2h 46m" (running time) for the same Acela 2151 (_macros.html:100 vs :145) — pitfall #7 |
4. Task quality — 14 accepted (8 with small clarifications), 3 re-anchored, 1 rejected and replaced
Every ground truth below was computed from the seed through the app's own build_search_results / ORM and then confirmed on the rendered page during the live matrix. Steps = minimum agent actions.
| id | Solvable via UI | Leak | Distractors | Steps | Verdict | Note |
|---|---|---|---|---|---|---|
| 0 | yes | sorted-by-duration list puts the answer first (task asks to sort, acceptable) | 13 options (5 direct + 8 transfers); Carolinian/Silver Meteor 3h 12m near-miss | 6 | ACCEPT (clarified: report route + train no. + travel time) | "Acela" is real-world recallable → verifier requires train 2151 and the printed duration; both durations on the card accepted |
| 1 | yes | none (needs price sort) | NER $26.88 vs Carolinian $26.95 (7-cent near miss) | 6 | ACCEPT (clarified: leave direct-only off, report route + fare) | |
| 2 | yes | n/a | 12 outbound / 14 return options | 12 | RE-ANCHORED | original asked to "continue until you reach fare selection" — no verifiable answer; now fastest legs (Acela 2152 / 2151) → per-traveler Business fare $124.83 on the fare page |
| 3 | yes | n/a | leg 1: 2 options; legs 2–3: 1 each (Coast Starlight) | 12 | RE-ANCHORED | same problem; now cheapest option per leg → per-traveler Value fare $169.65. Distractors thin on legs 2–3 |
| 4 | yes | none | 1 itinerary, 3 room types | 8 | ACCEPT (clarified: report room type + extra cost) | "Roomette is cheapest" is recallable → verifier requires the synthetic +$316.00 |
| 5 | yes | list shows code/origin/date directly (browse-and-read task) | 15 bookings for Alice, 9 upcoming | 5 | RE-ANCHORED | "next upcoming trip" was ambiguous: two bookings share the earliest upcoming date (04-18) and the dashboard lists Completed trips first (§3). Now: upcoming Denver trip on My Trips → ALJDAM / CHI / Apr 20, 2026 |
| 6 | yes | none | 60 booking codes; lookup needs code + email/last name | 5 | ACCEPT (clarified: email given in the task) | Acela Express / Apr 20, 2026 |
| 7 | yes | none | — | 5 | ACCEPT (password added) | 4786 points, page-specific |
| 8 | yes (stateful) | — | — | 8 | ACCEPT (clarified) | original said "verify … reflected on the account" but /account never shows the preferred station; now read it back on /account/rewards. Verified by DB after-state |
| 9 | yes | route list card shows origin/destination only | 18 routes | 3 | ACCEPT | mirror uses VAN (Vancouver, WA) not the real VAC → recall gives the wrong code; verifier requires exact order VAN, SEA, PDX, EUG |
| 10 | yes | route page shows the alert title+message but not the next step; results cards do | 16 alerts, 4 Major Advisories | 3 | ACCEPT | next step = "Review the fare page before booking if you need a sleeper on a tighter schedule." |
| 11 | yes | also on /stations/DEN |
16 alerts, 6 station-scoped | 3 | ACCEPT | "track 3 instead of track 2" — verifier rejects "track 2" and "track 2 instead of track 3" |
| 12 | yes | none | 1 itinerary (only the Zephyr serves CHI→DEN) | 5 | ACCEPT | California Zephyr, $96.56 (Saver $72.06 is the near miss) |
| 13 | yes | none | 1 itinerary | 5 | ACCEPT (clarified: Saver fare class) | Capitol Corridor $37.22 (Value $42.80 otherwise) |
| 14 | yes | /stations list cards carry a "Checked baggage" tag (45/62 stations) — answerable from the list |
62 stations | 4 | ACCEPT | verifier gates on both detail pages; consider hiding the tag from list cards (§5.10) |
| 15 | yes | help card summary already states the 45-minute cutoff | help search returns 4 articles | 4 | ACCEPT (clarified) | "45 minutes" is real-world recallable → task now also asks which departures it applies to ("long-distance … staffed stations"), gate on the article page |
| 16 | — | — | — | — | REJECTED → replaced | "determine whether this mirror connects to a real payment or external credit system" is a meta question answered by the banner on every page (and by the task itself); not a web task. Replacement: help search "refund" → open the article → exact title "Refund language used in the mirror" + category "Refunds" (help search returns a single hit, so this is a browse-and-open task, low difficulty) |
| 17 | yes (stateful) | — | 5 direct options | 16 | ACCEPT (password added) | full funnel; verified by DB after-state (new booking row, 1 direct Business segment, ticket, passenger, mock payment, reward activity, +points) |
Difficulty: ≥5 actions on 12 tasks; multi-step comparison on 0/1/2/3/4/14; 2, 3 and 17 (12–16 actions with session state) would challenge a frontier agent. Search-based distractor density is modest (the mirror is a booking funnel, not a catalog): tasks 12, 13, 4 have exactly one matching itinerary, and 16 has one help hit.
5. Required fixes before merge
- Assets (BLOCKER).
refs/pr/30onChilleD/WebHarborlists only.gitattributes,README.mdand the 15 base tarballs —amtrak.tar.gzis not there (hf download … --revision refs/pr/30→ "File not found"). It only exists atlxr-max/WebHarbor@eac108f14be667b7e78972b0805d116ddea931d2. Open an HF PR that actually contains the tarball, merge it, then repin.assets-revision(currentlyrevision: main;mainis now pinned to65c479f894763f64c6073e0d180ebf542d1d2c02). Until thenscripts/fetch_assets.shcannot fetch the site and the image cannot be built from the pin. app.py:1921-1931account()— filter the "Next trips" query withis_upcoming(or order upcoming first). Five Completed trips currently head the list.app.py:1116-1126log_search()(calls at 1373, 1395, 1468) — read-only pages must not commit; drop theSearchLogwrite (or keep it in the session). The verifiers excludesearch_logsfrom their immutability checks for now.templates/index.html:20fffeature tiles: fix the text colour (white on pale).- Route hero cards: the SVG title is cropped (
main.css:363-366object-fit: coveron.visual-card img); either draw the title centred or useobject-fit: contain. templates/_macros.html:100vs:145— one duration per itinerary (2h 50m incl. dwell vs 2h 46m running time).templates/station_detail.html:28— let.info-table-wrapscroll horizontally (overflow-x: auto) so the page does not overflow at 400 px.- Branded 404 page (
@app.errorhandler(404)). app.py:1865-1866register: require first/last name instead of defaulting to "Traveler".templates/stations.htmllist cards: consider hiding the "Checked baggage" tag (task 14 is answerable from the list; grading is unaffected because the verifier gates on both detail pages).- The PR's edits to
.claude/skills/*,.gitignore,AGENTS.md,CONTRIBUTING.md,README.mdwere stale (15→16 sites) and were dropped in the rebase; the 25-site sweep is in the review PR — nothing to redo, just rebase on it.
6. Grading contract (reviewer deliverable)
sites/amtrak/verify/ — verify_lib.py + verify_0.py … verify_17.py + README.md; verifier_path and an English judge_rubric (rules / fact checkpoints, no answers) on all 18 tasks.jsonl rows. Deterministic-first, no LLM dependency (--no_llm True changes nothing; ground truth is hardcoded in the verifiers):
- Package validation — task id, non-empty final answer,
terminated/agent_done, ≥1 step, every URL HTTP on the same loopback origin/port asstart_url, every referenced screenshot a decodable PNG. - Navigation gates —
/booking/resultsquery-parameter gates (station code / "(CODE)" label / station or city name, ISO date,sort,fare_class,passengers,direct_only), ordered funnels viacheck_paths_in_order(round trip: outbound →leg=return→ select-trip → select-fare; multi-city; sleeper rooms; checkout: login → results → select-trip → select-fare → passengers → review → checkout → confirmation), login as the named account (the email must have been typed on/login). - Answer matchers — affirmative matching (negations, "instead of", "rather than" and per-clause attribution are respected) for money (
$96.56/96.56), durations (2h 50m/2 hours 50 minutes/2:50), dates (Apr 20, 2026/2026-04-20/April 20 2026), booking codes, track numbers, stop order, and station attribution (task 14). - SQLite snapshot contract —
initial.dbmust be the untouched seed (22-table schema, seeded counts, 4 benchmark users);after.dbmust keep the schema and the 13 catalog tables row-identical. Read-only tasks: all 8 mutable tables row-identical. Task 8: Alice'susersandreward_accountsrows differ only inpreferred_station_code, nothing else changes. Task 17: exactly one new booking for Alice (NYP→WAS, 2026-04-20, one-way, Confirmed) with a single direct Business segment, ≥1 Business ticket, its passenger rows, one Approved mock payment equal to the total, one reward activity naming the code, points credited by exactlyreward_points_earned, other users untouched; the reported code must equal the new row's code.
Snapshots come from --initial_db/--after_db, <run_dir>/initial.db|after.db, or docker cp from $WH_CONTAINER (default wh-review); missing/invalid snapshots fail closed. agent_demo/eval_judge.py --run_dir <run> --verifier True runs them (verified: exit 0 on genuine runs 9 and 17, exit 1 on a no-op).
7. Validation of the grading (CONTRIBUTING §C)
Live matrix — sites/amtrak/verify/tests/live_matrix.py drove every task through the real UI in Chromium, recorded agent_demo/agent.py-shaped run dirs (trajectory + screenshots + initial.db/after.db, site reset between stateful runs) and ran each verifier on five run kinds:
| task | no-op (empty answer) | pass (genuine drive) | shortcut (answer, no navigation) | wrong answer (near miss) | state-mismatch (DB unchanged) |
|---|---|---|---|---|---|
| 0 | FAIL final_answer_nonempty |
PASS | FAIL visited_results_nyp_was_0420_sorted_by_duration |
FAIL answer_names_route |
n/a |
| 1 | FAIL | PASS | FAIL visited_results_…_price_with_transfers |
FAIL answer_names_route |
n/a |
| 2 | FAIL | PASS | FAIL round_trip_workflow_in_order |
FAIL answer_has_business_fare |
n/a |
| 3 | FAIL | PASS | FAIL multi_city_workflow_in_order |
FAIL answer_has_value_fare |
n/a |
| 4 | FAIL | PASS | FAIL visited_results_sea_lax_0422_two_passengers |
FAIL answer_names_room_type |
n/a |
| 5 | FAIL | PASS | FAIL visited_login_page |
FAIL answer_has_booking_code |
n/a |
| 6 | FAIL | PASS | FAIL visited_trip_lookup |
FAIL answer_names_route |
n/a |
| 7 | FAIL | PASS | FAIL visited_login_page |
FAIL answer_has_points_balance |
n/a |
| 8 | FAIL | PASS | FAIL visited_login_page |
FAIL answer_has_station_code |
FAIL user_preferred_station_updated |
| 9 | FAIL | PASS | FAIL visited_cascades_route_page |
FAIL answer_lists_stops_in_order (VAC…) |
n/a |
| 10 | FAIL | PASS | FAIL visited_service_alerts |
FAIL answer_has_next_step |
n/a |
| 11 | FAIL | PASS | FAIL visited_service_alerts |
FAIL answer_has_boarding_track |
n/a |
| 12 | FAIL | PASS | FAIL visited_results_chi_den_0420 |
FAIL answer_has_flexible_price |
n/a |
| 13 | FAIL | PASS | FAIL visited_results_sac_sjc_0416_saver |
FAIL answer_has_starting_fare |
n/a |
| 14 | FAIL | PASS | FAIL visited_station_ANA |
FAIL answer_names_checked_baggage_station |
n/a |
| 15 | FAIL | PASS | FAIL visited_checked_baggage_timing_article |
FAIL answer_has_cutoff_minutes |
n/a |
| 16 | FAIL | PASS | FAIL performed_refund_search |
FAIL answer_has_exact_title |
n/a |
| 17 | FAIL | PASS | FAIL visited_login_page |
FAIL answer_has_new_booking_code |
FAIL exactly_one_new_booking |
90/90 cells as expected. The first full run exposed one verifier weakness that the matrix was built to catch: task 14's wrong answer "Anaheim (ANA) supports checked baggage; Santa Barbara (SBA) is carry-on only" passed because the matcher only looked for negations before a station mention. Fixed with per-clause attribution (affirmative_clauses_mentioning) and a trailing-negator rule ("instead of X", "rather than X", "not X") that also hardens task 11; the two tasks were re-run on a fresh seed (0 deviations).
Offline matrix — python -m unittest discover -s sites/amtrak/verify/tests (synthetic trajectories + seed-derived snapshots, no Playwright/docker/LLM): 13 tests OK — genuine, snapshot discovery in run dir, no-op, shortcut, 40 wrong-answer variants, other task id, unterminated, foreign origin (an amtrak.com URL), corrupt PNG, schema tamper, catalog tamper, seed tamper, collateral writes on read-only tasks (users / bookings / reward_accounts), search_logs tolerance, answer phrasing variants ("4,786", "2 hours 50 minutes", "track #3 rather than track 2", "Santa Barbara, not Anaheim, …"), task 8 state-mismatch / partial update / over-reach / wrong order, task 17 state-mismatch / coach instead of Business / points not credited / code mismatch / skipped fare page.
8. Evidence
# rebase state (Mac worktree)
$ git rev-list --count HEAD..upstream/main -> 133 (before) $ git log --oneline -6 (after)
57c92ae chore(amtrak): record the grading contract in tasks.jsonl; re-anchor tasks 2, 3, 5, 16
fcd4697 test(amtrak): add deterministic task verifiers, verify_lib and validation harness
4f18dce chore(amtrak): sweep docs and tests to 25 sites (40000-40024), tasks web -> 40024
d8d964a Finalize Amtrak asset workflow and checks (XuanRui LI)
2af6a47 Add Amtrak WebHarbor mirror (XuanRui LI)
3600493 Merge pull request #86 from evanz37/feat/walmart_careers
# HF assets
$ hf download ChilleD/WebHarbor amtrak.tar.gz --repo-type dataset --revision refs/pr/30
Error: File not found in repository.
$ curl -s https://huggingface.co/api/datasets/ChilleD/WebHarbor/tree/refs%2Fpr%2F30 -> 17 entries, no amtrak.tar.gz
$ curl -s https://huggingface.co/api/datasets/lxr-max/WebHarbor/tree/eac108f14be667b7e78972b0805d116ddea931d2 | grep amtrak
amtrak.tar.gz 116525
$ hf download lxr-max/WebHarbor amtrak.tar.gz --revision eac108f14be667b7e78972b0805d116ddea931d2 ; md5sum amtrak.tar.gz
1224b94239a4e23ace5691f632c896ee amtrak.tar.gz
$ tar -tzf amtrak.tar.gz | wc -l ; tar -tzf amtrak.tar.gz | grep -E '(^|/)\._|\.DS_Store' || echo none
308 / none
$ python3 scripts/validate_asset_archive.py ../tarball/amtrak.tar.gz amtrak
[fetch] validated 308 managed members for amtrak
$ python3 scripts/extract_asset_archive.py ../tarball/amtrak.tar.gz sites amtrak
[fetch] installed managed roots for amtrak
# standalone boot (Werkzeug, Dockerfile pins) + byte identity
$ python3 -m py_compile sites/amtrak/app.py sites/amtrak/seed_data.py sites/amtrak/_health.py && echo compile-ok -> compile-ok
$ grep -rn "from sites\.\|import sites" sites/amtrak || echo none -> none
$ rm -rf instance && cp -a instance_seed instance && python -c "from app import app; app.run(host='127.0.0.1', port=45005)" &
$ curl -so /dev/null -w '%{http_code}\n' http://127.0.0.1:45005/ ; curl -s http://127.0.0.1:45005/_health
200
{"db":true,"ok":true,"site":"amtrak"}
$ md5sum instance/amtrak.db instance_seed/amtrak.db # boot #1
cb4b6a5939b663017d9774f2d3a86884 instance/amtrak.db
cb4b6a5939b663017d9774f2d3a86884 instance_seed/amtrak.db
$ ... after the 44-page read-only Playwright sweep
be90a3ce612592f1aca661794877c368 instance/amtrak.db # <- search_logs: 40 rows
$ sqlite3 instance/amtrak.db "select query, category, result_count from search_logs limit 4"
('checked baggage','help',4) ('refund','help',1) ('baggage','global',4) ('Coast Starlight','global',5)
$ bash reset_site.sh && md5sum instance/amtrak.db instance_seed/amtrak.db # boot #2
cb4b6a5939b663017d9774f2d3a86884 instance/amtrak.db
cb4b6a5939b663017d9774f2d3a86884 instance_seed/amtrak.db
$ find static/images -type f -newer site.log | wc -l -> 0 (no SVG rewritten at boot)
# read-only audit (Playwright)
pages: 44 links: 215 console_errors: 2 (the two 404 probes) page_errors: 0
non-200 pages: [('missing_route', 404), ('missing_station', 404)] broken imgs: [] nav non-200: {} of 215
responsive: 768 ok ×3; 400: home 400/400, results 400/400, station 434/400 <- overflow
# live validation matrix (Playwright + verifiers)
$ AMTRAK_BASE=http://127.0.0.1:45005 ... uv run python sites/amtrak/verify/tests/live_matrix.py --out_dir runs/matrix
deviations: 1 (task 14 wrong-answer cell) -> matcher fixed ->
$ ... live_matrix.py --only 11,14
deviations: 0
$ python -m unittest discover -s sites/amtrak/verify/tests -p "test_*.py"
Ran 13 tests in 23.602s OK
$ uv run python eval_judge.py --run_dir runs/ej_17 --verifier True
pass: True success: True reason: all checks passed (exit 0)
$ uv run python eval_judge.py --run_dir runs/ej_noop --verifier True
pass: False success: False reason: final_answer_nonempty (exit 1)
Screenshots (review server ~/wh-review/amtrak/shots/): ro_home.png, ro_results_nyp_was_duration.png, ro_station_SBA.png, ro_service_alerts.png, ro_help_checked_baggage_timing.png, resp_400_station.png, fx_login_wrong.png, fx_account_dashboard.png, fx_fare_selection.png, fx_passenger_validation.png, fx_confirmation.png, fx_rooms.png, fx_trip_change.png, fx_stations_list.png; per-step screenshots for every task drive under runs/matrix/pass_N/screenshots/.
Required before merge (recap)
- Real HF PR with
amtrak.tar.gzagainstChilleD/WebHarbor, merged, and.assets-revisionrepinned to the merge SHA (§5.1). account()upcoming-trips filter (§5.2) and no DB writes on read-only pages (§5.3).- Rebase Add Amtrak WebHarbor mirror #39 onto the review branch (or let the
Review:PR be the merge vehicle) so the port is 40024 and the docs/tests are in sync; the maintainer may reassign the port at merge.
The grading contract does not depend on any of these: every verifier gates on navigation and, where it matters, on DB after-state, so the 18 tasks are gradable as soon as the assets are in place.
Screenshots
All cited screenshots: https://github.com/JeremyJC67/WebHarbor/tree/pr-screenshots/reviews/amtrak
| homepage | account dashboard — "Next trips" lists Completed trips first |
|---|---|
 |
 |
| station page at 400 px — overflow | checkout confirmation (task 17 funnel) |
|---|---|
 |
 |
Follow-up review —
|
| Item | State |
|---|---|
amtrak.tar.gz is not in any ChilleD/WebHarbor revision |
unchanged from the first review |
$ hf download ChilleD/WebHarbor amtrak.tar.gz --repo-type dataset --revision refs/pr/30
Error: File not found in repository.
$ curl -s https://huggingface.co/api/datasets/lxr-max/WebHarbor/tree/eac108f14be667b7e78972b0805d116ddea931d2 | grep amtrak
amtrak.tar.gz 116525
Open an HF PR against ChilleD/WebHarbor that actually contains the tarball, get it
merged, then repin .assets-revision to the merge SHA. Until then
scripts/fetch_assets.sh cannot fetch the site.
The archive itself is sound — we re-checked it this pass:
| Check | Result |
|---|---|
validate_asset_archive.py |
ok — 308 managed members |
| Seed md5 | cb4b6a5939b663017d9774f2d3a86884 |
| Shipped images reconciled against 17 template refs + 293 DB refs | ok — 1 zero-reference file (images/.gitkeep), 0 referenced-but-missing |
2. What this pass found
Severity is the merge checklist's, not ours. Everything below is fixed unless the last
column says otherwise.
Form prefill — the densest problem on a booking funnel
| Finding | Evidence | Fixed |
|---|---|---|
booking_defaults() shipped origin=NYP, destination=WAS and both dates into the search widget on / and /booking/search. Tasks 0, 1 and 17 all say "from NYP to WAS", so the station pair was typed for the agent before it acted. |
fresh cookie jar per page: {"control":"text[origin]","value":"NYP"} → now {"control":"text[origin]","placeholder":"Try NYP, WAS, CHI, SEA, or LAX"} |
ok |
booking_multi_city.html:35 hardcoded value="{{ ['2026-04-18','2026-04-20','2026-04-22'][idx] }}" — the exact three dates task 3 dictates, in order. |
date inputs now ship empty | ok |
"Start from this station" / "Search fares" / "Search later" / "Search this route" linked to /booking/results?origin=…&destination=WAS&departure_date=2026-04-20. From the NYP station page that one click was task 0. |
all four now open the empty search form | ok |
/trip-lookup shipped placeholder="alice.j@test.com" and placeholder="Jordan". On that form a surname is an accepted alternative to the email for authenticating a lookup, so the page displayed one of the two credentials it checks. |
placeholders now describe the field | ok |
Checked and not leaks, for the record: /account/edit (exempt — the user's own
profile form), the Value radio preselected on /booking/select-fare (all four fare
prices are printed simultaneously; the answer is the price), the /schedules landing
defaults (no task targets it), the passenger <select> defaults (no task names a
traveler type / age band / seat preference), and the "Use saved profile" button — the
name inputs are verified empty until it is clicked (alice_name_in_input_value: false).
Application robustness
| Severity | Finding | Evidence | Fixed |
|---|---|---|---|
| blocker | SECRET_KEY = "webharbor-amtrak-demo-key". Signing {"_user_id": bob} with it and sending the cookie from a jar that never logged in returned Bob's account. |
"forged_cookie_greets": "<h2>Welcome back, Bob Castillo.</h2>" → now lands on /login, authenticated: false |
ok |
| major | passengers was unbounded: /booking/passengers with passengers=100000 rendered 245 MB of HTML. |
{"html_bytes": 245000470, "passenger_cards": 100000} → {"html_bytes": 15496, "passenger_cards": 0} |
ok |
| major | MAX_CONTENT_LENGTH unset; a 60 MB body was accepted. |
POST /login 60 MB: 200 → 413 |
ok |
| major | No CSRF token on any form, and a cross-origin POST really did rewrite the profile. | display_name "Alice Jordan" → "CSRF Was Here" → now 403, unchanged |
ok |
| major | Invalid input succeeded silently. /booking/results with no query string at all searched NYP→WAS on the reference date and returned 13 options; an unparseable date silently became 2026-04-18; and station_lookup's substring fallback resolved origin=a to Albany and origin=New to Detroit. |
11 cases, all 200 → all 400 | ok |
| major | Open redirect: login?next=https://example.com/pwned followed it. |
now lands on /account |
ok |
| minor | /logout answered GET and HEAD, so a prefetcher could sign an agent out mid-task. |
200 → 405; POST → 302 | ok |
| minor | PRAGMA foreign_keys was 0; an orphan tickets row inserted happily. |
0 → 1; orphan insert now rejected: IntegrityError |
ok |
| minor | No commit was wrapped, so a failed one left the session in PendingRollbackError. |
register() and booking_checkout() now roll back and flash |
ok |
| minor | 404s were bare Werkzeug pages (207 bytes, no site chrome). | branded 400/403/404/405/413/500 handlers | ok |
| minor | register() defaulted a blank surname to "Traveler", so the header greeted a name the user never entered. |
both names now required | ok |
| minor | All three ID allocators are counter-based (X.query.count()+k). |
measured non-exploitable: 0 collisions with seeded codes over counts 0..399, injective over 2000 counts, and no route deletes a booking or user | report only |
Authorization was already correct and we left it alone: a foreign booking is 403 on
/trip/<code> and on /trip/<code>/cancel, 404 on /booking/confirmation, and all
eight account/funnel routes bounce an anonymous visitor to /login.
The two site defects from the first review
| Finding | Evidence | Fixed |
|---|---|---|
log_search() committed a SearchLog row from GET /search, /help?q= and /booking/results — read-only pages writing to the DB, the checklist's named case. |
after a full browse: instance md5 == instance_seed md5 == cb4b6a59…, search_logs 0 rows (was be90a3ce…, 40 rows) |
ok |
account() ordered "Next trips" by departure_date with no is_upcoming filter, then limited to 6 — so five Completed Apr 14–17 trips led the card. |
replaced by upcoming_bookings_for(), which filters before it limits |
ok |
UI / accessibility
| Finding | Evidence | Fixed |
|---|---|---|
Homepage feature tiles inherited color: white from .page-hero onto a #e7f3f7 tile. |
pixel-sampled (not read off background-color): 1.131:1 → 13.911:1; WCAG AA needs 4.5:1 |
ok |
| Hero SVG titles centre-cropped — "Acela Express" rendered as "la Express". | 18 of 24 sampled titles cropped → 0 of 24; object-position: left center moves the visible band from [150.3, 1049.7] to [0, 899.4] and the titles start at x=72 |
ok |
Horizontal overflow at 320 and 390 px: /stations/SBA 434, /stations/NYP 441, /schedules 494 (the first review missed /schedules). |
0 findings across 4 widths × 21 routes | ok |
One itinerary printed two durations: 2h 50m on the option head, 2h 46m on its own segment tag. |
now "2h 50m total" and "2h 46m on board" — both are correct, they were just unlabelled | ok |
Clean at all four widths: 0 broken images, 0 console errors, 0 page errors, 0 failed
requests, 0 external requests.
Our own grading contract
Two defects in the artifact we shipped, fixed in the same branch:
verify_libexcludedsearch_logsfrom every immutability check, because the site
used to write to it. Once that write was removed the tolerance became a hole — a route
that started committing again would have graded clean.search_logsnow joins the
read-only table set, and tasks 8 and 17 assert it too.- The screenshot check accepted any decodable PNG including a 1×1, so a forged
thumbnail satisfied it. Added a 64×64 floor and moved the offline harness onto a
320×200 fixture.
3. Every change we made to your code
Five reviewer commits, none of them amending yours (2af6a47, d8d964a are untouched):
0a832ac fix(amtrak): stop read-only pages writing to the DB; filter the account dashboard
d1e0de1 fix(amtrak): validate booking input and harden the app
1dd833f fix(amtrak): remove task-shaped prefills and fix the UI defects
dc52281 test(amtrak): tighten the verifiers now that read-only pages are read-only
df9220f fix(amtrak): drop the seeded identity from the trip-lookup placeholders
| File | What we changed |
|---|---|
app.py |
removed log_search() and its 3 call sites; replaced dashboard_upcoming_trip() with upcoming_bookings_for() and used it in account(); emptied origin/destination/both dates in booking_defaults(); made station_lookup() exact (code, (CODE) label, or an unambiguous station/city name, hub tie-break) instead of a substring match; replaced parse_int() with bounded_int()/bounded_passengers() (1..8); rewrote the head of booking_results() to validate trip type, leg, sort, time window, fare class, passengers, both dates and both stations and abort(400) instead of defaulting; same for schedules_page() and the multi-city search phase; SECRET_KEY from AMTRAK_SECRET_KEY or a per-process random; MAX_CONTENT_LENGTH = 2 MB; SESSION_COOKIE_HTTPONLY / SAMESITE=Lax; a before_request that 403s unsafe methods with a foreign Origin; PRAGMA foreign_keys=ON on connect; logout is POST-only; safe_next_url() for login?next=; first and last name required at registration; IntegrityError rollback in register() and booking_checkout(); branded 400/403/404/405/413/500 handlers |
templates/_macros.html |
search-widget values now come from the emptied defaults; the two durations labelled "total" and "on board" |
templates/booking_multi_city.html |
dropped the hardcoded three-date array |
templates/station_detail.html, index.html, routes.html, route_detail.html |
the four quick links now open /booking/search instead of a pre-built /booking/results query |
templates/base.html |
"Sign out" is a POST form (matches the logout route) |
templates/trip_lookup.html |
three placeholders no longer name the seeded account |
templates/error.html |
new — the branded error page |
static/css/main.css |
explicit color on .progress-step; object-position: left center on .media-card .media-top img; overflow-x: auto on .info-table-wrap plus min-width: 0 on grid children; .inline-logout |
Nothing in seed_data.py, tasks.jsonl or instance_seed/ was touched, so the seed md5
is unchanged and your reported counts still hold.
4. Regression evidence
Every contributor-code change was re-verified, not just compiled.
# full live validation matrix, real Chromium, after all five commits
$ AMTRAK_BASE=http://127.0.0.1:46006 ... uv run python sites/amtrak/verify/tests/live_matrix.py
| Amtrak--0 | FAIL (final_answer_nonempty) | PASS | FAIL (visited_results_...) | FAIL (answer_names_route) | n/a |
... 18 rows ...
deviations: 0 # 90/90 cells
# offline matrix
$ python -m unittest discover -s sites/amtrak/verify/tests -p "test_*.py"
Ran 15 tests in 25.373s
OK # was 13 tests
# byte identity after a full browse + every probe in the audit
$ md5sum instance/amtrak.db instance_seed/amtrak.db
cb4b6a5939b663017d9774f2d3a86884 instance/amtrak.db
cb4b6a5939b663017d9774f2d3a86884 instance_seed/amtrak.db
$ sqlite3 instance/amtrak.db 'select count(*) from search_logs'
0
# the tightened station lookup did not change any task's result set
NYP->WAS 13 options WAS->PHL out 12 / return 14 SEA->LAX 1 CHI->DEN 1 SAC->SJC 1
Each "this is clean now" claim was also shown able to fail. Two examples:
# authorization
foreign booking as Alice -> HTTP 403 (baseline)
inject: user_can_view_booking() -> return True
foreign booking as Alice -> HTTP 200 (the check can fail)
reverted -> HTTP 403
# read-only pages must not write
instance md5 == seed md5, search_logs 0 rows (baseline)
inject: put the SearchLog commit back on /search
instance md5 44124db5... != seed, search_logs 1 row
verifier verdict: pass=False reason=read_only_search_logs_unchanged exit=1
reverted -> instance md5 cb4b6a59..., search_logs 0 rows
5. Not verified in this pass
Docker was unavailable to this pass, so the container side needs re-running against
df9220f before merge. The first review did build and exercise the full 25-site
environment from this branch and it passed every control-plane check, but that was
before these five commits.
hf download lxr-max/WebHarbor amtrak.tar.gz --repo-type dataset \
--revision eac108f14be667b7e78972b0805d116ddea931d2 --local-dir sites/
python3 scripts/validate_asset_archive.py sites/amtrak.tar.gz amtrak
python3 scripts/extract_asset_archive.py sites/amtrak.tar.gz sites amtrak
./scripts/build.sh webharbor:amtrak-audit
docker run -d --rm --name wh-amtrak-audit -p 8201:8101 -p 41000-41024:40000-40024 webharbor:amtrak-audit
curl -s http://localhost:8201/health | python3 -m json.tool | head
for p in $(seq 41000 41024); do curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/; done
curl -X POST http://localhost:8201/reset/amtrak
docker exec wh-amtrak-audit md5sum /opt/WebSyn/amtrak/instance/amtrak.db \
/opt/WebSyn/amtrak/instance_seed/amtrak.db
# the M6 fix, verified in the image: a read-only sweep must leave the DB byte-identical
curl -s 'http://localhost:41024/search?q=baggage' >/dev/null
curl -s 'http://localhost:41024/help?q=refund' >/dev/null
curl -s 'http://localhost:41024/booking/results?origin=NYP&destination=WAS&departure_date=2026-04-20' >/dev/null
docker exec wh-amtrak-audit md5sum /opt/WebSyn/amtrak/instance/amtrak.db
docker exec wh-amtrak-audit sqlite3 /opt/WebSyn/amtrak/instance/amtrak.db 'select count(*) from search_logs'
curl -X POST http://localhost:8201/reset-all
cd agent_demo && AMTRAK_BASE=http://127.0.0.1:41024 WH_CONTAINER=wh-amtrak-audit \
AMTRAK_RESET_CMD='curl -sX POST http://localhost:8201/reset/amtrak' \
uv run python ../sites/amtrak/verify/tests/live_matrix.py --out_dir runs/amtrak_container
docker stop wh-amtrak-auditAlso not done, and out of scope for us: re-uploading a corrected tarball to your HF
bundle (§1), and content judgement calls on the seed — for example booking ALZN9E is
WAS → WAS, a trip that goes nowhere, which only you can decide about.
Summary: the Flask app is now sound — it boots byte-identical, keeps that identity
through a full read-only browse, rejects invalid input with 400 instead of guessing, and
no longer hands the agent a pre-typed answer on any of the four form surfaces. The 18
tasks are gradable and the grading contract is stricter than it was. The only thing
standing between this and a merge is the Hugging Face upload.
Container re-verification of this branch (df9220f, full 25-site image)
Docker was unavailable to the audit pass itself; it has now been run. amtrak.tar.gz was taken from
lxr-max/WebHarbor@eac108f, validated (308 managed members) and extracted, because blocker 1 is still open
and no ChilleD/WebHarbor revision carries the archive.
| check | result |
|---|---|
docker build |
ok |
| HTTP sweep, alt ports 44000-44024 | 25/25 return 200 |
GET /health |
ok, 25/25 alive and ready, amtrak on internal 40024 |
POST /reset/amtrak → md5 instance vs instance_seed |
cb4b6a5939b663017d9774f2d3a86884 both |
| dirty request then reset | still cb4b6a59… |
docker restart then md5 |
still cb4b6a59… |
POST /reset-all |
ok, 25/25 ready, 1.52 s |
verifier no-op matrix, in-container DBs via docker cp |
18/18 FAIL, 0 errors |
walmart_careers / rotten_tomatoes registry + docs tests |
4 passed, 3,343 subtests |
| offline verifier suite inside the image | 15 passed, 119 subtests |
The five reviewer commits behave in the image exactly as they did standalone:
GET /logout 405
GET /no-such-page 404, 10664-byte branded page
GET /login?next=https://evil.example/pwn redirect_url empty (not followed)
GET /booking/results (no query string at all) 400 (was: silently searched NYP->WAS)
GET /booking/results?origin=a&... 400 (was: resolved to Albany)
GET /booking/results?...&passengers=100000 400 (was: a 245 MB response)
POST /account/edit with Origin: http://evil.example 403
md5 before / after GET /search + /help?q= + /booking/results
cb4b6a5939b663017d9774f2d3a86884 / cb4b6a5939b663017d9774f2d3a86884 -> byte-identical
homepage search widget: name="origin" value="" placeholder="Try NYP, WAS, CHI, SEA, or LAX"
Two notes where the first probe run was mis-specified rather than the code being wrong. A 300 KB body
returns 200 because the cap is 2 MB, not 300 KB; 1 MB and 3 MB bodies both return 413. And
POST /logout with no Origin header returns 302, not 403 — protection here is an Origin check plus
SameSite=Lax, not a per-form token, so a request that presents no origin at all is treated as same-site.
The cross-origin case, which is the one that matters, is 403.
|
Correction to one sentence in my previous comment, before it gets carried forward as a measured fact. I wrote that "a 300 KB body returns 200 because the cap is 2 MB, not 300 KB; 1 MB and 3 MB bodies both
Nothing changes in the finding or the fix: before the change a 60 MB body was accepted, and now it is not. |
Tooling disclosureAdded after the fact, because it belongs in the public record and not only in our internal notes. The second-pass audit had a general-purpose LLM available to it as a judgment aid — used for For this site I have no record of it changing any conclusion; the findings above came from the probes and No call log was kept, so I cannot enumerate every consultation — that is the honest limit of this |
2 participants
Verification date: 2026-06-02
Site name: Amtrak
Site slug:
amtrakUpstream URLs:
Contribution type: New Website
Contributor: XuanRui LI / Lxr-max
Contact email: mingduanmu845@gmail.com
Safety note: deterministic demo travel and booking data only; no real payment, no real booking, no live Amtrak calls.
Implementation summary
This PR adds a self-contained local Amtrak mirror built with Flask, SQLAlchemy, Jinja2, and SQLite. The site covers route browsing, station discovery, one-way / round-trip / multi-city search, schedule-style results, fare comparison, sleeper room selection, passenger details, mock checkout, booking confirmation, trip lookup, trip management, and rewards/account flows.
The implementation uses deterministic synthetic data only and does not make external runtime calls. A small follow-up commit polishes a few remaining user-visible checkout / refund labels so the mirror reads like a finished demo instead of a template.
Key user flows
Seed row counts
Benchmark users
alice.j@test.combob.c@test.comcarol.d@test.comdavid.k@test.comTestPass123!Task count and categories
Selected WebSyn port
40015Key routes
//home/routes/routes/<route_slug>/stations/stations/<station_code>/destinations/destinations/<city_slug>/deals/schedules/status/service-alerts/help/search?q=/booking/search/booking/results/booking/select-trip/booking/select-fare/booking/rooms/booking/passengers/booking/review/booking/checkout/booking/confirmation/booking/multi-city/trip-lookup/trip/<booking_code>/account/account/trips/account/rewardsVerification summary
Syntax and task validation
py -m py_compile sites/amtrak/app.py sites/amtrak/seed_data.pysites/amtrak/tasks.jsonlvalidated successfullyweb_name,id,ques,web,upstream_urlhttp://localhost:40015/Docker build and run
bash scripts/check_assets.shbash scripts/build.sh webharbor:amtrak-prdocker run -d --rm --name webharbor-amtrak-submit -p 8202:8101 -p 42000-42015:40000-40015 webharbor:amtrak-pr40015Runtime smoke tests
Public route checks returned
200for://home/routes/stations/booking/search/booking/results?origin=NYP&destination=WAS&departure_date=2026-04-20&passengers=1&fare_class=saver/trip-lookup/service-alerts/loginSeed-backed flow checks passed for:
alice.j@test.comSEA -> LAX, trip173, roomette)ALLTC3Reset and MD5 result
POST http://127.0.0.1:8202/reset/amtrakreturned{"pid":309,"ready":true,"site":"amtrak"}cb4b6a5939b663017d9774f2d3a86884 /opt/WebSyn/amtrak/instance/amtrak.dbcb4b6a5939b663017d9774f2d3a86884 /opt/WebSyn/amtrak/instance_seed/amtrak.dbScreenshot paths
Local review screenshots were refreshed on 2026-06-02 at:
C:\Users\34475\Desktop\VScode\WebHarborRepo-amtrak\sites\amtrak\scraped_data\submission_reviewIncluded captures:
home.pngbooking_search.pngbooking_results.pngfare_selection.pngsleeper_room_selection.pngpassenger_details.pngcheckout.pngconfirmation.pngroute_page.pngstation_page.pngtrip_lookup.pngaccount_trips.pngservice_alerts.pnglogin_flow.pngmobile_home.pngHF asset status
HF asset packaging and upstream PR creation have now succeeded.
C:\Users\34475\Desktop\VScode\wh-static-pr-amtrak\amtrak.tar.gzPer the repo docs,
.assets-revisionis intentionally unchanged until the HF asset PR is merged and the merge SHA is available.Known limitations
.assets-revisionstill points atmainuntil HF discussion Add NBA mirror #30 is merged upstream.Manual reviewer notes