Review: Add Kaggle mirror site (#54) - #106
Merged
Merged
Conversation
Addresses review on PR aiming-lab#53: - Rebase onto current main; register healthline as index 16 -> port 40016 (append after merriam_webster in websyn_start.sh + control_server.py; Dockerfile EXPOSE -> 40000-40016). merriam_webster preserved. - websyn_start.sh site-count comments reconciled to 17. - .assets-revision pinned to HF PR aiming-lab#40 (clean tarball, no macOS AppleDouble junk, based on current main -> all 17 tarballs). Bump to merged SHA once HF PR aiming-lab#40 lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer deliverable for the Healthline mirror (site by @JeremyJC67, PR aiming-lab#53): one deterministic verifier per task under sites/healthline/verify/, plus verifier_path + judge_rubric recorded in every tasks.jsonl row. No answer key in tasks.jsonl — ground truth lives only inside the verifiers. Deterministic-first: (1) trajectory navigation gate (anti knowledge-shortcut, important here since several answers are medically recallable), (2) SQLite DB after-state for the stateful tasks (save / register / password-change, plus DB cross-checks for saved-count and reading-history), (3) answer vs frozen ground truth, with the LLM only as an anchored consistency check. Validated against the official react agent (agent_demo/agent.py): a no-op run fails all 20 verifiers; a human answer-check confirms every frozen value matches what the page renders. On the full 20-task run, after fixing one too-strict nav gate the run itself surfaced, verifier and LLM judge agree 15/20, and all 5 remaining divergences are the deterministic verifier being correct while the LLM judge false-positives (blank answer / knowledge-shortcut) or false-negatives (DB-confirmed save / password change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Search result cards showed drug_class (e.g. 'ACE inhibitor'), letting T16 be answered from the results page without opening the drug pages. Show the broader category (e.g. 'Heart Medications') instead — same card layout, but the ACE-inhibitor-vs-statin distinction now requires opening each detail page. drug_class still shown on the drug detail page; search backend still matches on drug_class. Addresses DEM1TASSE review note on PR aiming-lab#53. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- web: all 20 tasks pointed at :40015 (Merriam-Webster) after the site was rebased to index 16 / :40016; corrected to http://localhost:40016/. - judge_rubric for Healthline--3 and Healthline--16: spell out the full ground-truth answer the page states (T3 symptom list, T16 what each drug treats) so the LLM judge grades against complete facts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Offline Flask mirror of https://doctor.webmd.com/ ("WebMD Care"): deterministic synthetic directory of 224 doctors across 10 specialties and 8 cities around Newark, DE 19711, with hospitals, group practices, Choice Awards, real auth, saved providers, appointment requests and pending reviews. - Build-generated seed (.build-generated-seed): the Dockerfile regenerates instance_seed/webmd_doctor.db plus 224 Pillow initials avatars and 90 video poster frames from seed_data.py; no Hugging Face assets, .assets-revision untouched. - One seeded RNG (20260910), literal reference date 2026-09-10, hardcoded scrypt hashes for the four benchmark users, whole-function seed gates, seed_metadata version + expected-count validation; byte-identical after /reset and docker restart. - Deterministic longest-match query parser (specialty / condition / procedure / insurer / gender / virtual), conjunctive filters, Best Match / Distance / Average Rating / Number of Ratings ranking, numbered pagination. - Registered as index 24 in websyn_start.sh and control_server.py; Dockerfile EXPOSE raised to 40024, header comment bumped to 25 sites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
Contributor-side task definitions only (web_name, id, ques, web, upstream_url): 9 lookups (0-7, 15), 4 multi-constraint (9-12), 2 compares (13-14), 5 stateful (8, 16-19). Every ques routes through a specialty + city, a hub page or an awards class before naming a doctor (name search is a non-goal, as upstream); credentials are embedded for login tasks; no answers, verifiers or rubrics. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
- seed: two extra West Chester cardiology slots appended after the grid (task 12 now has 8 doctors on the city page, 6 rated 4+ with exactly one male); deterministic language top-up for physicians covering three offices (task 1 target speaks 3 languages); EXPECTED_COUNTS refreshed (226 doctors, 202 in radius) - pagination: disabled prev/next rendered as spans instead of live links to an empty page - filter bar: empty and default params are dropped from auto-submitted URLs Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
Section-by-section pass against scraped_data/reference at 1440 px: - profile: booking widget overhangs the hero into the rail, tabs span the main column only, colleagues tile grid / hospital affiliation card in the stack, Basic rail below the tabs, NPI folded into the certification card, nearby + transparency panels full width, star picker, review controls, check-circle perspective icons, two-column bulleted top-20 lists - home: icon tile row for popular specialties, filled uppercase View Profile buttons, upstream heading sizes - filter pills in dark text with navy active state; typeahead no longer clipped by the search bar, prefix highlighting - hospitals / group practices hubs rebuilt as landing pages (state chips, name search, top-4 cards, search-all button, care-type chips); state lists drop the sliders button and gain a name filter - hospital / practice details: specialty select, map-on-top locations, ratings card, text flag lines - awards page art panels + larger headings; recipients grouped by state with a state filter; guidelines as a centred numbered card; sign-up terms line and mm/dd/yyyy date field; login/state/city copy per upstream - landing "Highest Rated" strip now the 25-mile ring so no task target is showcased there Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
discussion_new.html renders errors for the title and the body but not for the forum select, so posting an invalid forum re-rendered the form with no visible feedback (the thread is correctly not created). The forum errors are now shown next to the field like the others. Verified: the error text appears after a bad submit, no row is created, and removing the block fails the checker again.
The forum select listed only real forums, so the browser preselected the first
one (General) and a submit that never touched the select silently created the
thread in the wrong forum. The field now offers an explicit empty first choice
("Select a forum") and DataRequired rejects it, so the form has to be answered.
Verified: the empty option is first with nothing preselected, an untouched submit
shows the validation error and creates no row, an explicit choice still persists,
and removing the empty option fails the checker again.
Brand cyan was used for both action surfaces and link text: white on #20beff is
2.13:1 and #1ba7e0 text on white is 2.74:1, with tier badges between 2.05 and 3.05
(measured 26 link nodes and 22 button nodes below 4.5 on the audited pages), and
.focus-visible was absent on form fields because .field input:focus set
outline:none with only a 2.13:1 border colour change.
Added --action (#0b6fa4, 5.49:1 in both directions) for buttons, chips, tabs,
vote buttons, links and the hero-adjacent surfaces, kept the brand cyan decorative,
darkened the five tier colours to 5.49-9.39:1, gave the letter-tile fallback the
action gradient, and replaced outline:none with a global
:focus-visible { outline: 2px solid var(--focus); outline-offset: 2px } rule (plus
a white ring inside the dark hero). Audited with the pixel sampler: 0 low-contrast
text nodes on /, /competitions, /discussions, /rankings?category=notebooks and
/login (was 5 on the home page alone); the focused field now reports
'rgb(11, 111, 164) solid 2px'. Re-injecting #eeeeee card titles and outline:none in
a scratch instance makes the auditor flag them again.
The wrapped header layout started at max-width 760px, but the single-row header (logo + 520px search + six nav links + auth buttons) needs about 820px, so at 768px and 800px the document was 50px wider than the viewport and the Register button was clipped; the search input collapsed to a circle. The breakpoint moves to 980px, which is where the card grid already switches to two columns (line 105), so the tablet range now gets the wrapped header. A sweep over 320/360/390/600/760/768/800/900/980/1024/1200/1440 on five routes reports no horizontal overflow and no clipped elements; restoring 760px makes 768px and 800px fail again.
/competitions/<slug>/leaderboard and templates/leaderboard.html had no inbound reference (the detail page's Leaderboard tab links to ?tab=leaderboard), so the route was orphaned. The leaderboard tab now links it as a deep link; the tab itself is unchanged, so task 4's flow still works.
user_profile rendered a user's hosted competitions in insertion order, which put sara_timeseries's earliest-deadline competition (Global Wheat Yield Forecast, 40 days left) in the first card, so the comparison the task asks for could be answered by guessing the first item. The list is now ordered by title (Arctic Sea Ice, City Energy Load, Global Wheat Yield) so the answer has to be found by comparing the deadlines shown on the cards.
The Code listing rendered only 'Most comments' and 'Recently run', so a browser preselected 'comments' while the backend defaulted to 'recent': any form submit silently changed the ordering, and there was no way to sort by votes even though the handler fell through to it. The select now lists 'Recently run' first (the backend default), then 'Most votes', then 'Most comments', and the handler has an explicit votes branch with the no-query fallback documented. Verified: the selected option matches the default, ?sort=votes returns the most-voted notebook first, and removing the option fails the check.
/datasets/<slug>/download was a GET that wrote the download counter, so a prefetcher, crawler or link checker moved the counter (verified: a single GET incremented it), and the route's side effect contradicted the read-only model. It is now POST-only with a CSRF token, the detail page renders a form button with the same styling (.download-form), and task 3's reviewer plan clicks that button. Verified: GET -> 405 with the counter unchanged, token-less POST -> 400 with the counter unchanged, token POST -> 302 and +1.
No MAX_CONTENT_LENGTH was configured, so an 8 MB (and earlier a 60 MB) multipart body was accepted: Werkzeug's 500 KB max_form_memory_size only covers non-file parts, and file parts streamed to disk without a cap. The limit is now 2 MiB (largest legitimate form is a 240-char title plus a discussion body) and a new 413 handler renders 413.html with the site chrome. Verified: an 8 MB multipart POST returns 413 with the site page, and a normal login POST still returns 302; the pre-fix app returns 200 for the same request.
The templates' image fallback pointed at avatars/default.png, which is byte-identical to avatars/alicejdata.png (md5 cba9811627f5a439dae16a273f458a62), and User.avatar_url defaulted to the same file, so every account without an avatar - including any newly registered user - rendered Alice's photo. A new avatar() macro renders the stored image when present and otherwise a letter tile (initial on the action-colour gradient); User.avatar_url and register() now default to an empty string, and all nine call sites (nav, profile, account, search rows, rankings, competition host, dataset owner, notebook author, model owner) use the macro. Verified: a new account stores '' and shows a letter tile, seeded users keep their own photos, default.png is no longer referenced, and a 21-route sweep returns 200 everywhere (a missing macro import briefly returned 500 on /rankings and /search and was fixed before this commit).
base.html's nav calls avatar(current_user, 32) but only imported the macro into child templates, so every page whose own template does not import it returned 500 while signed in (jinja2 UndefinedError on /account/edit, /discussions/new, /account/password; logged-out renders were unaffected because the nav avatar sits behind current_user.is_authenticated). Caught by the authenticated route sweep. base.html now imports the macro itself; the authenticated and logged-out sweeps both return 200/404 as expected.
…ccount deletion SQLite leaves foreign keys off unless a connection enables them, so the schema's declared FKs were decoration and /account/delete removed only the users row: a verified probe showed an orphan vote insert succeeding and a deleted account leaving its votes behind (author usernames in discussions/comments dangled too). A connect-time event now sets PRAGMA foreign_keys=ON (app context reports [(1,)] and an orphan vote insert raises IntegrityError), and account deletion removes the account's comments, discussions, submissions, votes, bookmarks, follows (both directions) and competition entries before deleting the user, with a rollback plus a flash message on SQLAlchemyError instead of a 500. Verified: after deletion the user row is gone, every table reports 0 orphan rows and PRAGMA foreign_key_check is empty; switching the pragma back to OFF reproduces the orphan vote.
The verifiers read only the URLs, the answer and the DB state, so a trajectory
with its tail cut off still passed: the truncated Kaggle--6 variant (last step
'check' instead of 'done') returned PASS against the pre-fix verifier, and the
same class passed for Kaggle--10/12 in stage 1.
verify_lib.run_complete() now requires the fields agent_demo always writes:
terminated is True, termination_reason is 'agent_done', the last step is the
final 'done' step, and that step carries an answer. All 20 verifiers assert it via
a run_complete check. Verified: truncated variants now fail with reason
run_complete ('the last step is ... not the final done step'), runs that stopped on
max_steps fail too, and the real runs pass ('terminated with agent_done after 15
steps').
Records the stage-1 findings for sites/kaggle (issue-table.md) and their dispositions after remediation: the pin/registry/verifier-contract blockers that were fixed, the application and verifier hardening (H1-H9, M1-M10, L1-L9), the three items that stay blocked on the Hugging Face asset archive or an LLM endpoint, the ground truth for all 20 tasks derived from the seed database, and the validation runs (production grading command 20/20, 227-case negative matrix with zero mismatches, four-width crawl, cold image build with 25/25 sites alive and byte-identical reset-all, container-generated seed equality, leak re-scan).
Add the reviewed FedEx mirror as site 25 on port 40024. Offline FedEx mirror with 18 tasks. The seed database is generated from tracked source at image build time, every graded target is derived from the supplied initial database by verify/ground_truth.py, and the answer-isolation tests keep graded values out of tracked source, help copy and static page surfaces. Media is distributed through the pinned Hugging Face archive at revision 68dcbf2c.
…WebMD Doctor branch Registry: append `webmd_doctor` after `fedex`, so FedEx keeps index 24 / port 40024 and WebMD Doctor moves to index 25 / port 40025 (26 sites, 40000-40025). Conflict resolutions: - `.assets-revision`: keep `ad6f424f` (current HF dataset main, which carries `webmd_doctor.tar.gz` and the byte-identical `fedex.tar.gz` blob) instead of main's FedEx-only pin `68dcbf2c`. - `scripts/fetch_assets.sh`: take main's registry-scoped fetch implementation. - `Dockerfile`: keep both site blocks (FedEx asset gate + WebMD generated-asset gate and source-built seeds) and widen `EXPOSE` to 40000-40025. - `README.md`, `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, `agent_demo/README.md`, `.claude/skills/*`: 26 mirrors, port range 40000-40025, alt ports 41000-41025. - `sites/walmart_careers/tests/test_integration.py`, `sites/rotten_tomatoes/tests/test_environment_quality.py`: take main's registry-derived assertions and extend them for `webmd_doctor` (40025) and its generated-asset gate. Follow-on work required by the port move: - `sites/webmd_doctor/tasks.jsonl`, `README.md`, `verify/verify_lib.py` comment, `tests/test_task_breadth.py` (now derives the port from the registry), `verify/tests/test_verify_lib.py`: port 40025. - new `sites/webmd_doctor/tests/conftest.py`: rebuild the build-generated seed when it is absent and install the runtime copy the app binds, so the suite runs after `scripts/fetch_assets.sh` alone (the Walmart mirror's convention). In that state the suite previously failed 7 tests. Verified after the merge: webmd site tests 19, webmd verify 447, walmart 51, fedex 91 (+1341 subtests), rotten_tomatoes 56 (+4129 subtests), compass 94.
feat(webmd_doctor): add WebMD Doctor mirror (site 25, port 40024)
…thline to index 26 (port 40026)
…h_assets.sh (cherry picked from commit c944482a7e91d3dfedd6e65f2baf4eadf7a34cf7)
…hline Review: Add Healthline mirror + task verifiers (site by @JeremyJC67, verifiers by reviewer) (aiming-lab#59)
…kaggle to index 27 (port 40027)
…asset-pin rationale Shared docs (AGENTS.md, CLAUDE.md, CONTRIBUTING.md, README.md and the five .claude/skills/*/SKILL.md files) still described 27 sites / 40000-40026 after kaggle was appended as site 28 (index 27, port 40027). Bump the site counts and port ranges to 28 / 40000-40027 (alt-port mapping 41000-41027) and add Kaggle to the README site enumeration. .assets-revision: rewrite the pin rationale to the measured state. The pin 64264d065cdb0b7755ee99dab356be5733d9ddef is the HF dataset `main` head, its kaggle.tar.gz is 11278106 bytes (sha256 dd6f1ab3...) and passes scripts/validate_asset_archive.py with 73 managed members and no bare kaggle/static directory entry; refs/pr/37 covers only 17 of the 28 registered sites. The pin value itself is unchanged.
Contributor
|
Thanks for your contribution! @TabsPhasers @KaKituken @Django-Jiang |
Raibows
marked this pull request as ready for review
September 13, 2026 15:06
evanz37
added a commit
to evanz37/WebHarbor
that referenced
this pull request
Sep 13, 2026
… into the UC Berkeley branch Registry: append `berkeley` after `kaggle`, so Healthline keeps index 26 / port 40026 and Kaggle keeps index 27 / port 40027; UC Berkeley moves to index 28 / port 40028 (29 sites, 40000-40028). Conflict resolutions (append rule): - `websyn_start.sh`, `control_server.py`: keep all of upstream's 28 entries in order and append `berkeley` last (both registries identical, 29 entries). - `Dockerfile`: keep every upstream site block (Healthline's migrate + prune step included) and berkeley's build-generated seed block; header 28 -> 29 sites; `EXPOSE 8101 40000-40027` -> `40000-40028`. - `README.md`, `AGENTS.md`, `CONTRIBUTING.md`, `CLAUDE.md`, `agent_demo/README.md`, `.claude/skills/*`: take upstream's text, then 29 sites, port range 40000-40028, alt ports 41000-41028, and `UC Berkeley` appended to README's mirror list. - `scripts/fetch_assets.sh`, `scripts/check_assets.sh`: take upstream's registry-scoped implementations and keep berkeley's `.build-generated-seed` exemption on top. Follow-on work required by the port move: - `sites/berkeley/tasks.jsonl` (22 `web` rows), `app.py` PORT default, `README.md`, `verify/TASK_REVIEW.md`, `verify/verify_lib.py` comment, `tests/test_integration.py` (SITE_INDEX 28 / SITE_PORT 40028) and `verify/tests/test_tasks_contract.py`: port 40028. - `scripts/check_site_registry.py` (upstream's new gate) reports 29 sites consistent across both registries, `Dockerfile EXPOSE` and every `tasks.jsonl` port. Verified after the merge: registry gate green; berkeley site 27, berkeley verify 534, healthline 27; walmart_careers 50 passed and rotten_tomatoes 55 passed (+4129 subtests) each with one expected red that is pre-existing on upstream/main and reproduced there from a pristine `git archive` tree (VERIFICATION.md §re-slot); kaggle ships no pytest suite. Container rebuilt and re-run on 44000-44028: 29/29 sites 200, /health 29 alive+ready, `POST /reset/berkeley` byte-identical (`f2f0187c...`) before and after `docker restart`, no-op matrix 22/22 FAIL with 0 infra errors. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
基于当前
main（36004932bdf82afbe36dc14e00f66841eccf9946，24 站）Review 并接管 @KaKituken 的 #54，延续 @Django-Jiang 的 #70；保留原作者与提交归属。Kaggle 追加为 index 23 / 容器端口40024（25 站），README/AGENTS/CLAUDE/CONTRIBUTING、Dockerfile、control_server.py、websyn_start.sh随之更新。Draft：素材交付（HF PR #74）尚未合并，
.assets-revision仍指当前 main pin，未设为可合并状态。Original review resolution
映射 #54 的
REQUEST CHANGES（MufanQiu）逐项：ChilleD/WebHarborReview: Add Phys.org mirror + task verifiers (site by @Within-yao, verifiers by reviewer) #74（kaggle.tar.gz）。blocking：待 merge 后把.assets-revision指到 merge SHA。40024；control_server.py与websyn_start.sh顺序一致。ResNet-50 Chest X-Ray Classifier(Apache 2.0) 与 TensorFlowDenseNet-121 Chest X-Ray Classifier(MIT)。.DS_Store/AppleDouble 重打：本地标准化重包完成，交付走 HF Review: Add Phys.org mirror + task verifiers (site by @Within-yao, verifiers by reviewer) #74。Review changes
sites/kaggle/：Flask+SQLite 站点（app/seed/templates/static）。sites/kaggle/verify/verify_lib.py+verify_0.py…verify_19.py；tasks.jsonl回填verifier_path+judge_rubric，无answerkey。control_server.py、websyn_start.sh、Dockerfile（40024），文档站数/端口文本更新。Asset delivery status
kaggle.tar.gz，11,278,264bytes，SHA-256c533c283fb68f924989523b6fb686beecf66846cb13579ca0eae32b524c99343，82 entries.assets-revision：待 pin（HF merge 后指向 merge SHA；本 PR 暂不改）Validation
sha256:732be610…，seed2f29f638…）；health / reset byte-identity PASS。exit=1（正常 FAIL）；正例全exit=0（PASS）。Known non-blocking observations (recorded, not fixed)
以下为本地完整遍历站点时发现的镜像保真度/可用性差异。逐条评估后本 PR 不修，理由统一为：没有任何 Task 依赖这些能力，且都不影响判分契约（no-op FAIL、正例 PASS、DB after-state 由站点本地 verifier 判定），修改它们会改变环境、作废已通过的 20 条真实纯视觉轨迹，却不会提升任何 Task 的判定质量。如需真实站点行为，建议另开
site-fidelity分支处理。/competitions无“按奖金排序”：路由仅sort=teams|deadline|newest（app.py），未暴露已有的reward_value字段。无 Task 依赖——Kaggle--16（最大现金奖）可通过读卡片完成。不改。/competitions的 “Completed” 筛选项恒为空：seed 的 17 场比赛deadline均在将来，is_active为真，故status=completed返回 0 条。无 Task 依赖；属筛选/数据不自治。不改。/competitions筛选下拉onchange即时生效，Apply仅服务文本框：UX 不一致，但键盘选择与 Apply 均可达目标。无 Task 依赖。不改。/discussions列表评论数与详情真实评论数不一致：列表显示反规范化字段comment_count（seed 展示数），详情显示真实Comment行，二者在 seed 中独立，故普遍不符、部分帖显示有评论但详情为空。verifier 以 DB delta 判定（Kaggle--10/11 不受影响）；属数据一致性瑕疵。如需修，列表改用d.comments.count()或对齐 seed。本 PR 不改。/learn的 “Start Course” 仅为页内锚点、lesson 行不可点：Learn 模块仅按 Task 覆盖深度实现到课程层（/learn、/learn/<slug>），无 lesson 路由/内容；唯一相关 Task Kaggle--14 只需课程 lesson 数。要“真”起来需新增 lesson 路由与内容，超出本 Review 范围。不改。其他低危 observation（同样不影响判分，记录备查）：
/user/<u>点 Follow 后页面粉丝数未即时刷新；/models的 DenseNet 卡复用 ResNet 缩略图；World Happiness 数据集封面素材与主题不符；首页 “Featured Competitions” rail 混入一张 Research 徽章卡；登录后个别封面/头像短暂空白（lazy-load，自恢复）。Scope
adapter_tasks.jsonl、本地 overlay。workdir兼容）暂留本地，如需上游化另开独立分支。Blocking checklist
.assets-revision指向 merge SHAReviews #54, #70.