Review: Add Healthline mirror + task verifiers (site by @JeremyJC67, verifiers by reviewer) (#59) - #105
Conversation
Addresses review on PR aiming-lab#53: - Rebase onto current main; register healthline as index 16 -> port 40016 (append after merriam_webster in websyn_start.sh + control_server.py; Dockerfile EXPOSE -> 40000-40016). merriam_webster preserved. - websyn_start.sh site-count comments reconciled to 17. - .assets-revision pinned to HF PR aiming-lab#40 (clean tarball, no macOS AppleDouble junk, based on current main -> all 17 tarballs). Bump to merged SHA once HF PR aiming-lab#40 lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer deliverable for the Healthline mirror (site by @JeremyJC67, PR aiming-lab#53): one deterministic verifier per task under sites/healthline/verify/, plus verifier_path + judge_rubric recorded in every tasks.jsonl row. No answer key in tasks.jsonl — ground truth lives only inside the verifiers. Deterministic-first: (1) trajectory navigation gate (anti knowledge-shortcut, important here since several answers are medically recallable), (2) SQLite DB after-state for the stateful tasks (save / register / password-change, plus DB cross-checks for saved-count and reading-history), (3) answer vs frozen ground truth, with the LLM only as an anchored consistency check. Validated against the official react agent (agent_demo/agent.py): a no-op run fails all 20 verifiers; a human answer-check confirms every frozen value matches what the page renders. On the full 20-task run, after fixing one too-strict nav gate the run itself surfaced, verifier and LLM judge agree 15/20, and all 5 remaining divergences are the deterministic verifier being correct while the LLM judge false-positives (blank answer / knowledge-shortcut) or false-negatives (DB-confirmed save / password change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Search result cards showed drug_class (e.g. 'ACE inhibitor'), letting T16 be answered from the results page without opening the drug pages. Show the broader category (e.g. 'Heart Medications') instead — same card layout, but the ACE-inhibitor-vs-statin distinction now requires opening each detail page. drug_class still shown on the drug detail page; search backend still matches on drug_class. Addresses DEM1TASSE review note on PR aiming-lab#53. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- web: all 20 tasks pointed at :40015 (Merriam-Webster) after the site was rebased to index 16 / :40016; corrected to http://localhost:40016/. - judge_rubric for Healthline--3 and Healthline--16: spell out the full ground-truth answer the page states (T3 symptom list, T16 what each drug treats) so the LLM judge grades against complete facts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…te on current registry
…the mutable refs/pr/40 ref
…ree matches its own Dockerfile and registry Conflict resolutions: - Dockerfile: keep '<N> Flask mirror sites' as 25 and EXPOSE 8100+healthline at index 24 (40000-40024); main's per-site build steps for ikea/phys_org/target/ted/osu/ rotten_tomatoes/compass/walmart_careers are retained. - control_server.py / websyn_start.sh: keep the 25-entry SITES list with 'healthline' appended (index 24 -> container port 40024). - .assets-revision: keep the immutable sha of HF PR aiming-lab#70 (superset of the dataset main branch: 28 tarballs including healthline.tar.gz).
…m per-process value
…tead of int() on raw session data
…ve view tracking to an explicit POST
… related-drug list
…ot sign the user out
…pe values with 400
…lying on framework defaults
… phones have no horizontal overflow
…ync the seed DB via migrate_seed.py
…ory-root scraped_data rule
…tent and never crash on missing inputs
… databases to be unchanged
…ful payload verification
|
@TabsPhasers thanks for picking this up. A heads-up so you don't redo work that is already This branch is based on #59's July head (ports 40015/40016). The site PR #53 was rebased onto It has since had a second pass against the maintainer merge-blocker checklist, verified on a full
On assets: HF #40 is closed. It was superseded by Three of your verifier predicates are things I do not have, and they are good: So rather than two vehicles for one site, either of these works and both credit your review:
Tell me which you prefer and I will do my side of it today. One small correction for the record: the Scope section credits @DEM1TASSE as the original author. |
|
Thanks — we'll take option 2: #53 stays the merge vehicle. We're closing #105 as superseded; the branch remains at Requested cherry-pick with authorship preserved — 8 predicate deltas, not just the three helpers you named (we diffed our tree against #53's head
Two things for the record:
On assets we're aligned: HF #40 closed → #70; If a direct cherry-pick conflicts, we're happy to push a small patch branch on top of #53's head instead — just say which you prefer. (This reply was posted automatically via OpenCode.) |
…ewer Ports the eight predicate deltas @TabsPhasers requested on aiming-lab#105, with their authorship. Applied surgically onto this branch's hardened verify_lib rather than cherry-picking their commit, because their tree predates the evidence-binding and read-only-state work here. 1. count_groups + verify_3: overlapping symptom tokens count once 2. amount_with_unit + verify_11: a standalone 50 mg with a unit, not a substring 3. verify_16: reject swapping the two drug classes 4. verify_6: fail closed when the initial DB is unavailable 5. verify_8: require the reviewer's name, not just the credential 6. verify_15: broadened affirm/negate pair 7. verify_17: require the condition identity 8. verify_19: same grouping fix for migraine triggers number_mentioned was not taken: this branch's anchoring is equivalent, by their assessment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…thline to index 25 (port 40025)
…thline to index 26 (port 40026)
…h_assets.sh (cherry picked from commit c944482a7e91d3dfedd6e65f2baf4eadf7a34cf7)
be65778 to
431e8b0
Compare
|
Thanks for your contribution! @TabsPhasers @JeremyJC67 @DEM1TASSE |
Upstream advanced to a437bd4, merging Healthline (aiming-lab#105), which took index 26 and port 40026 -- the slot this branch was using. Resolved by taking upstream's registry verbatim and appending versus after healthline: healthline keeps 40026, versus takes 40027. No earlier site moves. Dockerfile site count and EXPOSE range follow the 28-site registry, and the task rows and site README follow the slot. The regression test derives the port from both launchers rather than freezing it, so it follows this move on its own; so does the verifiers' origin check. Recorded trajectories carry the old port, so the runs are re-recorded at 40027 rather than graded with an origin override.
|
@Raibows @TabsPhasers — congratulations on landing this, and one follow-up finding on the merged Two unvalidated redirects are live in Both follow an attacker-supplied absolute URL. The fix that was on #53 is small and self-contained: def safe_redirect_target(target, fallback):
"""Only follow same-origin relative paths; anything else falls back."""
if not target:
return fallback
parsed = urlparse(target)
if parsed.scheme or parsed.netloc or not target.startswith("/") or target.startswith("//"):
return fallback
return targetwith the two call sites becoming Say the word and I will open a one-file PR against |
… into the UC Berkeley branch Registry: append `berkeley` after `kaggle`, so Healthline keeps index 26 / port 40026 and Kaggle keeps index 27 / port 40027; UC Berkeley moves to index 28 / port 40028 (29 sites, 40000-40028). Conflict resolutions (append rule): - `websyn_start.sh`, `control_server.py`: keep all of upstream's 28 entries in order and append `berkeley` last (both registries identical, 29 entries). - `Dockerfile`: keep every upstream site block (Healthline's migrate + prune step included) and berkeley's build-generated seed block; header 28 -> 29 sites; `EXPOSE 8101 40000-40027` -> `40000-40028`. - `README.md`, `AGENTS.md`, `CONTRIBUTING.md`, `CLAUDE.md`, `agent_demo/README.md`, `.claude/skills/*`: take upstream's text, then 29 sites, port range 40000-40028, alt ports 41000-41028, and `UC Berkeley` appended to README's mirror list. - `scripts/fetch_assets.sh`, `scripts/check_assets.sh`: take upstream's registry-scoped implementations and keep berkeley's `.build-generated-seed` exemption on top. Follow-on work required by the port move: - `sites/berkeley/tasks.jsonl` (22 `web` rows), `app.py` PORT default, `README.md`, `verify/TASK_REVIEW.md`, `verify/verify_lib.py` comment, `tests/test_integration.py` (SITE_INDEX 28 / SITE_PORT 40028) and `verify/tests/test_tasks_contract.py`: port 40028. - `scripts/check_site_registry.py` (upstream's new gate) reports 29 sites consistent across both registries, `Dockerfile EXPOSE` and every `tasks.jsonl` port. Verified after the merge: registry gate green; berkeley site 27, berkeley verify 534, healthline 27; walmart_careers 50 passed and rotten_tomatoes 55 passed (+4129 subtests) each with one expected red that is pre-existing on upstream/main and reproduced there from a pristine `git archive` tree (VERIFICATION.md §re-slot); kaggle ships no pytest suite. Container rebuilt and re-run on 44000-44028: 29/29 sites 200, /health 29 alive+ready, `POST /reset/berkeley` byte-identical (`f2f0187c...`) before and after `docker restart`, no-op matrix 22/22 FAIL with 0 infra errors. Co-Authored-By: Claude Code <noreply@anthropic.com>
Summary
基于当前
main接管 @DEM1TASSE 的 #59(Healthline 镜像 + task verifiers)。本 PR 为 Draft 占位版本,用于承载 reviewer 的修复与验证结果。Original review resolution
main的 40016 已归ikea;改为追加式分配 40024,并保留 main 全部站点(25 站)。.assets-revision:素材 HF PR Add AKC mirror site #40 未合并、原 pin 为refs/pr/40;本轮不改 pin(保留 main 的稳定 revision),待 HF Add AKC mirror site #40 合并后 pin 到 merge SHA。Review changes
sites/healthline/verify/*+verify_lib.py:数值边界匹配、否定/肯定区分、药物-类别关联、DB 缺失 fail-close、导航门收紧、分组计数。sites/healthline/tasks.jsonl:web 端口40016 → 40024。Dockerfile/control_server.py/websyn_start.sh:保留 main 站点列表并追加healthline(端口 40024)。Validation
--no_llm):20/20 exit 0。eval_judge.py语义,轨迹 + 末 4 帧):20/20 success,无 false checkpoint。Asset delivery status
.assets-revision:保持 main 的 revision(本 PR 未修改)。在该 PR 合并并 pin merge SHA、clean-fetch 通过前,本 GitHub PR 不得合并。Scope
Blocking checklist
main并解决冲突,把 diff 收敛为站点 + 必要集成改动.assets-revisionpin merge SHA → clean checkout 执行fetch_assets.sh healthline+ build/reset 验证Reviews #59.