Add Healthline mirror + task verifiers (site by @JeremyJC67, verifiers by reviewer) - #59
Conversation
Addresses review on PR aiming-lab#53: - Rebase onto current main; register healthline as index 16 -> port 40016 (append after merriam_webster in websyn_start.sh + control_server.py; Dockerfile EXPOSE -> 40000-40016). merriam_webster preserved. - websyn_start.sh site-count comments reconciled to 17. - .assets-revision pinned to HF PR aiming-lab#40 (clean tarball, no macOS AppleDouble junk, based on current main -> all 17 tarballs). Bump to merged SHA once HF PR aiming-lab#40 lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer deliverable for the Healthline mirror (site by @JeremyJC67, PR aiming-lab#53): one deterministic verifier per task under sites/healthline/verify/, plus verifier_path + judge_rubric recorded in every tasks.jsonl row. No answer key in tasks.jsonl — ground truth lives only inside the verifiers. Deterministic-first: (1) trajectory navigation gate (anti knowledge-shortcut, important here since several answers are medically recallable), (2) SQLite DB after-state for the stateful tasks (save / register / password-change, plus DB cross-checks for saved-count and reading-history), (3) answer vs frozen ground truth, with the LLM only as an anchored consistency check. Validated against the official react agent (agent_demo/agent.py): a no-op run fails all 20 verifiers; a human answer-check confirms every frozen value matches what the page renders. On the full 20-task run, after fixing one too-strict nav gate the run itself surfaced, verifier and LLM judge agree 15/20, and all 5 remaining divergences are the deterministic verifier being correct while the LLM judge false-positives (blank answer / knowledge-shortcut) or false-negatives (DB-confirmed save / password change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Search result cards showed drug_class (e.g. 'ACE inhibitor'), letting T16 be answered from the results page without opening the drug pages. Show the broader category (e.g. 'Heart Medications') instead — same card layout, but the ACE-inhibitor-vs-statin distinction now requires opening each detail page. drug_class still shown on the drug detail page; search backend still matches on drug_class. Addresses DEM1TASSE review note on PR aiming-lab#53. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- web: all 20 tasks pointed at :40015 (Merriam-Webster) after the site was rebased to index 16 / :40016; corrected to http://localhost:40016/. - judge_rubric for Healthline--3 and Healthline--16: spell out the full ground-truth answer the page states (T3 symptom list, T16 what each drug treats) so the LLM judge grades against complete facts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@DEM1TASSE thanks again for the verifier suite. Since Nothing from this PR is lost — the merge vehicle is just #53 now. If you'd rather keep #59 as the vehicle, rebasing it onto |
|
🔒 Review lock (backfilled): this review is being carried out by @TabsPhasers. The lock was not posted before work started, so this is an honest backfilled registration rather than a claim of a prior lock. I verify this contribution against the reviewer contract and the existing review points, run the standard environment/reset/task checks, and track an independent Review PR here. Follow-up Review PR: #105 (#105). It is a Draft that preserves @DEM1TASSE's contribution and ownership, takes over the original contribution, and tracks the remaining blockers (rebase onto current |
Review: Add Healthline mirror + task verifiers (site by @JeremyJC67, verifiers by reviewer) (#59)
4 participants
Brings the Healthline mirror + its task verifiers into
mainas one PR, following the samefork-of-fork pattern as CarMax (#57) and NVIDIA (#58). Companion to the site PR #53.
Authorship is preserved — both contributors stay credited. This branch is @JeremyJC67's
Healthline site commits with the reviewer's verifier commit on top; each commit keeps its
original author (site = @JeremyJC67, verifiers = the reviewer). Please merge with a regular
merge (not squash) so both signatures are kept.
Contents
sites/healthline/site (Flask app, templates, seed, 20 tasks) — @JeremyJC67 (from feat(healthline): add Healthline mirror + task verifiers (site 25, port 40024) #53)sites/healthline/verify/verify_<0..19>.py+verify_lib.py, andverifier_path+judge_rubricin everytasks.jsonlrow — reviewer deliverable. Noanswerkey; groundtruth lives only inside the verifiers.
Why this is close to mergeable
Unlike the other verifier PRs, #53 was already rebased onto current
mainby the author:Healthline is registered at index 16 / port 40016 with Merriam-Webster preserved at 40015, so
this PR does not conflict. I independently re-verified: 17/17 sites return 200, byte-identical
reset (
a8a9b54…) matches the seed and the author's reported hash, and reset-all runs in ~1 s.Verifier design & validation
Deterministic-first per task: navigation gate (anti knowledge-shortcut — important since several
medical facts are LLM-recallable) + SQLite DB after-state for the stateful tasks (save / register /
password-change) + answer vs frozen ground truth, LLM only as an anchored check. Validated with
the official react agent: no-op fails all 20; a human answer-check confirms every frozen value
matches the rendered page. On the full 20-task run, after fixing one too-strict nav gate the run
surfaced, verifier and LLM judge agree 15/20, and all 5 remaining divergences are the verifier
being correct while the LLM judge false-positives (blank answer / knowledge-shortcut) or
false-negatives (DB-confirmed writes).
One dependency
.assets-revisionpoints at HF discussion #40 (the author's clean asset PR). Merge that HF PR,then repin
.assets-revisionto its merged SHA. A full review is posted as a comment on #53.