Skip to content

review(ted): complete PR #65 grading contract - #85

Merged
Raibows merged 17 commits into
aiming-lab:mainfrom
TabsPhasers:review/ted-pr65-remediation-public
Sep 7, 2026
Merged

review(ted): complete PR #65 grading contract#85
Raibows merged 17 commits into
aiming-lab:mainfrom
TabsPhasers:review/ted-pr65-remediation-public

Conversation

@TabsPhasers

@TabsPhasers TabsPhasers commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

This draft pull request is the reviewer-authored remediation package for WebHarbor PR #65. It completes and hardens the TED task grading contract that was reviewed against the remote PR head 065e39a4e1bf33e61902abdfb3e9d35f1e62428e.

The original PR remains unchanged. Its remote head contains 18 tasks and 18 verifiers; this draft adds the missing two high-difficulty comparison tasks and carries the fixes required by the Review Environment Skill. Maintainers can review this as a replacement/companion PR before deciding how to land the remediation.

What changed

  • Added TED--18 and TED--19, two filtered, multi-page view-count comparison tasks with dedicated deterministic verifiers and judge rubrics.
  • Re-anchored narrow tasks on visible TED searches with distractor results and explicit selection steps.
  • Tightened verifier contracts for exact filters, required detail/listing navigation, title/speaker/year constraints, non-empty final answers, and state deltas.
  • Kept all ground truth inside the reviewer-authored verifier code. sites/ted/tasks.jsonl has no answer key and contains only the permitted task and grading-contract fields.
  • Retained raw Browser Use trajectories, screenshots, and before/after SQLite snapshots in the local review archive. They are intentionally excluded from this public PR because they contain session-sensitive artifacts; public-safe aggregate reports and controls are included below.

Review outcome for PR #65

The correct status for the current remote PR head is REQUEST_CHANGES until equivalent changes are present on that PR. The remote head has the following blocking gaps:

  1. It has 18 rather than the complete 20-task contract.
  2. Several tasks omit a broad visible search path or use narrow queries, leaving first-result and distractor-quality risks.
  3. Task 11's verifier does not bind the requested TED2026 and duration filters.
  4. Task 3 accepts a speaker-only token for a talk-identification task.
  5. Task 16 does not require visiting a real /talks listing.
  6. Task 17 checks November without requiring the TEDNext 2025 event/year.
  7. Stateful verifiers do not consistently reject empty final answers.

Validation

  • 20/20 clean Luna Browser Use runs passed their deterministic verifiers.
  • 20/20 homepage-only, empty-answer no-op controls failed their verifiers (exit code 1).
  • 20/20 primary trajectory, screenshot, and database audits passed.
  • 20/20 sanitized blind Claude judge packets passed.
  • Full local environment smoke passed: control plane healthy, all 17 site ports returned HTTP 200, TED reset was ready, and instance/seed MD5s matched.
  • The pinned Hugging Face asset revision 597623a2f32898afa12e3bbeda15520f559aa7c7 was checked directly and returned HTTP 200.

The full-environment smoke used the existing local webharbor:ted-review image. A fresh source rebuild of the exact remote PR head was not established because unrelated local asset locks affected the earlier build; CI should remain the final reproducibility check.

Evidence

Review contract

Each task has one verifier under sites/ted/verify/verify_<N>.py and a corresponding judge_rubric in sites/ted/tasks.jsonl. The verifier is deterministic-first and checks navigation, visible task facts, final output, and SQLite state where applicable. The LLM judge is secondary; no-op failure and state-delta checks are retained as independent controls.

Checklist

  • 20 tasks and 20 dedicated verifiers
  • No answer key in the agent-facing task rows
  • No-op controls fail all verifiers
  • Clean positive-control matrix passes all verifiers
  • Browser screenshots and operation logs retained in the local review archive
  • Public PR excludes session-sensitive raw artifacts
  • No remote branch was modified by the original PR review
  • Maintainer review and merge decision pending

shanjiaming and others added 15 commits July 2, 2026 12:57
@shanjiaming
feat(ted): add TED mirror site
27eeb5b
@shanjiaming
fix(ted): hide durations in listings
ad303a9
@shanjiaming
fix(ted): address review feedback
6945aa5
@shanjiaming
fix(ted): improve card review fixes
9e6a161
@shanjiaming
fix(ted): tighten wine task data
e054cea
@Django-Jiang
Add TED task verifiers and fix task issues found in review
0a7c47a 
Based on aiming-lab#1 (TED mirror by @shanjiaming). Adds the reviewer grading
contract and fixes task-definition issues surfaced in review.

Verifiers (sites/ted/verify/): verify_lib.py + verify_0..17.py —
deterministic-first (nav-evidence + answer match + DB after-state),
LLM utilities anchored on frozen ground truth. verifier_path +
judge_rubric recorded in tasks.jsonl. Validated: no-op FAILs 18/18,
correct solve PASSes 18/18, wrong answers/actions FAIL.

Task fixes (sites/ted/tasks.jsonl):
- Add demo credentials to account-bound tasks (1/4/7/9/12).
- De-shortcut prior-knowledge-answerable tasks: TED--5 -> exact title;
  TED--17 -> event month (Nov 2025) instead of city (Atlanta).
- TED--7: retarget from TED Countdown Summit 2025 (Alice is
  seed-registered -> no-op) to TED2026.
@Django-Jiang
fix(ted): tighten verifier nav + integer checks
065e39a 
- verify_11: require /talks? (filter listing) so opening a /talks/<slug>
  detail page alone no longer satisfies the 'used filters' check.
- extract_ints: word-boundary match so a duration check can't be
  satisfied by an incidental integer (e.g. '15k' views).
@Django-Jiang
Merge 065e39a into 7269be8
35b1ba3
manager_of_pre_star_li
fix(ted): require final answers in stateful verifiers
22c769f
manager_of_pre_star_li
feat(ted): add two filtered comparison tasks
20d1b12
manager_of_pre_star_li
fix(ted): tighten task quality and verifier semantics
25d4932
manager_of_pre_star_li
fix(ted): accept recorded after URLs in verifier navigation
3e2ce44
manager_of_pre_star_li
fix(ted): normalize comma-separated view differences
c94cc25
manager_of_pre_star_li
docs(ted): add sanitized review evidence package
3fdc86d
manager_of_pre_star_li
chore: align TED remediation with current main
18fa53a
@TabsPhasers
TabsPhasers force-pushed the review/ted-pr65-remediation-public branch from 2b25f0d to 18fa53a Compare September 6, 2026 14:12
@Raibows
Raibows marked this pull request as ready for review September 7, 2026 14:56
@Raibows

Raibows commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for your contribution! @TabsPhasers

@Raibows
Raibows merged commit e911a6a into aiming-lab:main Sep 7, 2026
Raibows added a commit to TabsPhasers/WebHarbor that referenced this pull request Sep 14, 2026
@Raibows
fix(nvidia): pin assets at the HF revision that carries the distinct …
3c0d430 
…official images

HF dataset PR aiming-lab#85 ("Upload nvidia.tar.gz with distinct official product and hero
images (aiming-lab#107 follow-up)") is merged, so the pin moves from
a9d3ea79513fa36a51498129cb9d031e4cc096cc to b7e605c0ec5fc47de85b09e7427162cc50e38980,
the merged revision's squash commit on the dataset main.

Refreshed facts (all re-measured this round):
- nvidia.tar.gz at the new pin: 37 file members, 0 directory members,
  16,340,955 bytes, sha256
  617a3e3740ba6706bcab786c8a5c3f9a22ecbb39eff5728ad2c12e4992cb098b,
  and scripts/validate_asset_archive.py prints
  "[fetch] validated 37 managed members for nvidia" (exit 0);
- it carries 36 images with 36 distinct sha256 (33 product/news images plus three
  dedicated hero images under static/images/heroes/), so no byte-identical
  duplicate remains and no page reuses one file twice;
- instance_seed/nvidia.db is byte-identical to the previous pin
  (2143c954def96cc921760ab2bea79fe119de3d73212d1b01daf6c61792c2b38d).

.assets-revision states the new revision and the new archive identity, keeping the
superseded PR aiming-lab#84 archive as history. README.md's asset-delivery section now names
the merged PR aiming-lab#85 revision and lists the new archive first with the previous pin's
archive as the superseded row.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@TabsPhasers @Raibows @shanjiaming @Django-Jiang