Skip to content

Add Kaggle mirror + task verifiers (site by @KaKituken, verifiers by reviewer) - #70

Open
Django-Jiang wants to merge 1 commit into
aiming-lab:mainfrom
Django-Jiang:review/kaggle-verifiers
Open

Django-Jiang wants to merge 1 commit into
aiming-lab:mainfrom
Django-Jiang:review/kaggle-verifiers

Conversation

@Django-Jiang

Copy link
Copy Markdown
Contributor

Review: Kaggle mirror (PR #54) — reviewer verifiers + grading contract

Reviewed PR #54, rebased the site onto current main, built the 17-site image
webharbor:kaggle-rebase, ran on alt ports. Kaggle is registered at index 16 (port 40016).

Mechanical checks: PASS

  • All 16 sites return 200 (41400-41415)
  • Control plane healthy (_health → 17 competitions, 16 datasets, ok)
  • Byte-identical reset: md5 = 131cdab6ed840e4e74324ab5bd24dda4 (instance == instance_seed), stable after full task-driving + resets
  • POST /reset-all → all sites ready
  • Assets pinned to refs/pr/37 (HF discussions/37); fetch_assets.sh succeeds. Maintainer must bump to the merged SHA once the HF PR lands.

Visual / functional depth: PASS (driven via real Playwright form/JS actions, confirmed against the DB)

Login (all 4 benchmark users, TestPass123!); join competition (+ team name); upvote; bookmark;
follow; new discussion; post comment; profile-location edit; dataset download — all persist to SQLite.

Task quality: 1 verifier-side fix, otherwise sound

  1. Kaggle--11 was pre-satisfied by the seed (fixed in the verifier). seed_discussions already
    seeds one davidtran comment on the titanic-feature-ideas thread, so an existence check
    couldn't confirm the agent's new comment. The verifier now uses a delta (davidtran comment
    count on that thread must increase vs the seed). No task-text change needed.

Minor (non-blocking): Kaggle--0 ("Titanic passenger survival") — two Getting-Started Titanic
competitions exist (titanic-survival, spaceship-titanic-rescue), but both use metric
"Classification Accuracy", so the answer is unambiguous; the verifier requires navigating the
titanic-survival page. Several read-only answers (accuracy, ROC AUC, Apache 2.0, CC0, 7 lessons)
are frontier-LLM-guessable; the anti-shortcut nav gate (must open the specific competition/model/
course/dataset page) mitigates this, and site-specific tasks (team names, host schedules,
carolwong's notebook, rankings) anchor the harder ones.

Rebased onto current main in this PR

PR #54 branched from a 16-site main and registered kaggle at index 15 (a slot now taken by
merriam_webster). This reviewer PR rebases the site onto current main:

  • kaggle re-registered as index 16 / port 40016 in websyn_start.sh, control_server.py, and
    Dockerfile (EXPOSE 40000-40016).
  • .assets-revision pinned to refs/pr/37 (HF discussions/37), which contains all 16 current-main
    site tarballs plus kaggle.tar.gz — so this PR builds end-to-end. Maintainer: after HF discussions/37
    is rebased onto HF main and merged, bump .assets-revision to the merged SHA.
  • The full 17-site image builds; all 17 sites return 200; kaggle serves at 40016; byte-identical reset
    holds; the verifiers were re-confirmed against the rebased container (read-only PASS, no-op FAIL,
    and a stateful drive+PASS at the new port). The current-main agent_demo provides the
    eval_judge --verifier True PRIMARY-grader path.

Grading contract authored (reviewer)

  • sites/kaggle/verify/verify_lib.py (shared deterministic + anchored-LLM utilities) + 20 per-task
    verifiers verify_0.py … verify_19.py.
  • tasks.jsonl backfilled with verifier_path + judge_rubric per row; no answer key.
  • Deterministic-first: navigation evidence (anti knowledge-shortcut) + on-page answer match +
    SQLite after-state (competition_entries, votes, bookmarks, follows, discussions/comments, download
    counter, user profile). LLM/screenshot checks anchored on ground truth and SKIP when the LLM is
    unavailable.

Grading validation (verifiers run standalone against the live container DB)

  • No-op (homepage only, empty answer, DB = seed): all 20 verifiers FAIL. Zero false positives.
  • Genuine completion (real form/JS-driven actions + correct read-only answers): all 20 PASS.
  • Wrong answer (read-only naming the wrong metric/license/user/team): FAIL.
  • Shortcut (correct answer, no detail-page nav): FAIL on the nav gate.
  • State-mismatch (correct nav, DB reverted to seed): FAIL on the DB gate.
  • Byte-identical reset holds after the full drive (md5 unchanged).

Verdict

Grading contract is ready and validated; the site is rebased onto current main (kaggle at index
16 / port 40016) and builds from refs/pr/37. Ready to merge once the maintainer bumps
.assets-revision to the merged HF SHA after HF discussions/37 lands on HF main.

@TabsPhasers

Copy link
Copy Markdown
Contributor

后续 Review PR 已创建:#106https://github.com/aiming-lab/WebHarbor/pull/106)。

该 PR 保留 @KaKituken 的站点贡献(#54)并延续本 PR 的 verifier/rubric 工作,基于当前 main(3600493)接管,Kaggle 注册为 index 23 / 端口 40024。剩余 blocker:HF ChilleD/WebHarbor #74 合并后 pin .assets-revision 到 merge SHA 并 clean-fetch 复验。当前为 Draft,未满足合并条件。后续 Review 请转到 #106

JeremyJC67 added a commit to JeremyJC67/WebHarbor that referenced this pull request Sep 13, 2026
No merged asset revision contains healthline.tar.gz yet, so main's pin cannot fetch this site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
JeremyJC67 added a commit to JeremyJC67/WebHarbor that referenced this pull request Sep 13, 2026
HF PR aiming-lab#70 merged on 2026-09-13, so .assets-revision moves off the temporary refs/pr/70
ref and onto the immutable merge commit. That revision carries every tarball the 27-site
build needs, healthline.tar.gz included.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
Raibows added a commit to TabsPhasers/WebHarbor that referenced this pull request Sep 13, 2026
…ree matches its own Dockerfile and registry

Conflict resolutions:
- Dockerfile: keep '<N> Flask mirror sites' as 25 and EXPOSE 8100+healthline at index 24
  (40000-40024); main's per-site build steps for ikea/phys_org/target/ted/osu/
  rotten_tomatoes/compass/walmart_careers are retained.
- control_server.py / websyn_start.sh: keep the 25-entry SITES list with 'healthline'
  appended (index 24 -> container port 40024).
- .assets-revision: keep the immutable sha of HF PR aiming-lab#70 (superset of the dataset main
  branch: 28 tarballs including healthline.tar.gz).
Raibows pushed a commit that referenced this pull request Sep 13, 2026
…reviewer)

Rebased onto current main (24 sites); kaggle appended as index 23 / port 40024. Adds 20 tasks with per-task deterministic verifiers and judge rubrics; no answer key. Preserves #54 by @KaKituken and continues @Django-Jiang's #70.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants