Skip to content

Make sync-artifacts unable to half-publish a benchmark run - #8

Merged
EHxuban11 merged 1 commit into
rf100vl-harnessfrom
fix/sync-artifacts-cannot-half-publish
Aug 3, 2026
Merged

EHxuban11 merged 1 commit into
rf100vl-harnessfrom
fix/sync-artifacts-cannot-half-publish

Conversation

@EHxuban11

Copy link
Copy Markdown
Contributor

Follow-up to #7. That PR guarded wrong numbers; this one guards incomplete artifacts.

Three times in the 2026-08 campaigns the uploader printed uploaded N, skipped 0 while shipping something unreproducible. A crash is loud and stops you. This produced a green log, a real HF URL, and a submission with correct numbers — everything you'd check as a sanity test passed. The only way I caught it was enumerating the repo afterwards.

All of it traced to one line in collect_artifacts:

def add(path, repo_path):
    if path.is_file():
        items.append(...)      # else: silence

"not present" and "not applicable" were the same thing.

What

  1. Pass the resolved recipe_path. cmd_sync_artifacts computed a fallback for the manifest, then handed collect_artifacts args.recipe instead. A sync without --recipe wrote a manifest naming a recipe and uploaded none. This is the actual root cause of three published runs citing a recipe_sha256 whose file exists nowhere.
  2. Collect the flat <root>/<dataset>/<weight file> layout too. That is what --weights-root documents and what the evaluator reads. Only .runs was collected, so any other weights root uploaded zero checkpoints silently. The uploader and evaluator disagreed about where checkpoints live.
  3. Refuse to publish an incomplete checkpoints/all tier. Every dataset with a stats.json must ship a checkpoint; recipe and versions lock must be present. results only reports, so ordinary syncs stay non-brittle.
  4. Verify the recipe against the hash the submission claims. The fallback resolves the packaged family recipe, wrong for any run with a custom --recipe.

Why 4 matters most

Fixing 1 alone would make things worse. The fallback would start uploading recipes/rf100vl/yolox.json as provenance for runs that actually used a custom recipe — labelling a run with a recipe that did not produce it. Shipping the wrong recipe is worse than shipping none: it converts a visible gap into an invisible contradiction. 1 and 4 have to land together.

There's a matching asymmetry this closes: the harness already refuses to call a submission valid when the recipe hash is missing (that is how it caught a corrected run this morning), but happily published when the recipe file was missing. It validated the pointer and never checked the referent.

Check

tests/test_publish_completeness.py (new, 7 tests) written against the three real failures: flat-layout checkpoints are collected, .runs still works, a trained dataset without a checkpoint raises, a missing recipe raises, results tier only reports, a mismatched recipe raises, a matching one is accepted.

Full harness suite: 153 passed.

One existing test (test_collect_tiers_add_weights_progressively) needed a recipe fixture — it exercises tier layering with tier="checkpoints", which the contract now requires a recipe for. Behaviour under test is unchanged.

Not verified

  • The completeness contract keys off stats.json as "this dataset trained". A campaign that writes checkpoints without stats would not be checked.
  • IncompletePublish is a hard failure for programmatic callers of collect_artifacts at publishable tiers, not just the CLI. That is deliberate, but it is an API behaviour change.
  • Not fixed here: the campaign auto-sync path has its own collect_now() closure that passes neither eval_dir nor submissions_dir, which is why auto-sync runs carry no predictions or submissions. That looks intentional (the standalone sync is the publishing step) but it is worth confirming.

Three times in the 2026-08 campaigns the uploader printed "uploaded N,
skipped 0" while shipping an artifact that could not be reproduced. A
crash is loud and stops you; this produced a green log, a real HF URL,
and a submission with correct numbers. Everything you would check as a
sanity test passed.

All of it came from one line: collect_artifacts's add() dropped any path
that did not exist, so "not present" and "not applicable" were the same
thing.

Pass the resolved recipe_path to collect_artifacts. cmd_sync_artifacts
computed a fallback for the manifest and then handed collect_artifacts
args.recipe instead, so a sync without --recipe wrote a manifest naming a
recipe and uploaded no recipe. That is the whole reason three published
runs cite a recipe_sha256 whose file exists nowhere.

Collect checkpoints from the flat <root>/<dataset>/<weight file> layout
as well as .runs. That flat layout is what --weights-root documents and
what the evaluator itself reads; only .runs was collected, so a weights
root built any other way uploaded zero checkpoints in silence. The
uploader and the evaluator disagreed about where checkpoints live.

Refuse to publish an incomplete checkpoints/all tier. Every dataset with
a stats.json must ship a checkpoint, and the recipe and versions lock
must be present. The results tier only reports, so ordinary syncs stay
non-brittle.

Verify the recipe against the hash the submission claims. The fallback
resolves the PACKAGED family recipe, which is the wrong file for any run
that used a custom --recipe; uploading it would label the run with a
recipe that did not produce it. Shipping the wrong recipe is worse than
shipping none: it turns a visible gap into an invisible contradiction.

Claude-Session: https://claude.ai/code/session_01D8uB5n3e1zYjix18gT2Y4w
@EHxuban11
EHxuban11 merged commit e13b7c4 into rf100vl-harness Aug 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant