Return verification v3: split-sealed data, one-time test opening, reload tolerance, fusion recompute

[Jeffreyliu0131]

Return verification line of the model-reception wave (task book 143, line 147). The data/model owner returns dev predictions, a single pass of test predictions after freeze, a fusion selection record, and a G2v dataset release. Until now verify-return accepted only dev, refused anything containing test, and validate-handoff read every partition. This line lets us verify all of these returns independently with offline CLIs, in three new v3 formats. Test stays sealed until a freeze record and an approved TestCampaign exist, and then opens exactly once. Per the 2026-10-12 decision, the user confirms each test opening in chat and the CLI has no bypass flag. Learners never see test, and there is no UI change.

What changes

• Split-sealed dataset rolecraft-dataset-split-sealed-v1 (spec docs/data/return-v3/dataset-split-sealed-v1.md):
  • The manifest gains layout, test_sealed and label_orders.
  • Each split has its own data/<s>.inputs.jsonl, data/<s>.labels.jsonl and audit/<s>/{annotations,metadata}.jsonl, sha256-bound in splits.<s>.files.
  • Reading train/dev opens no test file, which a subprocess audit hook proves.
  • Symlinked partition files → dataset_symlink_forbidden; byte-identical files across splits → dataset_split_file_not_isolated.
• Prediction rows v3 (spec predictions-v3.md): return_schema_version=3, task_type=relation. v2 rows are still accepted. Unknown task_type or a missing version is refused per row. validate-handoff accepts v3 rows too.
• Reload comparison:
  • Absolute probability tolerance: linear 1e-6, encoder 1e-4. Labels, status and evidence sets must match exactly.
  • The report gives tolerance (null when not reloaded), max_probability_deviation and mismatches; comparison is now probabilities_within_absolute_tolerance_labels_and_evidence_sets_exact.
  • Per-language counts{format_failures, abstentions, coverage}, with evaluation_setting=given_evidence_only. A missing language → blocked.
• verify-return --fusion-selection recomputes the whole dev grid under joint-first-dev-v1. The result is rejected or refused for: a different selected point, an endpoint gain claim, an alpha that disagrees with the registration, a wrong gain flag, a component mismatch, an invalid grid or protocol, or the test partition.
• One-time test opening (spec test-opening-v1.md):
  • Invocation: --partition test --freeze --freeze-hash --campaign --campaign-hash --operator. Freeze and campaign must share a root (HELD-07).
  • The record is exclusive-created at <freeze dir>/held-out/<dataset manifest sha256>/test.json. It holds campaign, candidate, freeze and campaign sha, registration, verifier digest, operator and time.
  • There is no --ledger; the location depends on neither the command line nor the environment.
  • Invariants:
    • A second opening → held_out_test_already_opened (403) with zero test reads, including after changing directory or environment, going through a symlink, or copying the opened freeze dir.
    • Approval arguments missing → held_out_test_authorization_required (403).
    • A corrupt campaign → held_out_campaign_invalid.
    • A linked or non-directory record location → held_out_record_location_invalid.
  • Pre-checkable errors are refused before the record is written, reading only test metadata: a revision that disagrees with the manifest or the registration, record_id/input_hash not the sealed records, metadata not split=test, a registration that cannot load on this machine. Errors after that point carry opened. A failed verification keeps the record and does not retry.
• validate-handoff:
  • Sealed mode checks only test bytes, line counts and structural isolation, and never parses test labels (labels_parsed=false).
  • It reports layout, sealed_test and per-language annotation_trace.
  • It accepts the v3 outer-manifest fields artifact_kind, artifact_protocol and label_tiers. Linear returns get fixed roles and need no artifact_roles.
  • --split test with a return manifest is refused on a sealed package.
• Untrained encoder heads:
  • Accepted only for a synthetic_fixture registration with weights_stage=pretrained_untrained_heads, checked by 146's validate_encoder_return_identity.
  • Without the encoder extra, only a synthetic untrained-head registration gets a dev reload of blocked_encoder_runtime_unavailable (neither pass nor fail). Any other encoder registration, external or trained, still exits 2 with pretrained_encoder_dependencies_unavailable. predictions-v3.md §3: exit 0 alone does not mean reloaded; only reload.status=matched does.
  • Test opening loads the registration before writing the record.
• No contract re-export; freeze.py, TestCampaign and 144 files unchanged. The new package status SYNTHETIC_MECHANISM_SAMPLE_NOT_TRAINING_DATA marks our mechanism samples.

Review and verification

• Independent review 080-review-147.md (154): first review plus two re-checks, three rounds in all; final head 1e2b85c. First round, on b797c95, sent back 1 P1 and 2 P2:
  • P1: the same approval reopened test from a new --ledger directory.
  • P2: deleting the record allowed a reopen; pre-checkable mismatches still used up the opening.
• First re-check, on 4698205: 0 P1 and 2 P2.
  • P2-a: after merging 146, the untrained-head path refused 146's real exported manifest with model_return_weights_stage_invalid.
  • P2-b: the docs overstated copy protection; copying the freeze dir before opening allows two openings.
  • Both were fixed in 5c2ea93: the path now delegates to 146's check and has a CLI-level case on a real registration, and the operating rules are documented.
  • In this round the reviewer ran subprocess bypass attempts with an audit hook on open. Each of these was refused with zero test reads: changing directory or environment, using symlinks, passing --ledger, adding a second campaign, editing the campaign, copying after opening. 6 concurrent first openings produced exactly 1 success. Deleting the record in place still reopens; this counts as tampering.
• Second re-check, on bbfe7ea: 0 P1 and 1 P2. With the encoder extra missing, even a trained external encoder return exited 0 with top-level partial, which looks the same as a matched reload. Last fix, 1e2b85c: the dependency-missing path is lenient only for synthetic untrained-head registrations, and external encoders still exit 2 (RET-05 +2). Test opening was already safe here: an unloadable registration is refused before the record is written.
• 119 new test IDs, all passing: CRV-03 2, DATA-06 15, FSEL-01 15, HELD-01 10, HELD-02 13, HELD-03 5, HELD-04 3, HELD-05 14, HELD-06 2, HELD-07 1, HELD-08 6, PRED-01 8, RET-03 10, RET-05 2, SEAL-01 9, TOL-01 4. Most were written test-first. For HELD-02/03/05/06/07/08 the reds came from mutations (8/5/3/2/1/4 red), with the diff recorded in red/*.mutation.log.
• Line checks:
  • Targeted: 10 files, 433 passed on 1e2b85c, including 146's encoder registration file.
  • Per-ID: 230 IDs against eb494c0 and 233 against main 95d4883, with 0 missing and 0 regressed in both.
  • lint: 0 new.
• Cross-check with 145 23fb9df (temporary merge): 4 test files, 215 passed. A real fusion registration recomputes as matched, and an altered point is rejected.
• CLI evidence rerun on the final tree, on a zh/en synthetic v3 package:
  • First opening: exit 0, held_out_test_opened=true.
  • Second opening: exit 2, already_opened.
  • Copied freeze dir with a different operator: exit 2.
• First-delivery gate on b797c95 passed (history only). Full gate ran once on the integration tree 11e53dd7… (146 b765c88 → 145 23fb9df → 147 1e2b85c → 148 261118e → 149 2e56be1, integration branch HEAD 855b366; 157 receipt).

Not claimed / registered

• Not built, by the 10-12 scope cut (task book §8): criterion verification and five-class metrics (CRV-01/02), and the Judge training-format export (JFMT-01/02).
• No real owner test opening has happened. Each one needs the user's confirmation in chat; the coordinator prepares the freeze, writes the campaign, runs it, and commits the freeze dir with campaign and record to git.
• Copying the freeze dir before opening cannot be blocked by mechanism. The operating rule is one copy, open only there, commit immediately. Deleting the record counts as tampering, and git history shows it.
• Encoder 1e-4 tolerance not run with --extra encoder; NumPy attention was used instead.
• The fusion_selection_not_registered code-name test waits for 145 to merge; the 145 cross-check did trigger it.
• Near-duplicate screening between test and train/dev not done, because it would require reading test inputs.
• P3 not done:
  • The ledger is keyed by manifest sha; resealing the same test bytes needs a new freeze and campaign, which means new user approval.
  • The opening record has no synthetic marker.
• Registered deviations: review.py gained model_kinds and test admission; DATA-06 numbering is shared with the v3 manifest cases; the comparison value change goes into the 150 manual.
• The three specs go into the v3 package and SHA256SUMS (150).

Receipt: 完成回执/claude-coordination/147-回传复核与数据格式v3.md.