Skip to content

Encoder return v3: public spec and exporter, pair_aggregate, version lock, synthetic untrained heads - #41

Merged
Jeffreyliu0131 merged 15 commits into
mainfrom
claude/146-encoder-return-v3
Oct 12, 2026
Merged

Jeffreyliu0131 merged 15 commits into
mainfrom
claude/146-encoder-return-v3

Conversation

@Jeffreyliu0131

Copy link
Copy Markdown
Collaborator

Encoder line of the model-reception wave (task book 143, line 146; merges first after 144). The data/model owner returns a fine-tuned XLM-R with a label head and an evidence-selection head, its checkpoint, tokenizer and config, and per-row dev predictions. Until now the layout we accept was our private save_checkpoint format, written only in the repo manual. This line publishes it as a v3 format the owner can follow without our code, adds the "judge each pair, then aggregate deterministically" structure, puts threshold and token limit in the manifest, and locks inference dependency versions. The product registers the return locally and serves it as R2.support relation advice; the zh/en feedback page shows the registration under "机制验证 / Mechanism validation". Per the 2026-10-12 decisions, a pretrained encoder with untrained heads may only be registered as a synthetic mechanism sample, and the criterion head is out of scope.

What changes

  • Public spec docs/data/return-v3/encoder-return.md: directory and file rules, outer manifest, checkpoint.json v3 fields, heads.npz (names, shapes, float32), encoder input text rolecraft-encoder-text-v1, the three variants, after_weights, the model_revision formula with a worked example, weights stages, dependency lock ("registration succeeding does not mean it loads"), and every registration-time and load-time refusal code (§13).
  • Reference exporter docs/data/return-v3/reference/export_hf_return.py (torch, transformers, numpy, safetensors; does not import this repo): export writes the v3 layout and prints model_revision, seal writes the outer manifest. It refuses non-safetensors weights, head keys that do not match, reports that disagree with the weights, and pair_aggregate thresholds below 0.5.
  • Variants pack / pair / pair_aggregate. pair_aggregate rule: each pair gives (s, c, u); τ must be ≥ 0.5; any c ≥ τ → CONTRADICTED; otherwise any pair with s ≥ τ and s > u → SUPPORTED; otherwise INSUFFICIENT. The decided class gets (1+m)/2, the others (1−m)/4. 13 spec examples (11 decisions, 2 refused thresholds) in encoder-pair-aggregate.examples.json, checked against both product and script.
  • Threshold and limit in the manifest: evidence_threshold (0<τ<1; pair_aggregate 0.5≤τ<1) and max_tokens (≥4, within tokenizer and position limits) take effect and enter model_revision = "hf-encoder-v3:" + sha256(canonical 8 fields). Fixture-v2-era layouts still register and load with τ fixed at 0.5 and hf-finetuned:<src>:<after>.
  • Dependency lock (D8): protocol advisory-encoder-registration-v2 records torch, transformers, tokenizers and safetensors versions from package metadata (torch is not imported) into the identity. Missing at registration → encoder_runtime_unavailable; changed at describe/load → registered_inference_runtime_changed; missing at load → pretrained_encoder_dependencies_unavailable. v1 registrations → encoder_registration_retired; feedback already stored stays readable.
  • Untrained heads stay synthetic: the weights_stage=pretrained_untrained_heads declaration (completed=false, no training partitions, no selection partition) is accepted only by the encoder-specific validate_encoder_return_identity, only after the return is identified as an encoder return, and only for synthetic_fixture. The shared validate_return_identity is strict again: any non-default weights_stage → model_return_weights_stage_invalid, so linear and fusion returns cannot borrow the relaxation.
  • Strict registration: every file must be listed in checkpoint_files (checkpoint_member_set_mismatch); suffix allow-lists for the root and encoder/ (checkpoint_file_type_not_allowed); pair_label_order only on pair_aggregate; float32 heads; unsupported structures → encoder_architecture_unsupported; a task other than relation → model_return_task_or_labels_invalid.
  • Identity: registration and describe_registration carry kind=huggingface_encoder, dataset{dataset_id, dataset_manifest_sha256, label_tiers} and weights_stage; the entry also records variant, threshold, limit and encoder_runtime.
  • measure() tokenizes without a forward pass; an over-limit prediction abstains with encoder_input_truncated and is never truncated.
  • 13 files, +3284/−178 against main 95d4883. No contract re-export; no 144 file touched.

Review and verification

  • Independent review 080-review-146.md (155). First round, on a63241f, sent back 1 P1 and 3 P2:
    • P1: the untrained-head relaxation sat in the shared manifest check, so once 145 was merged, an untrained, incomplete linear return registered as external_candidate.
    • P2: a pair the model rated "most likely unrelated" could count as support when τ < 0.5 or at a 0.5 tie; 5 in-function imports in tests; 4 spec/code mismatches.
    • The reviewer wrote pack, pair and pair_aggregate returns by hand from the spec, without the script. All three registered as external, and their predictions matched an independent computation exactly. 12 mutations all went red.
  • Rework 2608c3f, 8c7604e, c567b33. Second round on b765c88: 可合入, no new P1 or P2. The reviewer replayed P1-1 on 145 23fb9df merged with this head: all three untrained declarations are now refused model_return_weights_stage_invalid, and normal linear returns still register. 9 further mutations all went red. 20,000 random inputs with τ ∈ [0.5, 1) gave 0 violations.
  • Test IDs: the default-environment file tests/unit/test_w08_encoder_registration.py now has 96 (all 15 from eb494c0 kept); the optional tests/optional_encoder/check_encoder_v3.py has 17. Rework reds are full pytest logs:
    • SEAM-03: 2 failed, 1 passed → 3 passed.
    • P2-1: 8 + 2 failed → all passed.
    • P2-3: 5 failed → all passed.
    • P1-1 replay on 145: REGISTERED before the fix → REFUSED after.
  • Line checks on b765c88:
    • Targeted: 203 passed (default environment, no torch).
    • Optional encoder checks: 19 passed, offline.
    • Per-ID against eb494c0: the targeted set went from 80 to 222 IDs, with 0 missing and 0 failing.
    • lint: 0 new.
    • 145 overlay: 205 + 20 passed.
    • Tiny returns: 2 legacy and 4 v3 tiny returns registered and predicted through the CLI.
  • The first-round locked gate on a63241f was green: backend 3321 passed / 31 skipped; frontend 404 / 7; 72 state checks; build; per-ID 0 problems; browser 10 steps each in zh and en; MCP bilingual pass. Full gate ran once on the integration tree 11e53dd7… (146 b765c88 → 145 23fb9df → 147 1e2b85c → 148 261118e → 149 2e56be1, integration branch HEAD 855b366; 157 receipt).

Not claimed / registered

  • No real pretrained weights were downloaded or run. The tiny random BERT only proves format, loading, prediction and wiring, not model quality. The XLM-R smoke test belongs to 150.
  • Criterion head (ENC-04) not built, by the 10-12 scope cut.
  • Not in this line:
    • The UI does not yet show kind, training data or weights_stage (149).
    • Sentence splitting is in 148.
    • "Inference runtime versions" have no DTO field; the coordinator decides whether 149/150 picks this up or it is dropped.
  • Default-environment tests depend on METADATA-only declarations in tests/unit/encoder_runtime_metadata/. These are appended to sys.path and not restored, and they are not in the release package (about 70 failures if the release tests run without the extra). RT-03 is simulated in-process; the reviewer ran the real CLI without the extra and got encoder_runtime_unavailable, rc=2.
  • Registrations carry platform-local version strings (Linux 2.8.0+cpu), so re-register on each deploy machine. Reference-script equivalence is guarded only by the optional checks (150: add a gate step with the extra).
  • P3 from the second round:
    • §13 lacks an operational-codes section.
    • Strict listing makes a stray .DS_Store reject the whole return, so the spec should say "remove hidden files before delivery".
    • The untrained stage checks only the declaration.
    • HuggingFaceCandidate.schema is write-only.
  • Spec attachments go into the v3 package SHA256SUMS (150).
  • Pre-existing main defect found here: with any registration configured, delivering after reading scenario material failed the whole feedback job with object_not_found. Fixed in 148.

Receipt: 完成回执/claude-coordination/146-编码器回传v3.md.

…rations

Encoder registrations now use advisory-encoder-registration-v2: they record the
installed torch, transformers, tokenizers and safetensors versions from package
metadata (no torch import), refuse to register without them, refuse to load or
describe under different versions, and publish kind, dataset and weights_stage.
v1 registrations serve no new advice; feedback recorded before stays readable.
checkpoint.json v3 states variant (pack, pair, pair_aggregate), mean pooling,
evidence_threshold and max_tokens, and model_revision hashes them with the
weights. Registration checks head array names and shapes, the encoder config
and the tokenizer capacity, and returns encoder_architecture_unsupported for
structures outside the specification. The pretrained_untrained_heads stage
registers only as synthetic_fixture with an honest outer manifest. Legacy
fixture v2 era checkpoints keep registering with the fixed 0.5 threshold.
pair_aggregate's deterministic aggregation follows the specification examples.
HuggingFaceCandidate gains pair_aggregate, the declared evidence threshold, the
pretrained_untrained_heads stage (mark_untrained_heads) and writes the v3
layout with canonical JSON and byte-reproducible heads.npz; fixture v2 era
checkpoints still load with the fixed 0.5 threshold. The standalone reference
export script writes the same layout from a producer's encoder directory and
head state, and seals the outer manifest. Optional checks prove the script and
save_checkpoint register the same model for all three variants.
encoder-return.md states the layout, checkpoint.json fields, head arrays,
the encoder input text, the three variants and the pair_aggregate rule,
after_weights, model_revision, weights stages, registration and the runtime
pin, the reference script and every refusal code. The reference script also
gives the exact encoder input strings and refuses mismatched deliveries.
validate_return_identity checks only the fixture v2 outer-manifest fields that
every return kind shares, so linear returns that reuse it are accepted. The
encoder path checks artifact_kind and artifact_protocol itself (is_encoder_return),
encoder_registry.handles_return names the manifests it claims, and the encoder
protocol does not claim linear or fusion registrations.
validate_return_identity, which linear returns reuse, again requires a completed,
trained delivery and refuses any other weights_stage. Encoder returns go through
validate_encoder_return_identity, which admits the honest pretrained-untrained-heads
declaration only after the manifest is known to be an encoder return and only for
a synthetic_fixture registration. SEAM-03 covers linear manifests carrying it.
…e half

A pair counts as support only when its support is at or over the threshold and
above its unrelated probability, and pair_aggregate thresholds below 0.5 are
refused at registration, in the candidate and in the reference script, so an
unrelated-leaning pair never becomes support. The specification examples add
the tie and most-likely-unrelated cases and two refused low thresholds.
checkpoint.json tasks other than relation now fail as
model_return_task_or_labels_invalid, every file in the return directory must be
listed in checkpoint_files with an allowed suffix, pair_label_order is refused on
pack and pair, and heads must be float32. The specification lists every
registration and load refusal code and says registration does not prove loading.
@Jeffreyliu0131
Jeffreyliu0131 merged commit accb6c6 into main Oct 12, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant