Step 3 prep: k-shot sampling, validation-only pilots, curve runner, baselines, ModernBERT trial

[drewOrc]

Code and a small compatibility trial for step 3. No pilot, curve or baseline results are produced here; those runs come after this merges.

What changes

• Protocol (docs/PLAN.md section 4.1, configs/curve.yaml): steps = max(S_min, 5-epoch steps). S_min from {100, 200, 400} (pilot at k=5, seed 42, both encoders, mean val in-scope accuracy, ties to the smaller). Learning rate per encoder from {1e-5, 2e-5, 5e-5} (k=100, seed 42, val in-scope accuracy, then val OOS recall, then the smaller rate). Values are copied into configs/curve.yaml by hand; make curve refuses to start while any is null, and off-grid values are refused on load.
• Sampling (sampling.py): exactly k rows per intent; oos rows from the hardcoded table {1:3, 5:13, 10:25, 25:63, 50:125, 100:250}, checked against ceil(2.5k) by a test; k=100 returns the whole train split in its original order; k outside the table raises.
• Steps (steps.py, train.py): min_train_steps on RunConfig; the plan is computed before training, train() raises if the Trainer ran a different count, and the summary records planned steps, actual steps and decided_by. Checked against the real HF Trainer on a tiny local model.
• Pilots (pilots.py, make pilot-lr, make pilot-steps): validation only. predict_validation and validation_scores raise LeakageError on any other split; nothing calls evaluate(). The reused AC2 run is read with reusable_equivalent(..., validation_only=True): the archive file is hashed whole, only its metadata and validation arrays are read, and only run_name, config, training, logits and environment are kept from its results JSON (no metrics). Output in results/pilots/*.json.
• AC2 reuse (runs.equivalent_run): an AC2 run stands in for BERT k=100 only if every config field is equal after two stated rewrites (no sampling equals k=100 with 250 oos rows; an S_min that does not decide changes nothing), and the AC2 record's own train rows, oos rows, global_step and seed match, and it was made with the torch and transformers releases installed now. The archive's SHA-256 must agree in the results JSON, the manifest and on disk. Tested against the three committed AC2 records.
• Curves (curves.py, make curve MODEL=bert|modernbert, make oos-ablation): resume with the AC2 rules, keep no weights, write results/curves/<name>.json with the logits SHA-256 and step plan per point.
• Baselines (baselines.py): majority class (log training prior, constant per query) and TF-IDF centroid (100 x cosine) on the same (k, seed) sample as the encoders, archived and scored in the encoder format.
• Archive format 2: SHA-256 of the three split parquet files. Format 1 (AC2) archives are back-filled on read only when dataset_revision is the pinned one; otherwise refused.
• metrics.wilson_interval, checked against hand values and scipy.

ModernBERT trial (M4, MPS, 50 steps, batch 32, max_length 64)

	ModernBERT-base	bert-base-uncased
step time (median)	0.234 s	0.139 s
peak memory (Metal driver, sampled)	4.80 GiB	3.18 GiB
loss, first 5 -> last 5 steps	5.22 -> 4.22	5.08 -> 5.00
NaN	none	none

Full 5-epoch estimate for ModernBERT is about 24 minutes per seed (AC2's measured BERT time x 1.69; the short trial underestimates BERT by about 2.6x, see DEVLOG). ONNX export works with both torch.onnx.export exporters; ONNX Runtime CPU vs PyTorch max abs logit difference 2.1e-5 on 10 rows and 1.1e-5 on a batch of another shape. ONNX packages are in an optional onnx group; CI does not install them.

Checks

• make lint, make test (317 passed), make test-network (7 passed), make smoke, make verify-logits (the 3 AC2 archives still load as format 1).
• 22 mutations of the key checks (test-split guard, oos table, seed use, tie rules, equivalence fields, step-count check, dataset checksum and revision checks, Wilson formula, baseline sample, weight deletion) each turn at least one test red.

Review fixes (second round)

• PR text check: new pr-text.yml (pr-text-hygiene) runs on opened, edited, synchronize and reopened, and checks the PR title and body, which become the squash commit. Title and body reach the shell only through env. It shares .github/scripts/check-disallowed-text.sh and .github/disallowed-trailers.txt with commit-hygiene. docs/OPERATIONS.md notes the remaining gap (text typed into the merge dialog).
• Pilots never read test: see above; a test watches every NpzFile read and finds no test_* key, and the pilot still runs and picks the same values when the donor JSON has no metrics.
• Weights: deleted only after the results JSON, manifest and archive agree; the curve index checks the archive of every trained point.
• Sampler pinning: golden sample fingerprints in test_sampling.py; the curve index compares each run's recorded fingerprint with curve_sample(k, seed) drawn now and refuses a mismatch; reused AC2 points get the fingerprint computed now, marked as such.
• TF-IDF scale: docstring corrected; PLAN 4.1 says RQ3 uses only temperature-scaled signals and the margin for TF-IDF and leaves the majority baseline out; a test shows 1x and 100x give the same calibrated probabilities.
• Equivalence: schedule comparison has its own tests (max_steps=2385, num_train_epochs=4.999); torch and transformers releases must match (a +cpu build label is ignored).
• scipy is a direct dev dependency.
• 18 more mutations, each turning at least one test red.