PR0.5 — make the stack's base green: the tokenizer witness, and #45's unfinished metric-table migration

[can-goodfire]

can/protocol-refactor — the base the Qwen3.6 hookpoint stack (#46 merged, #47, #48) builds on — was red by 17 tests, not the 2 recorded in the plan note. Two unrelated regressions, one commit each. pytest -m "not golden" is now 1740 passed, 8 deselected, 0 failed.

1. The transformers 5.16.1 bump moved a tokenizer witness (2 tests)

#46 floored transformers at 5.16.1. That changed tokenization, which the parity goldens do not cover — 5.16.1 drops sentencepiece's legacy dummy prefix, so on the tiny Llama tokenizer encode(" Thursday") is now [Th, urs, day], not [▁, Th, urs, day].

The question was whether column_first_token_id's "skip the piece that decodes to whitespace" filter is now a no-op to delete, or is silently eating a real content token. Measured: neither — it is load-bearing. The lone ▁ still appears whenever the first character has no merged ▁X piece (digits, non-Latin scripts, emoji, ligatures — encode(" 3.14") is still [29871, 29941, 29889, 29896, 29946]), and byte-level BPE still splits a whitespace run off the front (gpt2 " ?" → ' ' + ' ?'). It fires on 20/25 probe values on tiny Llama, 5/25 on the Qwen MoE fixture, 2/25 on gpt2, and only ever skips a piece whose decode is empty-or-whitespace — which no content piece is.

Verified by mutation: with the filter removed, first_token credits ▁ (29871) instead of "3" (29941), and both tests below catch it.

So the code stays and the tests change:

• the lone-space contract is pinned against a stub tokenizer that cannot drift, because which values trigger it is a released-tokenizer property that just moved. A second test keeps a live witness (" 3.14") and reports "witness moved" rather than "behaviour broke" if it shifts again.
• the non-fallback contract needs a value whose bare form is one token while its space-prefixed form is several. 5.16.1 leaves no such value on sentencepiece, so the witness moves to gpt2, whose byte-level BPE is byte-identical across the bump ("haus" is [30404], " haus" is [387, 385]).

One finding worth flagging. On a sentencepiece family that has dropped the dummy prefix, " X" and "X" now encode to the same id — so token_form has one row to name there and _ambiguous_under_auto can never fire. The once-per-column warning exists because a punctuation match read a flat 0.000 across all 48 layers of a real gpt2-xl scan; that detector is now structurally dark on sentencepiece. Nothing to fix in the resolver, but it means the punctuation trap is a BPE-family hazard and a document cannot be checked against it by pinning token_form on a sentencepiece model. Recorded as a test plus a note in the module docstring.

Also corrects two docstrings that had inverted into passing for the opposite reason they state, with the reason now pinned so they cannot lie again silently.

2. #45 left its metric-table migration unfinished (15 tests)

Bisected: these pass at f55efa8 (the #46 merge) and fail at 6e467ea (after #45), so they arrived with #45 (workflow-v2), not the bump. #45 tightened §2.12 to "JSON and safetensors are the only two formats" and changed the validator — expected_ext = ".json" if is_metric else ".safetensors", was ".parquet" — but left the call sites declaring .parquet behind.

Nothing in production needed changing, which is the reassuring part: write_outputs → write_table already emits JSON unconditionally, so the declared extension was the only thing out of step. That also means test_run_corpus's two pd.read_parquet read-backs could never have run — pd is not imported in that module — so they move to table_frame, the JSON-table reader the rest of that file already uses.

Corpus digest pins regenerated with update_corpus_digests.py; the diff is confined to 12_probe_variable_im.json (2 lines, its save block changed) and the other 11 documents' pins are byte-identical.

Gates

• pytest -m "not golden" — 1740 passed, 8 deselected (the GPU-only golden tier), 0 failed
• pre-commit run --all-files — passes repo-wide (ruff 0.14.5 pinned)
• the parity goldens are untouched

Plan note: notes/causalab-hookpoint-vocabulary-plan.md §6.1.

🤖 Generated with Claude Code

[github-actions]

Test Results

1 739 tests  +2   1 731 ✅ +20   51s ⏱️ -6s
    1 suites ±0       8 💤 ± 0 
    1 files   ±0       0 ❌  - 18 

Results for commit ae0f88c. ± Comparison against base commit 8af9f89.

♻️ This comment has been updated with latest results.