Repository navigation
Publish the block-content position vectors and fixtures, and harden the position gate - #128
Merged
Merged
Conversation
Six walks scoped across both implementations, plus a derivation gate. Checkpoint commit so the work survives; not yet reviewed.
Fixtures pin what a READER reports; vectors pin what a CANONICALIZER produces, and the behaviour this branch changes is the canonicalizer's. Without these a third-party implementation that still merges inside `attributes` passes every published vector while computing a different document ID. Six vectors, six fixtures. Each canonical form was hand-derived from the §4.3 rules, its JCS bytes and digest generated by the independent Python oracle, then reproduced by the TypeScript canonicalizer through the adapter protocol. `caption` takes both a merge vector and a reject vector: a merge vector alone cannot catch an implementation scoping the run rule more narrowly than the merge, since both scopes yield the same bytes for a document that merges. The reverse mutant is caught by the merge vector alone. `hints` and `preamble` are covered because they are LIVE positions, not inert ones. Both declare `items` as `$defs/block`, and a `block` dispatches `type:"text"` to `$defs/textNode` in its first `allOf` branch — so a text node is admissible there exactly as in the root `blocks` array, and a schema-valid `academic:exercise` with a split combining sequence in `hints` is rejected today. Before this commit, dropping `hints`, `preamble` or `content` from BOTH transcriptions at once left every gate green except the schema-derivation gate; `hints` and `preamble` now have four guards each. `content` is deliberately left with that one guard. It is the position where the merge and the run rule diverge — canon does not recurse into a text node's `marks` — so pinning today's behaviour would publish an expectation the follow-on is scheduled to change. The divergence is stated in the vector text rather than left for an implementer to infer. The `attributes` witnesses nest their array one level below `semantic`, so the barrier flag must survive an array-to-item hop. A witness sitting directly on the annotation leaves that propagation untested: ten flag-propagation mutants across the six walks survived all 30 gates against the shallower form. All twelve content parts were schema-validated against content.schema.json with negative controls, which no gate does. That is why an earlier draft was wrong: it authored the `attributes` control through `attributes.hist`, which `blockAttributes` forbids (additionalProperties: false), making the artifact that pins "canonicalization rejected a conformant document" itself non-conformant. suiteVersion 0.2.3 -> 0.2.4 (patch position: added cases, pre-1.0). 209 -> 215 vectors, 157 -> 163 fixtures, 366 -> 378 checks.
The derivation gate was the sole guard for three of the six positions, and it could under-derive silently: `deriveKeys` accepted a property only when its own `type` was `array`, so a position declared behind a `$ref` to an array-typed `$def`, or on an inline nested object inside `items` (the shape `academic:abstractBlock.sections[].children` already uses), was invisible. A key the derivation never saw is a key both transcriptions agree about, so the gate reported agreement and exited 0. The derivation now recurses and resolves local `$ref`s; both shapes fail it, named. `CONTENT_BARRIER_KEY` was compared against nothing — the two implementations could barrier different fields, or a field no block carries, and every check still passed. It is now compared across the two and against `blockBase`. Two rows added to the raw-walk differential, and an agreement assertion for an id carried under `attributes`. The assertion is about agreement, not correctness: whether such an id belongs in the shared namespace is open. What must not happen is the raw walk and canon disagreeing, which is an inverted verdict rather than a missed one — a duplicate canon rejects while the walk feeding the verdict reports nothing. Two mutants that previously survived all 30 gates now fail it. The Python oracle's counterpart walk stays uncovered; it needs a witness that only exists in the disputed territory above. Comment corrections: the run rule and the merge share a key test but not a set of reachable positions, since canon skips a text node's `marks` and `content` occurs only there. Three comments claimed they mirror exactly.
CDX-E-PART-STRING-NOT-NFC described the run arm as scoped to hashed, non-deleted material. It has a third condition — position — and an implementer building from the registry alone over-reports and fails the new clean fixture. Note rewritten; errors.json stays at v0.14 / 86 codes, which its own compatibilityPolicy permits for a note. The changelog states two things it would be easy to leave implicit. The scope is a NARROWING whose licence is not yet in spec/: §4.3 applies the transforms across all block types "uniformly" and §4.3.2 binds the rule to "a block" with no field list, where item 5 names lines/subfigures as exhaustive when it means to scope by position. An extension block whose text array is not one of the six keys is no longer enforced. It also records what was implemented and reverted. Gating id-namespace membership on the barrier removes ids from the namespace but leaves `rewriteIds` relabelling them, so two documents differing only in an id under `attributes` canonicalize to identical bytes — a silent document-ID collision in place of a loud false REJECT, which is the worse trade in a content-addressed format. The barrier has to reach `definedId`, `rewriteIds` and the reference resolver together or not at all.
This was referenced Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the first half of #121. §4.3.1 item 4 (merge adjacent text nodes) and §4.3.2 item 2 (concatenated-NFC) were implemented on element shape — "is this object
{type:"text",value:string}?" — so any array whose members merely resembled text nodes was merged and run-checked, and canonicalization REJECTED conformant documents. Both rules are now keyed on position: the schema-declared block-content arrays, withattributesas a sticky barrier.The scoping itself landed in the two commits already on this branch. This PR adds what makes it checkable by someone outside this repository, and hardens the gate that guards it.
What is here
Six vectors, six fixtures. A fixture pins what a READER reports; a vector pins what a CANONICALIZER produces, and the behaviour that changed is the canonicalizer's — without them an implementation that still merges inside
attributespasses every published vector while computing a different document ID. Each canonical form was hand-derived from the §4.3 rules, its bytes and digest generated by the independent Python oracle, then reproduced by the TypeScript canonicalizer through the adapter protocol.hintsandpreambleare covered because they are live positions. Both declareitemsas$defs/block, and ablockdispatchestype:"text"to$defs/textNodein its firstallOfbranch — so a text node is admissible there exactly as in the rootblocksarray. A schema-validacademic:exercisewith a split combining sequence inhintsis rejected today. Before this PR, droppinghints,preambleorcontentfrom both transcriptions at once left every gate green except the schema-derivation gate; the first two now have four guards each.The gate could under-derive silently.
deriveKeysaccepted a property only when its owntypewasarray, so a position behind a$refto an array-typed$def, or on an inline nested object insideitems, was invisible — and a key it never saw is a key both transcriptions agree about, so it reported agreement and exited 0. It now recurses and resolves local$refs.CONTENT_BARRIER_KEYwas compared against nothing and now is.The
attributeswitnesses nest one level belowsemantic, so the barrier flag must survive an array-to-item hop. Ten flag-propagation mutants across the six walks survived all 30 gates against the shallower form.What is deliberately not here
The scope is a narrowing whose licence is not yet in
spec/— filed as #126. §4.3 applies the transforms across all block types "uniformly" and §4.3.2 binds to "a block" with no field list, where item 5 nameslines/subfiguresas exhaustive when it means to scope by position. Measured: avendor:calloutcarryingbody:[text,text]with a split combining sequence REJECTed before and loads clean now. Recorded in CHANGELOG.md so it is not silent.Gating id-namespace membership on the barrier was implemented and reverted — filed as #125. It removes ids from the namespace but
rewriteIdsstill relabels them, so two documents differing only in an id underattributescanonicalize to identical bytes (sha256:42893c8a…) — a silent document-ID collision in place of a loud false REJECT. The barrier has to reachdefinedId,rewriteIdsand the reference resolver together or not at all.Gate residue — filed as #127:
refsOfdoes not resolve local$refs,if/thenis unwalked, the floor is cardinality rather than names, and the gate's own schema-vs-pairwise property is unpinned.contentkeeps a single guard on purpose: it is the position where the merge and the run rule diverge, and pinning today's behaviour would publish an expectation the follow-on changes.Review
Two
/review-fixcycles, the branch's cap.Cycle 1 — 6 lenses launched, 5 reported (the tests lens went idle twice without answering; recorded as 5, not rounded up). 14 verified findings. Six were false factual claims in prose I had written, all green on 30 gates: the
hints/preamblejustification, a "MUST agree" claim in a published vector that is false atcontent, "either implementation … every gate green" forcaption(a pre-existing test catches the TypeScript half), and a walk-attribution error.Cycle 2 — 3 bounded lenses against a closed list. Found that a Cycle 1 fix of mine was a regression: the id-collision above. Reverted.
Every added case is the named killer of a mutant nothing else catches, measured against the two implementations independently.
Verification
30/30 gates green, plus the three CI steps that are not npm scripts (
npm ci,npm audit --audit-level=high→ 0 vulnerabilities,generate:templateinto a fresh empty dir).check:conformance: 378/378 (215 vectors + 163 fixtures), 0 failed, 0 advisory, 0 skipped. suiteVersion 0.2.3 → 0.2.4.