From 5f5989b5bc0f12ecbd9050043dd211babde7abe4 Mon Sep 17 00:00:00 2001 From: Greg von Nessi Date: Sun, 30 Aug 2026 20:23:53 +0100 Subject: [PATCH 1/6] Preserve ids carried under `attributes` rather than relabeling them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the collision half of #125. Two documents differing only in their CSL citation keys canonicalized to identical bytes: A: citation:[{id:"smith2020",...},{id:"jones2019",...}] B: citation:[{id:"OTHER-KEY",...},{id:"DIFFERENT",...}] both -> sha256:869299f2c3d550ca698e09ad7db9ac0152e5a23dae32370742ae48aa8546ce5c A signature over B verified A. §4.3.1 item 5's alpha-equivalence is for author-chosen LABELS; it was being applied to third-party vocabulary data where the keys are content — and item 5 already exempts a `semantic:bibliography` entry's id for exactly that reason. The same data one field over, under `attributes.semantic`, was relabeled. Whether it fired turned on `type` versus `@type`, so well-formed JSON-LD was accidentally immune and CSL-JSON was not. PRESERVED, not excluded. Excluding was implemented and reverted yesterday: it takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, which is a silent collision rather than a loud one. A preserved id stays in the map's input and out of the map, so `renameId` returns it verbatim — the mechanism `semantic:term` and `semantic:footnote` already use. It also inherits item 5's canonical-name prohibition, so an id under `attributes` spelling `b0` is now an error. Baseline differential, every witness schema-valid: before after CSL keys under attributes id -> b0 preserved duplicate anchor id there ID-COLLISION ID-COLLISION link href resolving into attributes href -> #b0 resolves by name id there consumes a b-index real block b1 real block b0 semantic:ref there, dangling DANGLING DANGLING id there duplicating a real block id ID-COLLISION ID-COLLISION CONTROL, same shapes in real content unchanged unchanged The duplicate cases still REJECT and that is correct: measured, a `link` href resolves INTO `attributes` and is rewritten, so that material is reachable and a duplicate there is a real collision. The comment claiming "nothing anchors into it" was a normative wish stated as fact. The b-index shift is deliberate: item 5 says a preserved occurrence "consumes no index", and states that reserving one instead would give identical content a different document ID. Zero documents in the corpus carry such an id, so nothing moves here. --- scripts/lib/canonicalize.ts | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/scripts/lib/canonicalize.ts b/scripts/lib/canonicalize.ts index c5dd5c0..9b8c412 100644 --- a/scripts/lib/canonicalize.ts +++ b/scripts/lib/canonicalize.ts @@ -810,7 +810,7 @@ export function collectDefinedIds(content: unknown, options: CollectOptions = {} ids.push(id); // Recorded on the FIRST occurrence only, alongside `ids` — a repeat is a // duplicate and throws before any of this is consumed. - if (!ctx.inMarks && typeof node.type === 'string' && PRESERVED_ID_BLOCK_TYPES.has(node.type)) { + if (ctx.blocked || (!ctx.inMarks && typeof node.type === 'string' && PRESERVED_ID_BLOCK_TYPES.has(node.type))) { preserved.add(id); } // Classified on the SAME first-occurrence branch as `ids`, so the two cannot @@ -840,6 +840,14 @@ export interface WalkContext { * it models a rewrite the canonicalizer never performs. */ inTextMarks: boolean; + /** + * The node sits at or below `attributes` (CONTENT_BARRIER_KEY) — a closed set of authored + * values (Content Blocks §3.1) whose members are data. An id there is content, not a label: + * it stays in the uniqueness namespace but is never relabeled, exactly as a + * PRESERVED_ID_BLOCK_TYPES id is. Removing it from the namespace instead relabels it via + * `rewriteIds`, which maps it onto a generated name. + */ + blocked: boolean; } /** @@ -874,7 +882,7 @@ export function walkContentNodes( return; } if (!isPlainObject(value)) return; - visit(value, { inMarks, inArray, parentKey, inTextMarks }); + visit(value, { inMarks, inArray, parentKey, inTextMarks, blocked }); for (const key of Object.keys(value).sort()) { if (raw && !inTextMarks && isDerivedField(value, key)) continue; // canon deletes it before relabeling const child = value[key]; @@ -1108,8 +1116,9 @@ function alphaRenameIds(content: unknown): unknown { for (const id of preserved) { if (CANONICAL_NAME.test(id)) { throw new CanonicalizationError( - `id "${id}" uses the reserved canonical-name form: the id of a block whose type is left as authored ` + - `(${[...PRESERVED_ID_BLOCK_TYPES].sort().join(', ')}) must not spell "b"`, + `id "${id}" uses the reserved canonical-name form: an id left as authored — carried under ` + + `\`${CONTENT_BARRIER_KEY}\`, or on a ${[...PRESERVED_ID_BLOCK_TYPES].sort().join(' / ')} block ` + + `— must not spell "b"`, ); } } From 16ea16f933fc07211dc2df959d17f96beae76fa7 Mon Sep 17 00:00:00 2001 From: Greg von Nessi Date: Mon, 31 Aug 2026 13:54:37 +0100 Subject: [PATCH 2/6] Pin the preserved-id behaviour, in tests and in the published suite A test encoding the whole differential, a published vector for the canonicalizer behaviour, and a published fixture for the reader's. The vector and the fixture kill DIFFERENT mutants, which is why both are here: relabeling returns (the fix reverted) -> canonicalize/attributes-id-preserved-not-relabeled expected "id":"smith2020", got "id":"b0" ids EXCLUDED from the namespace instead of preserved -> document/reject-content-attributes-id-collision disposition IGNORE outside [INTEGRITY-ERROR, REJECT] That second mutant is the approach tried and reverted on the #128 branch. It reports nothing here AND relabels the id anyway, which is the silent collision. Nothing in the published suite caught it before. The fixture expectation is a RANGE, not a pin: CDX-E-ID-COLLISION is registered INTEGRITY-ERROR and the later states escalate it, as the other id-collision cases already express. Pinning REJECT is what the first draft did, and check:conformance rejected it. The Python oracle needed no counterpart, measured rather than assumed: it does not relabel, so preserved-versus-relabeled is invisible to it, while uniqueness is unchanged. It confirms the new fixture independently (115 -> 116 injected defects confirmed). 216 vectors + 164 fixtures = 380 checks, all 30 gates green. --- .../case.cdx | Bin 0 -> 937 bytes .../case.json | 38 +++++++++++++ conformance/vectors/canonicalize.json | 11 ++++ scripts/lib/fixture-corpus.ts | 17 ++++++ scripts/test-canonicalize.ts | 50 ++++++++++++++++++ 5 files changed, 116 insertions(+) create mode 100644 conformance/fixtures/document/reject-content-attributes-id-collision/case.cdx create mode 100644 conformance/fixtures/document/reject-content-attributes-id-collision/case.json diff --git a/conformance/fixtures/document/reject-content-attributes-id-collision/case.cdx b/conformance/fixtures/document/reject-content-attributes-id-collision/case.cdx new file mode 100644 index 0000000000000000000000000000000000000000..2d4c811a9ad141d4e09d237067cb00b7e374ecaf GIT binary patch literal 937 zcmaJ=y;9sT5H>%_(4>PR-JzO0{O_}yz(A9M;uunexk9#Nti)$|!?@u(C zNhVV9kwqd4;02JVAti+jp%$7$CX;%GS6y0&mC;IHR~fZEKXP5)aolihrM^9O9M20< zZiQarv4C1Gvjac&ozM?hFm~KHb!;o}!qCRA?Qk;=NQ>EVpo{|5!C4*WqBxOC*Y|R9 zIM)snRL|3B+sP7_Pt()kJ#wgJ4`pTY4f(@WC1ionb6(?ICci5ZT{Q%5N*%JKie4*P zW>{&*I;%^~a3*r5IC@@9+k=(X6hMi%299_QsW`S1M56ac3+xL7LWEcEjMGI!*F|cP zou$yzUfmDR7vC { entries: [{ id: 'b0', type: 'text', value: 'x' }] }] }), 'sha256')); }); +test('an id under `attributes` is preserved, not relabeled', () => { + const parts = (content: unknown) => ({ manifest: '{}', content: JSON.stringify(content), dublinCore: '{}' }); + const canon = (content: unknown) => canonicalContent(parts(content)) as any; + const cited = (k1: string, k2: string) => ({ blocks: [{ type: 'paragraph', + attributes: { semantic: { citation: [{ id: k1, type: 'article-journal' }, { id: k2, type: 'book' }] } }, + children: [{ type: 'text', value: 'x' }] }] }); + + // Alpha-equivalence is for author-chosen LABELS. Applied to third-party vocabulary data it + // erases content: two bibliographies with different citation keys hashed identically, so a + // signature over one verified the other. + assert.notEqual( + computeDocumentId(parts(cited('smith2020', 'jones2019')), 'sha256'), + computeDocumentId(parts(cited('OTHER-KEY', 'DIFFERENT')), 'sha256'), + ); + assert.equal(canon(cited('smith2020', 'jones2019')).content.blocks[0].attributes.semantic.citation[0].id, 'smith2020'); + + // Preserved means preserved-and-still-counted: it consumes no index, so a real sibling takes + // the name the preserved occurrence would have had (§4.3.1 item 5). + const shifted = canon({ blocks: [ + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', note: { type: 'heading', id: 'ghost', children: [] } } }, children: [] }, + { type: 'paragraph', id: 'real', children: [] }] }); + assert.equal(shifted.content.blocks[0].attributes.semantic.note.id, 'ghost'); + assert.equal(shifted.content.blocks[1].id, 'b0', 'the preserved occurrence consumed no index'); + + // ... and still in the UNIQUENESS namespace. That material is reachable — a `link` href + // resolves into it — so a duplicate there is a real collision, not a false positive. + assert.throws(() => computeDocumentId(parts({ blocks: [ + { type: 'paragraph', id: 'dup', children: [] }, + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', n: { type: 'heading', id: 'dup', children: [] } } }, children: [] }] }), 'sha256'), + /duplicate id/); + + // A reference INTO the barriered subtree keeps resolving, because the target keeps its name. + const linked = canon({ blocks: [{ type: 'paragraph', + attributes: { semantic: { '@type': 'Q', children: [{ type: 'text', value: 't', marks: [{ type: 'anchor', id: 'hidden' }] }] } }, + children: [{ type: 'text', value: 'x', marks: [{ type: 'link', href: '#hidden' }] }] }] }); + assert.equal(linked.content.blocks[0].children[0].marks[0].href, '#hidden'); + + // Inherits item 5's canonical-name prohibition: left as authored beside a generated `b0`, + // two identifiers would reach the canonical output under one name. + assert.throws(() => computeDocumentId(parts({ blocks: [{ type: 'paragraph', + attributes: { semantic: { '@type': 'Q', n: { type: 'heading', id: 'b0', children: [] } } }, children: [] }] }), 'sha256'), + /reserved canonical-name form/); + + // CONTROL. Ordinary content must be untouched by all of the above — without this the + // assertions are equally satisfied by a canonicalizer that stopped relabeling entirely. + const plain = canon({ blocks: [{ type: 'paragraph', id: 'p1', children: [{ type: 'text', value: 't', marks: [{ type: 'anchor', id: 'a2' }] }] }] }); + assert.equal(plain.content.blocks[0].id, 'b0', 'a real block still relabels'); + assert.equal(plain.content.blocks[0].children[0].marks[0].id, 'b1', 'and a real anchor mark still relabels'); +}); + // --- The asset map in the raw walk (B1b-3b-2) -------------------------------- // `normalizeMarks` resolves a `link` mark's href to its asset HASH before // `mergeAdjacentText` compares mark sets, so whether two adjacent text nodes merge — and From 11c18230f7fcd1271523873cda57342dab400881 Mon Sep 17 00:00:00 2001 From: Greg von Nessi Date: Mon, 31 Aug 2026 13:55:42 +0100 Subject: [PATCH 3/6] Record the collision fix; suiteVersion 0.2.5 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 215 -> 216 vectors, 163 -> 164 fixtures, 378 -> 380 checks. Added cases take the patch position pre-1.0. The entry records the one deliberate cost — a preserved occurrence consumes no index, so a real sibling takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Zero in the corpus, measured. --- CHANGELOG.md | 2 ++ conformance/suite.json | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eb31f66..70f1445 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - Template generation script ### Fixed +- **An `id` carried under `attributes` is no longer relabeled, closing a silent document-ID collision.** §4.3.1 item 5 relabels author-chosen identifiers so two documents differing only in their LABELS share an id. That alpha-equivalence was reaching third-party vocabulary data, where an id is content: measured, two bibliographies whose CSL citation keys differed entirely — `smith2020`/`jones2019` versus `OTHER-KEY`/`DIFFERENT` — canonicalized to identical bytes, so **a signature over one verified the other**. Whether it fired turned on `type` versus `@type`, leaving well-formed JSON-LD accidentally immune and CSL-JSON exposed — the shape test leaking, where item 5 says membership is keyed on position. Item 5 already exempts a `semantic:bibliography` entry's `id` for exactly this reason; the same data one field over was not. **PRESERVED, not excluded**, and the distinction is the whole fix: excluding takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, which trades a loud failure for a silent one — that approach was implemented and reverted on the previous branch. A preserved id stays in the map's INPUT and out of the map, so `renameId` returns it verbatim, reusing the mechanism `semantic:term` and `semantic:footnote` already use, and inheriting item 5's canonical-name prohibition for free. **The duplicate cases still REJECT, and that is correct.** The premise that nothing can reference material under `attributes` is false as implemented: a `link` mark's `href` resolves into it and is rewritten, so a duplicate id there leaves a reference with two candidate referents. The code comment asserting “nothing anchors into it” stated a wish as a fact. One consequence is deliberate: a preserved occurrence **consumes no index**, so a real sibling block now takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Item 5 states that rule and records that reserving an index instead “would give identical content a different document ID”. Zero documents in the corpus carry one, measured. The remaining consumers of the barrier — the reference resolver's walks, and the structural id-uniqueness check — are unchanged and tracked separately; this closes the collision, not the whole barrier question. +- **Conformance suite `suiteVersion` 0.2.4 → 0.2.5.** 215 → 216 vectors, 163 → 164 fixtures, 378 → 380 checks. The vector and the fixture kill DIFFERENT mutants and both are needed: relabeling returning is caught only by the vector (`"id":"smith2020"` versus `"id":"b0"`), while excluding ids from the namespace rather than preserving them is caught only by the fixture (disposition IGNORE where INTEGRITY-ERROR is required). Nothing in the published suite caught that second case before, which is how the reverted approach reached a branch. The fixture expects a disposition RANGE rather than a pin, since `CDX-E-ID-COLLISION` is registered INTEGRITY-ERROR and the later states escalate it. The Python oracle needs no counterpart — measured, not assumed: it does not relabel, so preserved-versus-relabeled is invisible to it, and it confirms the new fixture independently. - **Canonicalization no longer REJECTs conformant documents carrying text-shaped data in a position that is not block content.** Document Hashing §4.3.1 item 4 (merge adjacent text nodes) and §4.3.2 item 2 (concatenated-NFC) were implemented on element **shape** — “is this object `{type:"text",value:string}`?” — so ANY array whose members merely resembled text nodes was merged and run-checked. Both rules are now keyed on **position**: the schema-declared block-content arrays `blocks`, `caption`, `children`, `content`, `hints`, `preamble`, plus `attributes` as a **sticky** barrier. The barrier is what the shape reading gets wrong in practice: `attributes.semantic` is a JSON-LD annotation with `additionalProperties: true` (semantic.schema.json), so an author may put a member named `children` there at any depth, and clearing the flag on the way down readmits the very false REJECT the scoping removes. Only the RUN rule is withheld beneath `attributes` — it is HASHED material, so both **per-string** invariants still bind there, pinned by two REJECT fixtures; without them the scope could be widened into a wholesale exemption with every gate green, only a non-NFC KEY having been covered. The set is **keys, not (type, key) pairs**, because several content arrays hang off sub-objects carrying no `type` at all (`exercisePart`, `solution`, a phantom's content), which a type-keyed map cannot reach. **The scope is a NARROWING, and its licence is not yet in `spec/`.** §4.3 applies the transforms across all block types “uniformly” and §4.3.2 binds the rule to “a block” with no enclosing-field list, where §4.3.1 item 5 names `lines`/`subfigures` as exhaustive when it means to scope by position. Measured: a `vendor:callout` carrying `body:[text,text]` with a split combining sequence REJECTed before and loads clean now. A spec erratum is required and is tracked separately; until it lands the reference under-enforces §4.3.2 for extension blocks whose text array is not one of the six keys. Strict `children` is the specific wrong answer, and it is measurable: it drops the ROOT `blocks` array that `reject-content-split-combining-sequence-mid-array` uses AND `figure.caption`, and now breaks **eight** published cases (four vectors, four fixtures) where before this slice it broke one. **Six walks** were scoped, four in `scripts/lib/canonicalize.ts` (`canon`, `walkContentNodes`, `validateStoredByteInvariants`, `collectStoredTextViolations`) and two in `conformance/oracles/document_oracle.py` (`_walk_text_violations`, `_walk_content`). `dropAbsorbedTextNodes` and `_drop_absorbed_text_nodes` — the pair an earlier withdrawn attempt missed — are unchanged in body; what was scoped is their INVOCATION, inside `walkContentNodes` and `_walk_content` respectively. Those two model the merge for the RAW walk, and scoping the other four alone makes canon keep a node the raw walks drop. **The barrier gates the merge and the run rule, NOT id-namespace membership**, and that asymmetry is deliberate for now rather than settled. `definedId` still counts an id carried under `attributes`, so two `anchor` marks sharing one there is a REJECT even though the merge treats the same subtree as inert. Gating `definedId` as well was implemented and REVERTED: it removes ids from the namespace but `rewriteIds` still relabels them, so two documents differing only in an id under `attributes` canonicalized to identical bytes — a silent document-ID collision in place of a loud false REJECT, which is the worse trade in a content-addressed format. The barrier has to reach `definedId`, `rewriteIds` and the reference resolver together or not at all; that is tracked separately. **Scan and canon agree at every position they both reach, with one KNOWN divergence.** `canon` skips a text node's `marks`, and `content` (a `presentation:footnote` mark's content blocks) occurs only there, so at that position the run rule fires and the merge does not — split text in a footnote mark's content is reported, while two mark-less nodes there stay unmerged. That asymmetry is #121's sequel, not a licence to widen the scope, and it is stated in the published vector rather than left for an implementer to infer. `check:block-content-positions` re-derives the set from the published schemas and fails on drift, comparing each transcription against the SCHEMA rather than against the other. Its derivation is **recursive and resolves local `$ref`s**: a position declared behind a `$ref` to an array-typed `$def`, or on an inline nested object (the shape `academic:abstractBlock.sections[].children` already uses), was previously invisible — and under-derivation is the one failure the gate cannot report on itself, since a key it never saw is a key both sets agree about. Both shapes now fail it. It also compares `CONTENT_BARRIER_KEY` across the two implementations and against `blockBase`, which nothing checked before. The change is a pure relaxation with **zero id blast radius** in this repository: every id-verifying gate passes unchanged and no example or fixture exercised the path. - **The published `anchor-range` structural rule no longer false-positives on a well-ordered range above 2^53.** `checkAnchors` ordered a Content Anchor URI's bounds by converting them to numbers, so two distinct integers that round to the same double compared equal and the range was reported inverted — under a message naming the same number twice. The URI grammar admits an unbounded `[0-9]+` and §4.3.2's 2^53 limit governs hashed *numbers* rather than digits inside a string, so such a URI is conformant. This mattered beyond its own rule: the document layer's position check had the identical defect and was fixed in the same change, which left two implementations in this repo disagreeing about the exact URI a new fixture pins. The comparison is now shared between them (`scripts/lib/decimal-digits.ts`), which is also what forced the leading-zero case to be handled — `[0-9]+` admits `007`, so a length-first digit comparison would otherwise have called `#p1/007-10` inverted. Two `anchor-range` vectors pin both - Broken cross-reference in presentation layers spec (Section 5.1.2 → 5.4) diff --git a/conformance/suite.json b/conformance/suite.json index 8d6d9ac..8cfbe13 100644 --- a/conformance/suite.json +++ b/conformance/suite.json @@ -2,7 +2,7 @@ "normative": false, "$comment": "NON-NORMATIVE. This is the entry-point manifest for the CDX conformance suite, a testing artifact that references the specification and never extends it. The specification (spec/) and the published schemas (schemas/) are the only normative artifacts. Where anything here appears to disagree with spec/, the specification governs.", "suite": "cdx-conformance", - "suiteVersion": "0.2.4", + "suiteVersion": "0.2.5", "specVersion": "0.1", "title": "CDX Conformance Suite", "description": "An independently-runnable suite that lets a third-party implementation check itself against the CDX specification. This manifest binds a suite version to the specification version its expectations were derived against.", From 93a5b5b612a308b0933d6d3a6435e5cb0ff8de30 Mon Sep 17 00:00:00 2001 From: Greg von Nessi Date: Mon, 31 Aug 2026 13:58:46 +0100 Subject: [PATCH 4/6] Amend item 5 for the position-keyed exception MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fix shipped behaviour the specification contradicts, which is the defect #126 was filed about, repeated one branch later. Item 5 said `semantic:term` and `semantic:footnote` "are the only namespace members whose ids are left as authored". An id under `attributes` is now a third kind, and the transform preamble still counted "the two id-relabeling exceptions item 5 names". `attributes` appeared once in the whole document, in item 4's barrier sentence, and never in item 5. Item 5 now states the position-keyed exception with its reason — an id under a closed set of authored values is a vocabulary key, not a label, and relabeling it applies alpha-equivalence to data the author did not choose as one — and marks the two extension block types as the further TYPE-keyed exception, which is the one that departs from this item's position principle. The "preserved id remains a full member" paragraph now covers both kinds, and the `b` prohibition was already generic to "a preserved id". The reason is the same one already given for a `semantic:bibliography` entry's `id`, so the erratum cites it rather than restating it. --- spec/core/06-document-hashing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/core/06-document-hashing.md b/spec/core/06-document-hashing.md index 5b790c6..97be348 100644 --- a/spec/core/06-document-hashing.md +++ b/spec/core/06-document-hashing.md @@ -161,13 +161,13 @@ The canonical content is a *transform* of the stored parts; the stored files are **Metadata projection.** From `metadata/dublin-core.json`, take the `terms` object and keep exactly `title`, `creator`, `subject`, `description`, and `language`, flattened to a `{term: value}` object (the `{version, terms}` wrapper and every other term — including the structured `creators` — are dropped). `title` and `description` are strings; `creator`, `subject`, and `language` are arrays (a scalar value is coerced to a one-element array). A term is omitted entirely when it is absent or wholly empty (`""` or `[]`); non-empty array elements are preserved verbatim, in authored order, and an empty (`""`) element inside an otherwise non-empty array is dropped (a term whose elements are all empty is likewise omitted). (`title` and `creator` are required and non-empty, so they always survive.) -**Content transforms.** Apply to `content/document.json`, across all block types including `namespace:type` extension blocks — uniformly, but for the position scoping item 4 names and the two id-relabeling exceptions item 5 names: +**Content transforms.** Apply to `content/document.json`, across all block types including `namespace:type` extension blocks — uniformly, but for the position scoping item 4 names and the id-relabeling exceptions item 5 names: 1. **Strip non-content fields**: remove `measurement.display` and `codeBlock.tokens` (presentational/regenerable; no canonical form), and any `crdt` field carried on a block or text node (transient collaboration synchronization state, excluded per section 4.1a). 2. **Resolve asset references**: replace every content-level asset *path* reference — an `image.src`, `svg.src`, or `signature.image`, or an archive-relative `href` on a `link` mark — with the referenced asset's content hash (`algorithm:hexdigest`). A reference resolves iff, after path normalization (no `.` or `..` segments; case-sensitive), it equals an asset's archive path — that asset's category directory (`assets/` followed by its category, i.e. the key under `manifest.assets` that registers it: the standard `images`, `fonts`, or `embeds`, or any additional category) joined with the asset's index `path` (Asset Embedding, section 3). The following are not asset paths and are left verbatim: an `href` beginning with `#` (an internal Content Anchor reference), a reference marked `external`, and any reference carrying a URL scheme (e.g. `https://`); an `svg` with inline `content` and no `src` has nothing to resolve; and an `image.fallback` path is left verbatim, being a backup rather than the bound resource. A reference beginning with `assets/` that matches no registered asset is a canonicalization error. The document ID thereby inherits the integrity of the asset index `hash` values, which MUST be verified against the asset bytes (Asset Embedding, section 8) before the ID can be trusted. 3. **Normalize marks**: within each text node, sort the `marks` array by each mark's JCS serialization (UTF-16 code-unit comparison — so bare formatting marks such as `"bold"` sort before structured marks such as `{"type":"link",…}`), remove marks with identical JCS serialization (deduplicate), and omit the `marks` key entirely when the array is empty (absent ≡ `[]`). Mark order is not semantic. 4. **Merge text nodes**: merge adjacent sibling text nodes that have no intervening non-text inline child and identical canonical mark-sets, concatenating their `value`s. Only a text node whose sole keys are `type`, `value`, and `marks` (after the step 1 stripping) is merge-eligible; a text node that also carries an `id`, `attributes`, or any other field is preserved unchanged and acts as a boundary, so no identifier or annotation is dropped. Offsets are defined over the concatenated text content (Anchors and References, section 3), so this moves no anchor. This rule and section 4.3.2 item 2 apply only within a **block-content array** — an array reached through one of the field names `blocks`, `caption`, `children`, `content`, `hints` or `preamble`, which is the exhaustive set. Membership is keyed on that field NAME and not on the owner's type: several such arrays hang off sub-objects carrying no `type` at all — a `subfigure`, an `academic:exercise` part, a `solution`, a structured abstract's section — and the content root, which is not a block, owns one. An implementation keying on the owner rather than the field misses them and computes a different document ID. Membership is keyed on the enclosing field, not on element shape: a text-shaped object an extension block carries in any other array is not block content and is left as authored, and `attributes` is a barrier at every depth beneath it — a closed set of authored values (Content Blocks, section 3.1) whose members are data even when shaped like inline content. An extension block therefore receives these transforms on a rich-text array exactly when it names that array with one of these keys; an array named anything else carries text the transforms do not reach, and a document containing one is still conformant. The two rules do not reach identical positions: this rule does not recurse into a text node's `marks`, so it never reaches a `presentation:footnote` mark's `content`, while section 4.3.2 item 2 does. -5. **Canonicalize identifiers**: relabel author-chosen identifiers to position-based canonical names, so that two documents differing only in their id *labels* produce the same document ID. A text node's `id` is outside this namespace entirely: it is never relabeled, occupies no index, and is not compared for duplicates — item 4 preserves it unchanged because it is content rather than a label. Because it survives into the canonical content, it MUST NOT spell a canonical name (`b` followed by digits); one that does is a canonicalization error, for the same reason the prohibition below applies to a preserved id — left as authored beside a generated `b0`, two identifiers reach the canonical output under one name. The relabeled namespace is the shared identifier namespace (Anchors and References, section 4): the `id` of every block other than a text node (including `namespace:type` extension blocks), the `id` of every `anchor` mark, and the `id` of every in-content sub-block element that carries one — an academic equation line and a `subfigure` — even though those carry no block `type`. These sub-block elements are the array items of an equation group's `lines` and a figure's `subfigures` — the exhaustive set of relabeled sub-block id arrays. Membership is keyed on WHERE an id sits (the enclosing field), not merely on whether the node has a `type` or is an array item: an id-bearing object an extension block carries in its own data array (any array other than `lines`/`subfigures`) is left as authored, even when shaped like a namespace member. In particular a `semantic:bibliography` entry's `id` — a CSL citation key in an `entries` array, referenced by a `citation` mark's `refs` (below) — is not relabeled, though the entry carries a CSL `type` that makes it resemble a block. Likewise an identifier carried on a singular named sub-object — such as a signature block's `signer` `id`, a Person identifier rather than a content-anchor target — is not part of this namespace. Two extension block types are an explicit exception to the *relabeling*, and are the only namespace members whose ids are left as authored: `semantic:term` and `semantic:footnote`. Each is the **defining** side of a reference that addresses a separate namespace and is itself never relabeled — a `glossary` mark's `ref` and a `semantic:term` `see` resolve against a `semantic:term` block's `id` when no external glossary is declared, and a `footnote` mark resolves against a `semantic:footnote` block (Semantic Extension sections 4.5.2 and 8.3) — so relabeling the definition while leaving the reference as authored left those rules unsatisfiable on the canonical form: the block became `b0` while the reference still named `term-algorithm`. A preserved id remains a full member of the namespace in every other respect: it occupies the shared uniqueness space, a duplicate is a canonicalization error exactly as above, and a Content Anchor URI resolves against it exactly as against a relabeled block id. Only the renaming skips it. Because it is left as authored, a preserved id MUST NOT itself spell a canonical name (`b` followed by digits); one that does is a canonicalization error. This mirrors the prohibition on unresolved references below and holds for the same reason: duplicate detection compares *authored* ids, so an authored `b0` does not collide with the `b0` generated for some other block, and both would reach the canonical output under one identifier. Membership in this exception is keyed on the block's `type` rather than on where the id sits — unlike the rest of this item, and of necessity, since a preserved id sits directly on its block, exactly where any other block id sits. Walk the canonical content depth-first — at each node assigning a name to the node's own `id` (if any) before recursing into its members (object members in JCS key order, array elements in index order) — and give each id-defining occurrence the next name in the sequence `b0`, `b1`, `b2`, …; a duplicate id within this namespace is a canonicalization error. An occurrence whose id is preserved takes no name and **consumes no index**: the sequence continues at the same name for the next relabeled occurrence, so a `semantic:term` followed by a paragraph yields `b0` on the paragraph. Reserving an index for the preserved occurrence instead is equally implementable and would give identical content a different document ID, so the choice is stated here rather than left to the implementation. Then rewrite every Content Anchor URI reference (`#id`, optionally followed by `/offset` or `/start-end`) whose id is one of these names — preserving the suffix — in the following reference fields: a `link` mark `href`; the `academic:theorem-ref`, `academic:equation-ref`, and `academic:algorithm-ref` mark `target`; the `academic:theorem` `uses`; the `academic:proof` `of`; and the `semantic:ref` and `presentation:reference` block `target`. A Content Anchor URI whose id is not a relabeled id — for example a cross-document anchor — is left unchanged, except that such an unresolved reference MUST NOT itself spell a canonical name (`b` followed by digits): an unresolved `#b0` is a canonicalization error, because left unchanged it would be indistinguishable from a reference that resolved to that block, collapsing two distinct documents onto one ID. Because relabeling an `anchor` id or a `link` `href` changes that mark's serialization, the marks of any affected text node are re-sorted and de-duplicated (item 3) after rewriting. +5. **Canonicalize identifiers**: relabel author-chosen identifiers to position-based canonical names, so that two documents differing only in their id *labels* produce the same document ID. A text node's `id` is outside this namespace entirely: it is never relabeled, occupies no index, and is not compared for duplicates — item 4 preserves it unchanged because it is content rather than a label. Because it survives into the canonical content, it MUST NOT spell a canonical name (`b` followed by digits); one that does is a canonicalization error, for the same reason the prohibition below applies to a preserved id — left as authored beside a generated `b0`, two identifiers reach the canonical output under one name. The relabeled namespace is the shared identifier namespace (Anchors and References, section 4): the `id` of every block other than a text node (including `namespace:type` extension blocks), the `id` of every `anchor` mark, and the `id` of every in-content sub-block element that carries one — an academic equation line and a `subfigure` — even though those carry no block `type`. These sub-block elements are the array items of an equation group's `lines` and a figure's `subfigures` — the exhaustive set of relabeled sub-block id arrays. Membership is keyed on WHERE an id sits (the enclosing field), not merely on whether the node has a `type` or is an array item: an id-bearing object an extension block carries in its own data array (any array other than `lines`/`subfigures`) is left as authored, even when shaped like a namespace member. In particular a `semantic:bibliography` entry's `id` — a CSL citation key in an `entries` array, referenced by a `citation` mark's `refs` (below) — is not relabeled, though the entry carries a CSL `type` that makes it resemble a block. Likewise an identifier carried on a singular named sub-object — such as a signature block's `signer` `id`, a Person identifier rather than a content-anchor target — is not part of this namespace. An id carried under `attributes` is left as authored, keyed on POSITION: item 4 makes that a barrier at every depth, and Content Blocks section 3.1 closes it to authored values, so an id there is content — a vocabulary key a JSON-LD or CSL consumer resolves against — and not a label this rule may rename. Relabeling it applies alpha-equivalence to data the author did not choose as a label: two documents whose bibliographic citation keys differ entirely would otherwise share a document ID, so a signature over one verifies the other. This is the same reasoning that exempts a `semantic:bibliography` entry's `id` above, applied to the same data one field over. Two extension block types are a further exception, keyed on TYPE rather than position: `semantic:term` and `semantic:footnote`. Each is the **defining** side of a reference that addresses a separate namespace and is itself never relabeled — a `glossary` mark's `ref` and a `semantic:term` `see` resolve against a `semantic:term` block's `id` when no external glossary is declared, and a `footnote` mark resolves against a `semantic:footnote` block (Semantic Extension sections 4.5.2 and 8.3) — so relabeling the definition while leaving the reference as authored left those rules unsatisfiable on the canonical form: the block became `b0` while the reference still named `term-algorithm`. A preserved id — by position or by type — remains a full member of the namespace in every other respect: it occupies the shared uniqueness space, a duplicate is a canonicalization error exactly as above, and a Content Anchor URI resolves against it exactly as against a relabeled block id. Only the renaming skips it. Because it is left as authored, a preserved id MUST NOT itself spell a canonical name (`b` followed by digits); one that does is a canonicalization error. This mirrors the prohibition on unresolved references below and holds for the same reason: duplicate detection compares *authored* ids, so an authored `b0` does not collide with the `b0` generated for some other block, and both would reach the canonical output under one identifier. Membership in the TYPE-keyed exception is keyed on the block's `type` rather than on where the id sits — unlike the rest of this item, and of necessity, since such an id sits directly on its block, exactly where any other block id sits. The `attributes` exception above is keyed on position, as the rest of this item is. Walk the canonical content depth-first — at each node assigning a name to the node's own `id` (if any) before recursing into its members (object members in JCS key order, array elements in index order) — and give each id-defining occurrence the next name in the sequence `b0`, `b1`, `b2`, …; a duplicate id within this namespace is a canonicalization error. An occurrence whose id is preserved takes no name and **consumes no index**: the sequence continues at the same name for the next relabeled occurrence, so a `semantic:term` followed by a paragraph yields `b0` on the paragraph. Reserving an index for the preserved occurrence instead is equally implementable and would give identical content a different document ID, so the choice is stated here rather than left to the implementation. Then rewrite every Content Anchor URI reference (`#id`, optionally followed by `/offset` or `/start-end`) whose id is one of these names — preserving the suffix — in the following reference fields: a `link` mark `href`; the `academic:theorem-ref`, `academic:equation-ref`, and `academic:algorithm-ref` mark `target`; the `academic:theorem` `uses`; the `academic:proof` `of`; and the `semantic:ref` and `presentation:reference` block `target`. A Content Anchor URI whose id is not a relabeled id — for example a cross-document anchor — is left unchanged, except that such an unresolved reference MUST NOT itself spell a canonical name (`b` followed by digits): an unresolved `#b0` is a canonicalization error, because left unchanged it would be indistinguishable from a reference that resolved to that block, collapsing two distinct documents onto one ID. Because relabeling an `anchor` id or a `link` `href` changes that mark's serialization, the marks of any affected text node are re-sorted and de-duplicated (item 3) after rewriting. The following are left as authored, so a document that differs only in one of these labels is not yet guaranteed to share a document ID: the reference fields that address other namespaces — the `footnote` mark `id`, the `glossary` mark `ref`, `semantic:term` `see`, `citation` `refs`, `entity` `uri`, `index` `term`, and the `academic:algorithm-ref` `line` (a label into an algorithm's line numbering, not an `id`). Equation-line ids and subfigure ids are not in this set: they share the relabeled uniqueness namespace described above. A `semantic:footnote` or `semantic:term` block's own `id` is left as authored as well, but for a different reason and with a different consequence: it does not address another namespace — it stays in the shared uniqueness-and-anchor namespace described above and merely receives no canonical name, so anchors still resolve to it and a duplicate is still an error. It is listed here only because a document differing in one of those labels does not share a document ID. (A single text node is expected to carry at most one `anchor` mark; canonical names for multiple anchors on one node depend on their canonical mark order.) 6. **Preserve everything else as authored**: no other field is added, removed, defaulted, or coalesced. In particular `null` and absent are distinct, and JSON-Schema `default`s (such as `colspan`) are never materialized into hashed content. From 99895c7038153f99b098ef7e1f6ce357e8d0cdf2 Mon Sep 17 00:00:00 2001 From: Greg von Nessi Date: Mon, 31 Aug 2026 18:29:22 +0100 Subject: [PATCH 5/6] Narrow the claims to what this branch actually fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The adversarial lens found the entry overclaiming in three places and the published artifacts wrong in two. The collision CLASS is not closed. The same harm is reachable through any extension data array not literally named `entries`: a `vendor:refs` block with a `citation` array gives one document ID for two entirely different key sets. Item 5 already states the broader rule and the reference implements it as a one-element denylist. Membership also still turns on `type` versus `@type`. Both are `definedId` keying membership on shape where item 5 says position, and both are #132 — this branch closes the `attributes` path through that, not the class. "a `link` mark's href resolves into it and is rewritten" was false. Measured, the href stays verbatim, and correctly so: a preserved id takes no canonical name, so there is nothing to rewrite it to. Only resolution reaches it. The vector did not discriminate. Its id sat in a `citation` ARRAY, which item 5's data-array sentence already leaves as authored — so an implementation that never applied the `attributes` rule reproduced the expected bytes, and the vector pinned the outcome rather than the rule. It now uses a SINGULAR sub-object, which nothing but the `attributes` rule reaches, and pins the index-consumption behaviour alongside. The fixture was named `reject-` while expecting INTEGRITY-ERROR, and its description opened "REJECT." flat, contradicting its own floor. Renamed to `integrity-error-attributes-id-collision`, matching the two existing collision fixtures, and the disposition stated as the state-invariant Error it is. --- CHANGELOG.md | 4 ++-- .../case.cdx | Bin .../case.json | 2 +- conformance/vectors/canonicalize.json | 8 ++++---- scripts/lib/fixture-corpus.ts | 4 ++-- 5 files changed, 9 insertions(+), 9 deletions(-) rename conformance/fixtures/document/{reject-content-attributes-id-collision => integrity-error-attributes-id-collision}/case.cdx (100%) rename conformance/fixtures/document/{reject-content-attributes-id-collision => integrity-error-attributes-id-collision}/case.json (57%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 70f1445..4c02697 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,8 +53,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - Template generation script ### Fixed -- **An `id` carried under `attributes` is no longer relabeled, closing a silent document-ID collision.** §4.3.1 item 5 relabels author-chosen identifiers so two documents differing only in their LABELS share an id. That alpha-equivalence was reaching third-party vocabulary data, where an id is content: measured, two bibliographies whose CSL citation keys differed entirely — `smith2020`/`jones2019` versus `OTHER-KEY`/`DIFFERENT` — canonicalized to identical bytes, so **a signature over one verified the other**. Whether it fired turned on `type` versus `@type`, leaving well-formed JSON-LD accidentally immune and CSL-JSON exposed — the shape test leaking, where item 5 says membership is keyed on position. Item 5 already exempts a `semantic:bibliography` entry's `id` for exactly this reason; the same data one field over was not. **PRESERVED, not excluded**, and the distinction is the whole fix: excluding takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, which trades a loud failure for a silent one — that approach was implemented and reverted on the previous branch. A preserved id stays in the map's INPUT and out of the map, so `renameId` returns it verbatim, reusing the mechanism `semantic:term` and `semantic:footnote` already use, and inheriting item 5's canonical-name prohibition for free. **The duplicate cases still REJECT, and that is correct.** The premise that nothing can reference material under `attributes` is false as implemented: a `link` mark's `href` resolves into it and is rewritten, so a duplicate id there leaves a reference with two candidate referents. The code comment asserting “nothing anchors into it” stated a wish as a fact. One consequence is deliberate: a preserved occurrence **consumes no index**, so a real sibling block now takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Item 5 states that rule and records that reserving an index instead “would give identical content a different document ID”. Zero documents in the corpus carry one, measured. The remaining consumers of the barrier — the reference resolver's walks, and the structural id-uniqueness check — are unchanged and tracked separately; this closes the collision, not the whole barrier question. -- **Conformance suite `suiteVersion` 0.2.4 → 0.2.5.** 215 → 216 vectors, 163 → 164 fixtures, 378 → 380 checks. The vector and the fixture kill DIFFERENT mutants and both are needed: relabeling returning is caught only by the vector (`"id":"smith2020"` versus `"id":"b0"`), while excluding ids from the namespace rather than preserving them is caught only by the fixture (disposition IGNORE where INTEGRITY-ERROR is required). Nothing in the published suite caught that second case before, which is how the reverted approach reached a branch. The fixture expects a disposition RANGE rather than a pin, since `CDX-E-ID-COLLISION` is registered INTEGRITY-ERROR and the later states escalate it. The Python oracle needs no counterpart — measured, not assumed: it does not relabel, so preserved-versus-relabeled is invisible to it, and it confirms the new fixture independently. +- **An `id` carried under `attributes` is no longer relabeled, closing one path to a silent document-ID collision.** §4.3.1 item 5 relabels author-chosen identifiers so two documents differing only in their LABELS share an id. That alpha-equivalence was reaching third-party vocabulary data, where an id is content: measured, two bibliographies whose CSL citation keys differed entirely canonicalized to identical bytes, so **a signature over one verified the other**. Item 5 already exempts a `semantic:bibliography` entry's `id` for exactly this reason; the same data under `attributes` was not. **PRESERVED, not excluded**, and the distinction is the whole fix: excluding takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, trading a loud failure for a silent one — that approach was implemented and reverted on the previous branch. A preserved id stays in the map's INPUT and out of the map, so `renameId` returns it verbatim, reusing the mechanism `semantic:term` and `semantic:footnote` already use and inheriting item 5's canonical-name prohibition. §4.3.1 item 5 is amended to state the position-keyed exception, which it previously contradicted by calling the two type-keyed ones “the only namespace members whose ids are left as authored”. **The collision CLASS is not closed, and this entry should not be read as closing it.** The same harm remains reachable through any extension data array not literally named `entries`: measured, a `vendor:refs` block carrying a `citation` array gives one document ID for two entirely different key sets. Item 5 already states the broader rule — an id in an extension block's own data array is left as authored — and the reference implements it as a one-element denylist. Membership also still turns on `type` versus `@type`, so well-formed JSON-LD is out of the namespace and CSL-JSON is in. Both are the same cause, `definedId` keying membership on element shape where item 5 says position, and both are tracked separately. **The duplicate cases still REJECT, and that is correct.** The premise that nothing can reference material under `attributes` is false as implemented: a `link` mark's `href` RESOLVES against such an id — it is not rewritten, since a preserved id takes no canonical name — so a duplicate there leaves a reference with two candidate referents. One consequence is deliberate: a preserved occurrence **consumes no index**, so a real sibling block now takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Item 5 states that rule and records that reserving an index instead “would give identical content a different document ID”. Zero documents in the corpus carry one, measured. +- **Conformance suite `suiteVersion` 0.2.4 → 0.2.5.** 215 → 216 vectors, 163 → 164 fixtures, 378 → 380 checks. The vector and the fixture kill DIFFERENT mutants and both are needed: relabeling returning is caught only by the vector, while excluding ids from the namespace rather than preserving them is caught only by the fixture (disposition IGNORE where INTEGRITY-ERROR is required). Nothing in the published suite caught that second case before, which is how the reverted approach reached a branch. The vector puts its id on a SINGULAR sub-object, not in an array, and that placement is the point: item 5 separately leaves as authored an id in an extension block's own data array, so an array case is reproduced by an implementation that never applies the `attributes` rule — it would pin the outcome and not the rule. The fixture expects a disposition RANGE rather than a pin, since an id collision is an Error in all states (Anchors and References §7.2) that escalates with the document state. The Python oracle needs no counterpart — measured, not assumed: it does not relabel, so preserved-versus-relabeled is invisible to it, and it confirms the new fixture independently. - **Canonicalization no longer REJECTs conformant documents carrying text-shaped data in a position that is not block content.** Document Hashing §4.3.1 item 4 (merge adjacent text nodes) and §4.3.2 item 2 (concatenated-NFC) were implemented on element **shape** — “is this object `{type:"text",value:string}`?” — so ANY array whose members merely resembled text nodes was merged and run-checked. Both rules are now keyed on **position**: the schema-declared block-content arrays `blocks`, `caption`, `children`, `content`, `hints`, `preamble`, plus `attributes` as a **sticky** barrier. The barrier is what the shape reading gets wrong in practice: `attributes.semantic` is a JSON-LD annotation with `additionalProperties: true` (semantic.schema.json), so an author may put a member named `children` there at any depth, and clearing the flag on the way down readmits the very false REJECT the scoping removes. Only the RUN rule is withheld beneath `attributes` — it is HASHED material, so both **per-string** invariants still bind there, pinned by two REJECT fixtures; without them the scope could be widened into a wholesale exemption with every gate green, only a non-NFC KEY having been covered. The set is **keys, not (type, key) pairs**, because several content arrays hang off sub-objects carrying no `type` at all (`exercisePart`, `solution`, a phantom's content), which a type-keyed map cannot reach. **The scope is a NARROWING, and its licence is not yet in `spec/`.** §4.3 applies the transforms across all block types “uniformly” and §4.3.2 binds the rule to “a block” with no enclosing-field list, where §4.3.1 item 5 names `lines`/`subfigures` as exhaustive when it means to scope by position. Measured: a `vendor:callout` carrying `body:[text,text]` with a split combining sequence REJECTed before and loads clean now. A spec erratum is required and is tracked separately; until it lands the reference under-enforces §4.3.2 for extension blocks whose text array is not one of the six keys. Strict `children` is the specific wrong answer, and it is measurable: it drops the ROOT `blocks` array that `reject-content-split-combining-sequence-mid-array` uses AND `figure.caption`, and now breaks **eight** published cases (four vectors, four fixtures) where before this slice it broke one. **Six walks** were scoped, four in `scripts/lib/canonicalize.ts` (`canon`, `walkContentNodes`, `validateStoredByteInvariants`, `collectStoredTextViolations`) and two in `conformance/oracles/document_oracle.py` (`_walk_text_violations`, `_walk_content`). `dropAbsorbedTextNodes` and `_drop_absorbed_text_nodes` — the pair an earlier withdrawn attempt missed — are unchanged in body; what was scoped is their INVOCATION, inside `walkContentNodes` and `_walk_content` respectively. Those two model the merge for the RAW walk, and scoping the other four alone makes canon keep a node the raw walks drop. **The barrier gates the merge and the run rule, NOT id-namespace membership**, and that asymmetry is deliberate for now rather than settled. `definedId` still counts an id carried under `attributes`, so two `anchor` marks sharing one there is a REJECT even though the merge treats the same subtree as inert. Gating `definedId` as well was implemented and REVERTED: it removes ids from the namespace but `rewriteIds` still relabels them, so two documents differing only in an id under `attributes` canonicalized to identical bytes — a silent document-ID collision in place of a loud false REJECT, which is the worse trade in a content-addressed format. The barrier has to reach `definedId`, `rewriteIds` and the reference resolver together or not at all; that is tracked separately. **Scan and canon agree at every position they both reach, with one KNOWN divergence.** `canon` skips a text node's `marks`, and `content` (a `presentation:footnote` mark's content blocks) occurs only there, so at that position the run rule fires and the merge does not — split text in a footnote mark's content is reported, while two mark-less nodes there stay unmerged. That asymmetry is #121's sequel, not a licence to widen the scope, and it is stated in the published vector rather than left for an implementer to infer. `check:block-content-positions` re-derives the set from the published schemas and fails on drift, comparing each transcription against the SCHEMA rather than against the other. Its derivation is **recursive and resolves local `$ref`s**: a position declared behind a `$ref` to an array-typed `$def`, or on an inline nested object (the shape `academic:abstractBlock.sections[].children` already uses), was previously invisible — and under-derivation is the one failure the gate cannot report on itself, since a key it never saw is a key both sets agree about. Both shapes now fail it. It also compares `CONTENT_BARRIER_KEY` across the two implementations and against `blockBase`, which nothing checked before. The change is a pure relaxation with **zero id blast radius** in this repository: every id-verifying gate passes unchanged and no example or fixture exercised the path. - **The published `anchor-range` structural rule no longer false-positives on a well-ordered range above 2^53.** `checkAnchors` ordered a Content Anchor URI's bounds by converting them to numbers, so two distinct integers that round to the same double compared equal and the range was reported inverted — under a message naming the same number twice. The URI grammar admits an unbounded `[0-9]+` and §4.3.2's 2^53 limit governs hashed *numbers* rather than digits inside a string, so such a URI is conformant. This mattered beyond its own rule: the document layer's position check had the identical defect and was fixed in the same change, which left two implementations in this repo disagreeing about the exact URI a new fixture pins. The comparison is now shared between them (`scripts/lib/decimal-digits.ts`), which is also what forced the leading-zero case to be handled — `[0-9]+` admits `007`, so a length-first digit comparison would otherwise have called `#p1/007-10` inverted. Two `anchor-range` vectors pin both - Broken cross-reference in presentation layers spec (Section 5.1.2 → 5.4) diff --git a/conformance/fixtures/document/reject-content-attributes-id-collision/case.cdx b/conformance/fixtures/document/integrity-error-attributes-id-collision/case.cdx similarity index 100% rename from conformance/fixtures/document/reject-content-attributes-id-collision/case.cdx rename to conformance/fixtures/document/integrity-error-attributes-id-collision/case.cdx diff --git a/conformance/fixtures/document/reject-content-attributes-id-collision/case.json b/conformance/fixtures/document/integrity-error-attributes-id-collision/case.json similarity index 57% rename from conformance/fixtures/document/reject-content-attributes-id-collision/case.json rename to conformance/fixtures/document/integrity-error-attributes-id-collision/case.json index 715ed52..91898ef 100644 --- a/conformance/fixtures/document/reject-content-attributes-id-collision/case.json +++ b/conformance/fixtures/document/integrity-error-attributes-id-collision/case.json @@ -1,5 +1,5 @@ { - "description": "A block `id` duplicated by an `id` carried under another block's `attributes.semantic`. REJECT. An id there is preserved as authored rather than relabeled — it is third-party vocabulary data, where an id is content — but it remains a full member of the shared UNIQUENESS namespace, exactly as a `semantic:term` id does (Document Hashing section 4.3.1 item 5: \"a preserved id remains a full member of the namespace in every other respect… Only the renaming skips it\"). That is not a technicality: the material is reachable, since a `link` mark's `href` resolves into it, so two nodes carrying one id genuinely leave a reference with two candidate referents. This is the case that distinguishes \"preserved\" from \"outside the namespace\" — an implementation choosing the latter reports nothing here, and then relabels the id anyway, computing one document id for two documents whose citation keys differ.", + "description": "A block `id` duplicated by an `id` carried under another block's `attributes.semantic`. INTEGRITY-ERROR, state-invariant (Anchors and References section 7.2 makes an id collision an Error in all states), escalating with the document state as the other collision cases do. An id there is preserved as authored rather than relabeled — it is third-party vocabulary data, where an id is content — but it remains a full member of the shared UNIQUENESS namespace, exactly as a `semantic:term` id does (Document Hashing section 4.3.1 item 5: \"a preserved id remains a full member of the namespace in every other respect… Only the renaming skips it\"). That is not a technicality: the material is reachable, since a `link` mark's `href` resolves into it, so two nodes carrying one id genuinely leave a reference with two candidate referents. This is the case that distinguishes \"preserved\" from \"outside the namespace\" — an implementation choosing the latter reports nothing here, and then relabels the id anyway, computing one document id for two documents whose citation keys differ.", "layer": "document", "requires": [ "container", diff --git a/conformance/vectors/canonicalize.json b/conformance/vectors/canonicalize.json index 8e49608..2d97889 100644 --- a/conformance/vectors/canonicalize.json +++ b/conformance/vectors/canonicalize.json @@ -894,14 +894,14 @@ }, { "name": "attributes-id-preserved-not-relabeled", - "description": "An `id` carried under `attributes` is preserved as authored, not relabeled (§4.3.1 item 5). Item 5's alpha-equivalence exists so two documents differing only in author-chosen LABELS share an id; `attributes.semantic` is a JSON-LD annotation holding third-party vocabulary data, where an `id` is CONTENT — a CSL citation key a `citation` mark's `refs` resolves against. Item 5 already exempts a `semantic:bibliography` entry's `id` for that reason; this is the same data one field over. An implementation that relabels it computes the SAME document id for two bibliographies with entirely different citation keys, so a signature over one verifies the other. Note what is NOT withheld: the id stays in the shared uniqueness namespace, because that material is reachable — a `link` mark's `href` resolves into it — so a duplicate there is a real collision.", + "description": "An `id` carried under `attributes` is preserved as authored, not relabeled, and consumes no index — the sibling block takes `b0` (§4.3.1 item 5). Item 5's alpha-equivalence exists so two documents differing only in author-chosen LABELS share an id; `attributes` is a closed set of authored values (Content Blocks §3.1), so an id there is a vocabulary key a JSON-LD or CSL consumer resolves against, and relabeling it makes two documents whose keys differ entirely hash identically. The id sits on a SINGULAR sub-object deliberately. Item 5 separately leaves as authored an id in an extension block's own DATA ARRAY, so a case putting this id in an array would be reproduced by an implementation that never applied the `attributes` rule at all — it would pin the outcome and not the rule. Nothing but the `attributes` rule reaches this position. Note what is NOT withheld: the id stays in the shared uniqueness namespace, because the material is reachable — a `link` mark's `href` resolves against it — so a duplicate there is a real collision.", "parts": { "manifest": "{}", - "content": "{\"version\":\"0.1\",\"blocks\":[{\"type\":\"paragraph\",\"attributes\":{\"semantic\":{\"citation\":[{\"id\":\"smith2020\",\"type\":\"article-journal\"}]}},\"children\":[{\"type\":\"text\",\"value\":\"x\"}]}]}", + "content": "{\"version\":\"0.1\",\"blocks\":[{\"type\":\"paragraph\",\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"note\":{\"type\":\"heading\",\"id\":\"keepme\",\"children\":[]}}},\"children\":[]},{\"type\":\"paragraph\",\"id\":\"real\",\"children\":[]}]}", "dublinCore": "{\"version\":\"1.1\",\"terms\":{\"title\":\"T\",\"creator\":\"C\"}}" }, - "expectedCanonicalJcs": "{\"content\":{\"blocks\":[{\"attributes\":{\"semantic\":{\"citation\":[{\"id\":\"smith2020\",\"type\":\"article-journal\"}]}},\"children\":[{\"type\":\"text\",\"value\":\"x\"}],\"type\":\"paragraph\"}],\"version\":\"0.1\"},\"metadata\":{\"creator\":[\"C\"],\"title\":\"T\"}}", - "expectedId": "sha256:291f85c407099eb2b1bbbbff29abe4b6609059197b64b1a3929646c24bed9537" + "expectedCanonicalJcs": "{\"content\":{\"blocks\":[{\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"note\":{\"children\":[],\"id\":\"keepme\",\"type\":\"heading\"}}},\"children\":[],\"type\":\"paragraph\"},{\"children\":[],\"id\":\"b0\",\"type\":\"paragraph\"}],\"version\":\"0.1\"},\"metadata\":{\"creator\":[\"C\"],\"title\":\"T\"}}", + "expectedId": "sha256:15e27b0632507a2206be473dff14e151fdb4ec87a9ca8c96377058f06aa2a1e8" } ] } diff --git a/scripts/lib/fixture-corpus.ts b/scripts/lib/fixture-corpus.ts index 2e49029..123ae44 100644 --- a/scripts/lib/fixture-corpus.ts +++ b/scripts/lib/fixture-corpus.ts @@ -2267,8 +2267,8 @@ export const FIXTURE_CORPUS: AuthoredCase[] = [ expect: rejectOnly('CDX-E-PART-STRING-NOT-NFC'), }, { - name: 'reject-content-attributes-id-collision', - description: 'A block `id` duplicated by an `id` carried under another block\'s `attributes.semantic`. REJECT. An id there is preserved as authored rather than relabeled — it is third-party vocabulary data, where an id is content — but it remains a full member of the shared UNIQUENESS namespace, exactly as a `semantic:term` id does (Document Hashing section 4.3.1 item 5: "a preserved id remains a full member of the namespace in every other respect… Only the renaming skips it"). That is not a technicality: the material is reachable, since a `link` mark\'s `href` resolves into it, so two nodes carrying one id genuinely leave a reference with two candidate referents. This is the case that distinguishes "preserved" from "outside the namespace" — an implementation choosing the latter reports nothing here, and then relabels the id anyway, computing one document id for two documents whose citation keys differ.', + name: 'integrity-error-attributes-id-collision', + description: 'A block `id` duplicated by an `id` carried under another block\'s `attributes.semantic`. INTEGRITY-ERROR, state-invariant (Anchors and References section 7.2 makes an id collision an Error in all states), escalating with the document state as the other collision cases do. An id there is preserved as authored rather than relabeled — it is third-party vocabulary data, where an id is content — but it remains a full member of the shared UNIQUENESS namespace, exactly as a `semantic:term` id does (Document Hashing section 4.3.1 item 5: "a preserved id remains a full member of the namespace in every other respect… Only the renaming skips it"). That is not a technicality: the material is reachable, since a `link` mark\'s `href` resolves into it, so two nodes carrying one id genuinely leave a reference with two candidate referents. This is the case that distinguishes "preserved" from "outside the namespace" — an implementation choosing the latter reports nothing here, and then relabels the id anyway, computing one document id for two documents whose citation keys differ.', layer: 'document', requires: ['container', 'document'], clause: 'Document Hashing section 4.3.1 item 5; Anchors and References section 4', From 51212229607d4d521b52a005afbeeba11e8e69aa Mon Sep 17 00:00:00 2001 From: Greg von Nessi Date: Tue, 1 Sep 2026 08:46:12 +0100 Subject: [PATCH 6/6] Scope the canonical-name prohibition to type-preserved ids MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The correctness review found the branch rejecting documents main accepts, measured across 30,000 generated documents: 1,590 regressions, all one error class, and the only new rejection class on the branch. Extending item 5's `b` prohibition to position-preserved ids made third-party vocabulary keys a hard canonicalization error. `b1`, `b2` is the CrossRef/JATS reference convention, so CSL exported from such a source carries exactly those keys — the data class this branch exists to protect. Worse, it was asymmetric: the identical CSL object in a `semantic:bibliography` `entries` array was accepted and kept its keys, while under `attributes` it was rejected. The erratum's own prose calls those two "the same reasoning applied to the same data one field over"; the code did the opposite. `collectDefinedIds` now reports `preservedByType` alongside `preserved`, and only the type-keyed set carries the prohibition. The two positions now behave identically, which is what the prose already claimed. Item 5 is corrected in two places. The prohibition sentence now says it does not extend to a position-preserved id, with the reason. And the membership sentence said "An id carried under `attributes`", whose plain reading makes every id there a namespace member — including an untyped array item, which the reference accepts as a duplicate. It now reads "An id that would otherwise be a member of this namespace", matching what `definedId` admits. The raw walk's `preserved` output was asserted by nothing: two barrier mutations survived all 30 gates. The differential now compares `preserved` as well as `duplicates` — its docstring already claimed to assert the raw walk reproduces canon, and it was checking one field of three. Two cases were added to reach it, since every existing `attributes` row REJECTs and the comparison skips those; the second sits in a mark INSIDE `attributes`, the only shape that catches the raw walk's dedicated `marks` branch dropping the flag. Both mutants now die. --- CHANGELOG.md | 2 +- scripts/lib/canonicalize.ts | 26 +++++++++++------- scripts/test-canonicalize.ts | 45 +++++++++++++++++++++++++++----- spec/core/06-document-hashing.md | 2 +- 4 files changed, 57 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c02697..6f38fc3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,7 +53,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - Template generation script ### Fixed -- **An `id` carried under `attributes` is no longer relabeled, closing one path to a silent document-ID collision.** §4.3.1 item 5 relabels author-chosen identifiers so two documents differing only in their LABELS share an id. That alpha-equivalence was reaching third-party vocabulary data, where an id is content: measured, two bibliographies whose CSL citation keys differed entirely canonicalized to identical bytes, so **a signature over one verified the other**. Item 5 already exempts a `semantic:bibliography` entry's `id` for exactly this reason; the same data under `attributes` was not. **PRESERVED, not excluded**, and the distinction is the whole fix: excluding takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, trading a loud failure for a silent one — that approach was implemented and reverted on the previous branch. A preserved id stays in the map's INPUT and out of the map, so `renameId` returns it verbatim, reusing the mechanism `semantic:term` and `semantic:footnote` already use and inheriting item 5's canonical-name prohibition. §4.3.1 item 5 is amended to state the position-keyed exception, which it previously contradicted by calling the two type-keyed ones “the only namespace members whose ids are left as authored”. **The collision CLASS is not closed, and this entry should not be read as closing it.** The same harm remains reachable through any extension data array not literally named `entries`: measured, a `vendor:refs` block carrying a `citation` array gives one document ID for two entirely different key sets. Item 5 already states the broader rule — an id in an extension block's own data array is left as authored — and the reference implements it as a one-element denylist. Membership also still turns on `type` versus `@type`, so well-formed JSON-LD is out of the namespace and CSL-JSON is in. Both are the same cause, `definedId` keying membership on element shape where item 5 says position, and both are tracked separately. **The duplicate cases still REJECT, and that is correct.** The premise that nothing can reference material under `attributes` is false as implemented: a `link` mark's `href` RESOLVES against such an id — it is not rewritten, since a preserved id takes no canonical name — so a duplicate there leaves a reference with two candidate referents. One consequence is deliberate: a preserved occurrence **consumes no index**, so a real sibling block now takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Item 5 states that rule and records that reserving an index instead “would give identical content a different document ID”. Zero documents in the corpus carry one, measured. +- **An `id` carried under `attributes` is no longer relabeled, closing one path to a silent document-ID collision.** §4.3.1 item 5 relabels author-chosen identifiers so two documents differing only in their LABELS share an id. That alpha-equivalence was reaching third-party vocabulary data, where an id is content: measured, two bibliographies whose CSL citation keys differed entirely canonicalized to identical bytes, so **a signature over one verified the other**. Item 5 already exempts a `semantic:bibliography` entry's `id` for exactly this reason; the same data under `attributes` was not. **PRESERVED, not excluded**, and the distinction is the whole fix: excluding takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, trading a loud failure for a silent one — that approach was implemented and reverted on the previous branch. A preserved id stays in the map's INPUT and out of the map, so `renameId` returns it verbatim, reusing the mechanism `semantic:term` and `semantic:footnote` already use. It does NOT inherit item 5's canonical-name prohibition, which stays scoped to the type-keyed exception: `b1`, `b2` is a common reference-numbering convention, so CSL exported from such a source carries exactly those keys, and rejecting them would fail the documents this rule exists to protect — while the identical data in an extension's own data array is accepted. An earlier draft of this change did reject them. §4.3.1 item 5 is amended to state the position-keyed exception, which it previously contradicted by calling the two type-keyed ones “the only namespace members whose ids are left as authored”. **The collision CLASS is not closed, and this entry should not be read as closing it.** The same harm remains reachable through any extension data array not literally named `entries`: measured, a `vendor:refs` block carrying a `citation` array gives one document ID for two entirely different key sets. Item 5 already states the broader rule — an id in an extension block's own data array is left as authored — and the reference implements it as a one-element denylist. Membership also still turns on `type` versus `@type`, so well-formed JSON-LD is out of the namespace and CSL-JSON is in. Both are the same cause, `definedId` keying membership on element shape where item 5 says position, and both are tracked separately. **The duplicate cases still REJECT, and that is correct.** The premise that nothing can reference material under `attributes` is false as implemented: a `link` mark's `href` RESOLVES against such an id — it is not rewritten, since a preserved id takes no canonical name — so a duplicate there leaves a reference with two candidate referents. One consequence is deliberate: a preserved occurrence **consumes no index**, so a real sibling block now takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Item 5 states that rule and records that reserving an index instead “would give identical content a different document ID”. Zero documents in the corpus carry one, measured. - **Conformance suite `suiteVersion` 0.2.4 → 0.2.5.** 215 → 216 vectors, 163 → 164 fixtures, 378 → 380 checks. The vector and the fixture kill DIFFERENT mutants and both are needed: relabeling returning is caught only by the vector, while excluding ids from the namespace rather than preserving them is caught only by the fixture (disposition IGNORE where INTEGRITY-ERROR is required). Nothing in the published suite caught that second case before, which is how the reverted approach reached a branch. The vector puts its id on a SINGULAR sub-object, not in an array, and that placement is the point: item 5 separately leaves as authored an id in an extension block's own data array, so an array case is reproduced by an implementation that never applies the `attributes` rule — it would pin the outcome and not the rule. The fixture expects a disposition RANGE rather than a pin, since an id collision is an Error in all states (Anchors and References §7.2) that escalates with the document state. The Python oracle needs no counterpart — measured, not assumed: it does not relabel, so preserved-versus-relabeled is invisible to it, and it confirms the new fixture independently. - **Canonicalization no longer REJECTs conformant documents carrying text-shaped data in a position that is not block content.** Document Hashing §4.3.1 item 4 (merge adjacent text nodes) and §4.3.2 item 2 (concatenated-NFC) were implemented on element **shape** — “is this object `{type:"text",value:string}`?” — so ANY array whose members merely resembled text nodes was merged and run-checked. Both rules are now keyed on **position**: the schema-declared block-content arrays `blocks`, `caption`, `children`, `content`, `hints`, `preamble`, plus `attributes` as a **sticky** barrier. The barrier is what the shape reading gets wrong in practice: `attributes.semantic` is a JSON-LD annotation with `additionalProperties: true` (semantic.schema.json), so an author may put a member named `children` there at any depth, and clearing the flag on the way down readmits the very false REJECT the scoping removes. Only the RUN rule is withheld beneath `attributes` — it is HASHED material, so both **per-string** invariants still bind there, pinned by two REJECT fixtures; without them the scope could be widened into a wholesale exemption with every gate green, only a non-NFC KEY having been covered. The set is **keys, not (type, key) pairs**, because several content arrays hang off sub-objects carrying no `type` at all (`exercisePart`, `solution`, a phantom's content), which a type-keyed map cannot reach. **The scope is a NARROWING, and its licence is not yet in `spec/`.** §4.3 applies the transforms across all block types “uniformly” and §4.3.2 binds the rule to “a block” with no enclosing-field list, where §4.3.1 item 5 names `lines`/`subfigures` as exhaustive when it means to scope by position. Measured: a `vendor:callout` carrying `body:[text,text]` with a split combining sequence REJECTed before and loads clean now. A spec erratum is required and is tracked separately; until it lands the reference under-enforces §4.3.2 for extension blocks whose text array is not one of the six keys. Strict `children` is the specific wrong answer, and it is measurable: it drops the ROOT `blocks` array that `reject-content-split-combining-sequence-mid-array` uses AND `figure.caption`, and now breaks **eight** published cases (four vectors, four fixtures) where before this slice it broke one. **Six walks** were scoped, four in `scripts/lib/canonicalize.ts` (`canon`, `walkContentNodes`, `validateStoredByteInvariants`, `collectStoredTextViolations`) and two in `conformance/oracles/document_oracle.py` (`_walk_text_violations`, `_walk_content`). `dropAbsorbedTextNodes` and `_drop_absorbed_text_nodes` — the pair an earlier withdrawn attempt missed — are unchanged in body; what was scoped is their INVOCATION, inside `walkContentNodes` and `_walk_content` respectively. Those two model the merge for the RAW walk, and scoping the other four alone makes canon keep a node the raw walks drop. **The barrier gates the merge and the run rule, NOT id-namespace membership**, and that asymmetry is deliberate for now rather than settled. `definedId` still counts an id carried under `attributes`, so two `anchor` marks sharing one there is a REJECT even though the merge treats the same subtree as inert. Gating `definedId` as well was implemented and REVERTED: it removes ids from the namespace but `rewriteIds` still relabels them, so two documents differing only in an id under `attributes` canonicalized to identical bytes — a silent document-ID collision in place of a loud false REJECT, which is the worse trade in a content-addressed format. The barrier has to reach `definedId`, `rewriteIds` and the reference resolver together or not at all; that is tracked separately. **Scan and canon agree at every position they both reach, with one KNOWN divergence.** `canon` skips a text node's `marks`, and `content` (a `presentation:footnote` mark's content blocks) occurs only there, so at that position the run rule fires and the merge does not — split text in a footnote mark's content is reported, while two mark-less nodes there stay unmerged. That asymmetry is #121's sequel, not a licence to widen the scope, and it is stated in the published vector rather than left for an implementer to infer. `check:block-content-positions` re-derives the set from the published schemas and fails on drift, comparing each transcription against the SCHEMA rather than against the other. Its derivation is **recursive and resolves local `$ref`s**: a position declared behind a `$ref` to an array-typed `$def`, or on an inline nested object (the shape `academic:abstractBlock.sections[].children` already uses), was previously invisible — and under-derivation is the one failure the gate cannot report on itself, since a key it never saw is a key both sets agree about. Both shapes now fail it. It also compares `CONTENT_BARRIER_KEY` across the two implementations and against `blockBase`, which nothing checked before. The change is a pure relaxation with **zero id blast radius** in this repository: every id-verifying gate passes unchanged and no example or fixture exercised the path. - **The published `anchor-range` structural rule no longer false-positives on a well-ordered range above 2^53.** `checkAnchors` ordered a Content Anchor URI's bounds by converting them to numbers, so two distinct integers that round to the same double compared equal and the range was reported inverted — under a message naming the same number twice. The URI grammar admits an unbounded `[0-9]+` and §4.3.2's 2^53 limit governs hashed *numbers* rather than digits inside a string, so such a URI is conformant. This mattered beyond its own rule: the document layer's position check had the identical defect and was fixed in the same change, which left two implementations in this repo disagreeing about the exact URI a new fixture pins. The comparison is now shared between them (`scripts/lib/decimal-digits.ts`), which is also what forced the leading-zero case to be handled — `[0-9]+` admits `007`, so a length-first digit comparison would otherwise have called `#p1/007-10` inverted. Two `anchor-range` vectors pin both diff --git a/scripts/lib/canonicalize.ts b/scripts/lib/canonicalize.ts index 9b8c412..3e4a389 100644 --- a/scripts/lib/canonicalize.ts +++ b/scripts/lib/canonicalize.ts @@ -705,6 +705,14 @@ export interface CollectedIds { * wants `ids`; only `alphaRenameIds` wants this. */ preserved: Set; + /** + * The subset of `preserved` held back by TYPE (`PRESERVED_ID_BLOCK_TYPES`) rather than by + * position. Only these carry item 5's canonical-name prohibition: an id under `attributes` is + * third-party vocabulary data the author does not choose — `b1`, `b2` is the CrossRef/JATS + * reference convention — and rejecting it would fail the documents this scoping protects, + * while the identical data in an `entries` array is accepted. + */ + preservedByType: Set; /** * Whatever `CollectOptions.classify` returned for each id, keyed by id. Empty when no * classifier was supplied. @@ -797,6 +805,7 @@ export function collectDefinedIds(content: unknown, options: CollectOptions = {} const ids: string[] = []; const duplicates: string[] = []; const preserved = new Set(); + const preservedByType = new Set(); const classified = new Map(); walkContentNodes( @@ -810,9 +819,9 @@ export function collectDefinedIds(content: unknown, options: CollectOptions = {} ids.push(id); // Recorded on the FIRST occurrence only, alongside `ids` — a repeat is a // duplicate and throws before any of this is consumed. - if (ctx.blocked || (!ctx.inMarks && typeof node.type === 'string' && PRESERVED_ID_BLOCK_TYPES.has(node.type))) { - preserved.add(id); - } + const byType = !ctx.inMarks && typeof node.type === 'string' && PRESERVED_ID_BLOCK_TYPES.has(node.type); + if (ctx.blocked || byType) preserved.add(id); + if (byType) preservedByType.add(id); // Classified on the SAME first-occurrence branch as `ids`, so the two cannot // disagree about which ids exist or about which node defines one. if (options.classify !== undefined) classified.set(id, options.classify(node, ctx)); @@ -821,7 +830,7 @@ export function collectDefinedIds(content: unknown, options: CollectOptions = {} options, ); - return { ids, duplicates, preserved, classified }; + return { ids, duplicates, preserved, preservedByType, classified }; } /** Where a visited node sits, which is what decides namespace membership (`definedId`). */ @@ -1103,7 +1112,7 @@ function alphaRenameIds(content: unknown): unknown { // Canonical content: `canon` has already stripped derived fields and deduped each // text node's marks, so the default (non-raw) collection is exactly pass 1 of the // original in-place traversal. - const { ids, duplicates, preserved } = collectDefinedIds(content); + const { ids, duplicates, preserved, preservedByType } = collectDefinedIds(content); if (duplicates.length > 0) { throw new CanonicalizationError(`duplicate id "${duplicates[0]}" in the shared identifier namespace`); } @@ -1113,12 +1122,11 @@ function alphaRenameIds(content: unknown): unknown { // not collide with the `b0` the sequence below generates for some other block, and // both would reach the canonical output carrying the same id — with a `#b0` // reference silently binding to whichever the reader met first. - for (const id of preserved) { + for (const id of preservedByType) { if (CANONICAL_NAME.test(id)) { throw new CanonicalizationError( - `id "${id}" uses the reserved canonical-name form: an id left as authored — carried under ` + - `\`${CONTENT_BARRIER_KEY}\`, or on a ${[...PRESERVED_ID_BLOCK_TYPES].sort().join(' / ')} block ` + - `— must not spell "b"`, + `id "${id}" uses the reserved canonical-name form: the id of a block whose type is left as authored ` + + `(${[...PRESERVED_ID_BLOCK_TYPES].sort().join(', ')}) must not spell "b"`, ); } } diff --git a/scripts/test-canonicalize.ts b/scripts/test-canonicalize.ts index 0b89ac5..d56ae19 100644 --- a/scripts/test-canonicalize.ts +++ b/scripts/test-canonicalize.ts @@ -1403,10 +1403,30 @@ test('collectDefinedIds: rawInput reproduces canon, checked AGAINST canon', () = ['duplicate block id under attributes.semantic', { blocks: [{ type: 'paragraph', attributes: { semantic: { '@type': 'Q', children: [{ type: 'paragraph', id: 'dup', children: [] }, { type: 'paragraph', id: 'dup', children: [] }] } }, children: [] }] }], + // The two rows above both REJECT, so the `preserved` comparison below skips them. These do + // not, which is what lets it reach the barrier at all — without them two barrier mutations + // survive every gate in the repository. + ['unique id under attributes.semantic', { blocks: [ + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', n: { type: 'heading', id: 'kept', children: [] } } }, children: [] }, + { type: 'paragraph', id: 'real', children: [] }] }], + // The barrier must survive the raw walk's dedicated `marks` branch. It only bites when + // `blocked` is ALREADY set, so the mark has to sit INSIDE `attributes`, not carry it. + ['unique anchor id in marks INSIDE attributes', { blocks: [ + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', children: [ + { type: 'text', value: 'x', marks: [{ type: 'anchor', id: 'deep' }] }] } }, children: [] }, + { type: 'paragraph', id: 'real', children: [] }] }], ]; for (const [label, content] of cases) { - const { duplicates } = collectDefinedIds(content, { rawInput: true }); - assert.equal(duplicates.length > 0, rejectsForDuplicateId(content), `raw walk disagrees with canonicalization: ${label}`); + const raw = collectDefinedIds(content, { rawInput: true }); + assert.equal(raw.duplicates.length > 0, rejectsForDuplicateId(content), `raw walk disagrees with canonicalization: ${label}`); + + // `preserved` too, not only `duplicates`. The docstring above says this differential asserts + // the raw walk reproduces canon; comparing one field of three made that broader than what it + // checked, and two barrier mutations survived every gate in the repository because of it. + if (!rejectsForDuplicateId(content)) { + const canonical = collectDefinedIds((canonicalContent({ manifest: '{}', content: JSON.stringify(content), dublinCore: '{}' }) as any).content); + assert.deepEqual([...raw.preserved].sort(), [...canonical.preserved].sort(), `raw vs canonical preserved: ${label}`); + } } }); @@ -1481,11 +1501,22 @@ test('an id under `attributes` is preserved, not relabeled', () => { children: [{ type: 'text', value: 'x', marks: [{ type: 'link', href: '#hidden' }] }] }] }); assert.equal(linked.content.blocks[0].children[0].marks[0].href, '#hidden'); - // Inherits item 5's canonical-name prohibition: left as authored beside a generated `b0`, - // two identifiers would reach the canonical output under one name. - assert.throws(() => computeDocumentId(parts({ blocks: [{ type: 'paragraph', - attributes: { semantic: { '@type': 'Q', n: { type: 'heading', id: 'b0', children: [] } } }, children: [] }] }), 'sha256'), - /reserved canonical-name form/); + // NOT subject to item 5's canonical-name prohibition, which is scoped to the TYPE-preserved + // blocks. `b1`, `b2` is the CrossRef/JATS reference convention, so CSL exported from such a + // source carries exactly these keys — rejecting them would fail the documents this scoping + // exists to protect, and the identical data in an `entries` array is accepted. + const cslKeys = (where: 'attributes' | 'entries') => where === 'attributes' + ? { blocks: [{ type: 'paragraph', attributes: { semantic: { citation: [{ id: 'b1', type: 'article-journal' }] } }, children: [] }, { type: 'paragraph', id: 'p', children: [] }] } + : { blocks: [{ type: 'semantic:bibliography', entries: [{ id: 'b1', type: 'article-journal' }] }, { type: 'paragraph', id: 'p', children: [] }] }; + for (const where of ['attributes', 'entries'] as const) { + const out = canon(cslKeys(where)) as any; + assert.doesNotThrow(() => computeDocumentId(parts(cslKeys(where)), 'sha256'), `${where}: a b vocabulary key is not an error`); + assert.equal(out.content.blocks[1].id, 'b0', `${where}: the real block still relabels`); + } + + // The prohibition still binds where item 5 scopes it — a TYPE-preserved block id. + assert.throws(() => computeDocumentId(parts({ blocks: [ + { type: 'semantic:term', id: 'b0', children: [] }] }), 'sha256'), /reserved canonical-name form/); // CONTROL. Ordinary content must be untouched by all of the above — without this the // assertions are equally satisfied by a canonicalizer that stopped relabeling entirely. diff --git a/spec/core/06-document-hashing.md b/spec/core/06-document-hashing.md index 97be348..9a5c656 100644 --- a/spec/core/06-document-hashing.md +++ b/spec/core/06-document-hashing.md @@ -167,7 +167,7 @@ The canonical content is a *transform* of the stored parts; the stored files are 2. **Resolve asset references**: replace every content-level asset *path* reference — an `image.src`, `svg.src`, or `signature.image`, or an archive-relative `href` on a `link` mark — with the referenced asset's content hash (`algorithm:hexdigest`). A reference resolves iff, after path normalization (no `.` or `..` segments; case-sensitive), it equals an asset's archive path — that asset's category directory (`assets/` followed by its category, i.e. the key under `manifest.assets` that registers it: the standard `images`, `fonts`, or `embeds`, or any additional category) joined with the asset's index `path` (Asset Embedding, section 3). The following are not asset paths and are left verbatim: an `href` beginning with `#` (an internal Content Anchor reference), a reference marked `external`, and any reference carrying a URL scheme (e.g. `https://`); an `svg` with inline `content` and no `src` has nothing to resolve; and an `image.fallback` path is left verbatim, being a backup rather than the bound resource. A reference beginning with `assets/` that matches no registered asset is a canonicalization error. The document ID thereby inherits the integrity of the asset index `hash` values, which MUST be verified against the asset bytes (Asset Embedding, section 8) before the ID can be trusted. 3. **Normalize marks**: within each text node, sort the `marks` array by each mark's JCS serialization (UTF-16 code-unit comparison — so bare formatting marks such as `"bold"` sort before structured marks such as `{"type":"link",…}`), remove marks with identical JCS serialization (deduplicate), and omit the `marks` key entirely when the array is empty (absent ≡ `[]`). Mark order is not semantic. 4. **Merge text nodes**: merge adjacent sibling text nodes that have no intervening non-text inline child and identical canonical mark-sets, concatenating their `value`s. Only a text node whose sole keys are `type`, `value`, and `marks` (after the step 1 stripping) is merge-eligible; a text node that also carries an `id`, `attributes`, or any other field is preserved unchanged and acts as a boundary, so no identifier or annotation is dropped. Offsets are defined over the concatenated text content (Anchors and References, section 3), so this moves no anchor. This rule and section 4.3.2 item 2 apply only within a **block-content array** — an array reached through one of the field names `blocks`, `caption`, `children`, `content`, `hints` or `preamble`, which is the exhaustive set. Membership is keyed on that field NAME and not on the owner's type: several such arrays hang off sub-objects carrying no `type` at all — a `subfigure`, an `academic:exercise` part, a `solution`, a structured abstract's section — and the content root, which is not a block, owns one. An implementation keying on the owner rather than the field misses them and computes a different document ID. Membership is keyed on the enclosing field, not on element shape: a text-shaped object an extension block carries in any other array is not block content and is left as authored, and `attributes` is a barrier at every depth beneath it — a closed set of authored values (Content Blocks, section 3.1) whose members are data even when shaped like inline content. An extension block therefore receives these transforms on a rich-text array exactly when it names that array with one of these keys; an array named anything else carries text the transforms do not reach, and a document containing one is still conformant. The two rules do not reach identical positions: this rule does not recurse into a text node's `marks`, so it never reaches a `presentation:footnote` mark's `content`, while section 4.3.2 item 2 does. -5. **Canonicalize identifiers**: relabel author-chosen identifiers to position-based canonical names, so that two documents differing only in their id *labels* produce the same document ID. A text node's `id` is outside this namespace entirely: it is never relabeled, occupies no index, and is not compared for duplicates — item 4 preserves it unchanged because it is content rather than a label. Because it survives into the canonical content, it MUST NOT spell a canonical name (`b` followed by digits); one that does is a canonicalization error, for the same reason the prohibition below applies to a preserved id — left as authored beside a generated `b0`, two identifiers reach the canonical output under one name. The relabeled namespace is the shared identifier namespace (Anchors and References, section 4): the `id` of every block other than a text node (including `namespace:type` extension blocks), the `id` of every `anchor` mark, and the `id` of every in-content sub-block element that carries one — an academic equation line and a `subfigure` — even though those carry no block `type`. These sub-block elements are the array items of an equation group's `lines` and a figure's `subfigures` — the exhaustive set of relabeled sub-block id arrays. Membership is keyed on WHERE an id sits (the enclosing field), not merely on whether the node has a `type` or is an array item: an id-bearing object an extension block carries in its own data array (any array other than `lines`/`subfigures`) is left as authored, even when shaped like a namespace member. In particular a `semantic:bibliography` entry's `id` — a CSL citation key in an `entries` array, referenced by a `citation` mark's `refs` (below) — is not relabeled, though the entry carries a CSL `type` that makes it resemble a block. Likewise an identifier carried on a singular named sub-object — such as a signature block's `signer` `id`, a Person identifier rather than a content-anchor target — is not part of this namespace. An id carried under `attributes` is left as authored, keyed on POSITION: item 4 makes that a barrier at every depth, and Content Blocks section 3.1 closes it to authored values, so an id there is content — a vocabulary key a JSON-LD or CSL consumer resolves against — and not a label this rule may rename. Relabeling it applies alpha-equivalence to data the author did not choose as a label: two documents whose bibliographic citation keys differ entirely would otherwise share a document ID, so a signature over one verifies the other. This is the same reasoning that exempts a `semantic:bibliography` entry's `id` above, applied to the same data one field over. Two extension block types are a further exception, keyed on TYPE rather than position: `semantic:term` and `semantic:footnote`. Each is the **defining** side of a reference that addresses a separate namespace and is itself never relabeled — a `glossary` mark's `ref` and a `semantic:term` `see` resolve against a `semantic:term` block's `id` when no external glossary is declared, and a `footnote` mark resolves against a `semantic:footnote` block (Semantic Extension sections 4.5.2 and 8.3) — so relabeling the definition while leaving the reference as authored left those rules unsatisfiable on the canonical form: the block became `b0` while the reference still named `term-algorithm`. A preserved id — by position or by type — remains a full member of the namespace in every other respect: it occupies the shared uniqueness space, a duplicate is a canonicalization error exactly as above, and a Content Anchor URI resolves against it exactly as against a relabeled block id. Only the renaming skips it. Because it is left as authored, a preserved id MUST NOT itself spell a canonical name (`b` followed by digits); one that does is a canonicalization error. This mirrors the prohibition on unresolved references below and holds for the same reason: duplicate detection compares *authored* ids, so an authored `b0` does not collide with the `b0` generated for some other block, and both would reach the canonical output under one identifier. Membership in the TYPE-keyed exception is keyed on the block's `type` rather than on where the id sits — unlike the rest of this item, and of necessity, since such an id sits directly on its block, exactly where any other block id sits. The `attributes` exception above is keyed on position, as the rest of this item is. Walk the canonical content depth-first — at each node assigning a name to the node's own `id` (if any) before recursing into its members (object members in JCS key order, array elements in index order) — and give each id-defining occurrence the next name in the sequence `b0`, `b1`, `b2`, …; a duplicate id within this namespace is a canonicalization error. An occurrence whose id is preserved takes no name and **consumes no index**: the sequence continues at the same name for the next relabeled occurrence, so a `semantic:term` followed by a paragraph yields `b0` on the paragraph. Reserving an index for the preserved occurrence instead is equally implementable and would give identical content a different document ID, so the choice is stated here rather than left to the implementation. Then rewrite every Content Anchor URI reference (`#id`, optionally followed by `/offset` or `/start-end`) whose id is one of these names — preserving the suffix — in the following reference fields: a `link` mark `href`; the `academic:theorem-ref`, `academic:equation-ref`, and `academic:algorithm-ref` mark `target`; the `academic:theorem` `uses`; the `academic:proof` `of`; and the `semantic:ref` and `presentation:reference` block `target`. A Content Anchor URI whose id is not a relabeled id — for example a cross-document anchor — is left unchanged, except that such an unresolved reference MUST NOT itself spell a canonical name (`b` followed by digits): an unresolved `#b0` is a canonicalization error, because left unchanged it would be indistinguishable from a reference that resolved to that block, collapsing two distinct documents onto one ID. Because relabeling an `anchor` id or a `link` `href` changes that mark's serialization, the marks of any affected text node are re-sorted and de-duplicated (item 3) after rewriting. +5. **Canonicalize identifiers**: relabel author-chosen identifiers to position-based canonical names, so that two documents differing only in their id *labels* produce the same document ID. A text node's `id` is outside this namespace entirely: it is never relabeled, occupies no index, and is not compared for duplicates — item 4 preserves it unchanged because it is content rather than a label. Because it survives into the canonical content, it MUST NOT spell a canonical name (`b` followed by digits); one that does is a canonicalization error, for the same reason the prohibition below applies to a preserved id — left as authored beside a generated `b0`, two identifiers reach the canonical output under one name. The relabeled namespace is the shared identifier namespace (Anchors and References, section 4): the `id` of every block other than a text node (including `namespace:type` extension blocks), the `id` of every `anchor` mark, and the `id` of every in-content sub-block element that carries one — an academic equation line and a `subfigure` — even though those carry no block `type`. These sub-block elements are the array items of an equation group's `lines` and a figure's `subfigures` — the exhaustive set of relabeled sub-block id arrays. Membership is keyed on WHERE an id sits (the enclosing field), not merely on whether the node has a `type` or is an array item: an id-bearing object an extension block carries in its own data array (any array other than `lines`/`subfigures`) is left as authored, even when shaped like a namespace member. In particular a `semantic:bibliography` entry's `id` — a CSL citation key in an `entries` array, referenced by a `citation` mark's `refs` (below) — is not relabeled, though the entry carries a CSL `type` that makes it resemble a block. Likewise an identifier carried on a singular named sub-object — such as a signature block's `signer` `id`, a Person identifier rather than a content-anchor target — is not part of this namespace. An id that would otherwise be a member of this namespace is left as authored when carried under `attributes`, keyed on POSITION: item 4 makes that a barrier at every depth, and Content Blocks section 3.1 closes it to authored values, so an id there is content — a vocabulary key a JSON-LD or CSL consumer resolves against — and not a label this rule may rename. Relabeling it applies alpha-equivalence to data the author did not choose as a label: two documents whose bibliographic citation keys differ entirely would otherwise share a document ID, so a signature over one verifies the other. This is the same reasoning that exempts a `semantic:bibliography` entry's `id` above, applied to the same data one field over. Two extension block types are a further exception, keyed on TYPE rather than position: `semantic:term` and `semantic:footnote`. Each is the **defining** side of a reference that addresses a separate namespace and is itself never relabeled — a `glossary` mark's `ref` and a `semantic:term` `see` resolve against a `semantic:term` block's `id` when no external glossary is declared, and a `footnote` mark resolves against a `semantic:footnote` block (Semantic Extension sections 4.5.2 and 8.3) — so relabeling the definition while leaving the reference as authored left those rules unsatisfiable on the canonical form: the block became `b0` while the reference still named `term-algorithm`. A preserved id — by position or by type — remains a full member of the namespace in every other respect: it occupies the shared uniqueness space, a duplicate is a canonicalization error exactly as above, and a Content Anchor URI resolves against it exactly as against a relabeled block id. Only the renaming skips it. Because it is left as authored, a TYPE-preserved id MUST NOT itself spell a canonical name (`b` followed by digits); one that does is a canonicalization error. This does not extend to an id preserved by POSITION: such an id is third-party vocabulary the author did not choose — `b1`, `b2` is a common reference-numbering convention — and rejecting it would fail exactly the documents the position rule exists to protect, while the same key in an extension block's own data array carries no such prohibition. This mirrors the prohibition on unresolved references below and holds for the same reason: duplicate detection compares *authored* ids, so an authored `b0` does not collide with the `b0` generated for some other block, and both would reach the canonical output under one identifier. Membership in the TYPE-keyed exception is keyed on the block's `type` rather than on where the id sits — unlike the rest of this item, and of necessity, since such an id sits directly on its block, exactly where any other block id sits. The `attributes` exception above is keyed on position, as the rest of this item is. Walk the canonical content depth-first — at each node assigning a name to the node's own `id` (if any) before recursing into its members (object members in JCS key order, array elements in index order) — and give each id-defining occurrence the next name in the sequence `b0`, `b1`, `b2`, …; a duplicate id within this namespace is a canonicalization error. An occurrence whose id is preserved takes no name and **consumes no index**: the sequence continues at the same name for the next relabeled occurrence, so a `semantic:term` followed by a paragraph yields `b0` on the paragraph. Reserving an index for the preserved occurrence instead is equally implementable and would give identical content a different document ID, so the choice is stated here rather than left to the implementation. Then rewrite every Content Anchor URI reference (`#id`, optionally followed by `/offset` or `/start-end`) whose id is one of these names — preserving the suffix — in the following reference fields: a `link` mark `href`; the `academic:theorem-ref`, `academic:equation-ref`, and `academic:algorithm-ref` mark `target`; the `academic:theorem` `uses`; the `academic:proof` `of`; and the `semantic:ref` and `presentation:reference` block `target`. A Content Anchor URI whose id is not a relabeled id — for example a cross-document anchor — is left unchanged, except that such an unresolved reference MUST NOT itself spell a canonical name (`b` followed by digits): an unresolved `#b0` is a canonicalization error, because left unchanged it would be indistinguishable from a reference that resolved to that block, collapsing two distinct documents onto one ID. Because relabeling an `anchor` id or a `link` `href` changes that mark's serialization, the marks of any affected text node are re-sorted and de-duplicated (item 3) after rewriting. The following are left as authored, so a document that differs only in one of these labels is not yet guaranteed to share a document ID: the reference fields that address other namespaces — the `footnote` mark `id`, the `glossary` mark `ref`, `semantic:term` `see`, `citation` `refs`, `entity` `uri`, `index` `term`, and the `academic:algorithm-ref` `line` (a label into an algorithm's line numbering, not an `id`). Equation-line ids and subfigure ids are not in this set: they share the relabeled uniqueness namespace described above. A `semantic:footnote` or `semantic:term` block's own `id` is left as authored as well, but for a different reason and with a different consequence: it does not address another namespace — it stays in the shared uniqueness-and-anchor namespace described above and merely receives no canonical name, so anchors still resolve to it and a duplicate is still an error. It is listed here only because a document differing in one of those labels does not share a document ID. (A single text node is expected to carry at most one `anchor` mark; canonical names for multiple anchors on one node depend on their canonical mark order.) 6. **Preserve everything else as authored**: no other field is added, removed, defaulted, or coalesced. In particular `null` and absent are distinct, and JSON-Schema `default`s (such as `colspan`) are never materialized into hashed content.