diff --git a/CHANGELOG.md b/CHANGELOG.md index eb31f66..6f38fc3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - Template generation script ### Fixed +- **An `id` carried under `attributes` is no longer relabeled, closing one path to a silent document-ID collision.** §4.3.1 item 5 relabels author-chosen identifiers so two documents differing only in their LABELS share an id. That alpha-equivalence was reaching third-party vocabulary data, where an id is content: measured, two bibliographies whose CSL citation keys differed entirely canonicalized to identical bytes, so **a signature over one verified the other**. Item 5 already exempts a `semantic:bibliography` entry's `id` for exactly this reason; the same data under `attributes` was not. **PRESERVED, not excluded**, and the distinction is the whole fix: excluding takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, trading a loud failure for a silent one — that approach was implemented and reverted on the previous branch. A preserved id stays in the map's INPUT and out of the map, so `renameId` returns it verbatim, reusing the mechanism `semantic:term` and `semantic:footnote` already use. It does NOT inherit item 5's canonical-name prohibition, which stays scoped to the type-keyed exception: `b1`, `b2` is a common reference-numbering convention, so CSL exported from such a source carries exactly those keys, and rejecting them would fail the documents this rule exists to protect — while the identical data in an extension's own data array is accepted. An earlier draft of this change did reject them. §4.3.1 item 5 is amended to state the position-keyed exception, which it previously contradicted by calling the two type-keyed ones “the only namespace members whose ids are left as authored”. **The collision CLASS is not closed, and this entry should not be read as closing it.** The same harm remains reachable through any extension data array not literally named `entries`: measured, a `vendor:refs` block carrying a `citation` array gives one document ID for two entirely different key sets. Item 5 already states the broader rule — an id in an extension block's own data array is left as authored — and the reference implements it as a one-element denylist. Membership also still turns on `type` versus `@type`, so well-formed JSON-LD is out of the namespace and CSL-JSON is in. Both are the same cause, `definedId` keying membership on element shape where item 5 says position, and both are tracked separately. **The duplicate cases still REJECT, and that is correct.** The premise that nothing can reference material under `attributes` is false as implemented: a `link` mark's `href` RESOLVES against such an id — it is not rewritten, since a preserved id takes no canonical name — so a duplicate there leaves a reference with two candidate referents. One consequence is deliberate: a preserved occurrence **consumes no index**, so a real sibling block now takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Item 5 states that rule and records that reserving an index instead “would give identical content a different document ID”. Zero documents in the corpus carry one, measured. +- **Conformance suite `suiteVersion` 0.2.4 → 0.2.5.** 215 → 216 vectors, 163 → 164 fixtures, 378 → 380 checks. The vector and the fixture kill DIFFERENT mutants and both are needed: relabeling returning is caught only by the vector, while excluding ids from the namespace rather than preserving them is caught only by the fixture (disposition IGNORE where INTEGRITY-ERROR is required). Nothing in the published suite caught that second case before, which is how the reverted approach reached a branch. The vector puts its id on a SINGULAR sub-object, not in an array, and that placement is the point: item 5 separately leaves as authored an id in an extension block's own data array, so an array case is reproduced by an implementation that never applies the `attributes` rule — it would pin the outcome and not the rule. The fixture expects a disposition RANGE rather than a pin, since an id collision is an Error in all states (Anchors and References §7.2) that escalates with the document state. The Python oracle needs no counterpart — measured, not assumed: it does not relabel, so preserved-versus-relabeled is invisible to it, and it confirms the new fixture independently. - **Canonicalization no longer REJECTs conformant documents carrying text-shaped data in a position that is not block content.** Document Hashing §4.3.1 item 4 (merge adjacent text nodes) and §4.3.2 item 2 (concatenated-NFC) were implemented on element **shape** — “is this object `{type:"text",value:string}`?” — so ANY array whose members merely resembled text nodes was merged and run-checked. Both rules are now keyed on **position**: the schema-declared block-content arrays `blocks`, `caption`, `children`, `content`, `hints`, `preamble`, plus `attributes` as a **sticky** barrier. The barrier is what the shape reading gets wrong in practice: `attributes.semantic` is a JSON-LD annotation with `additionalProperties: true` (semantic.schema.json), so an author may put a member named `children` there at any depth, and clearing the flag on the way down readmits the very false REJECT the scoping removes. Only the RUN rule is withheld beneath `attributes` — it is HASHED material, so both **per-string** invariants still bind there, pinned by two REJECT fixtures; without them the scope could be widened into a wholesale exemption with every gate green, only a non-NFC KEY having been covered. The set is **keys, not (type, key) pairs**, because several content arrays hang off sub-objects carrying no `type` at all (`exercisePart`, `solution`, a phantom's content), which a type-keyed map cannot reach. **The scope is a NARROWING, and its licence is not yet in `spec/`.** §4.3 applies the transforms across all block types “uniformly” and §4.3.2 binds the rule to “a block” with no enclosing-field list, where §4.3.1 item 5 names `lines`/`subfigures` as exhaustive when it means to scope by position. Measured: a `vendor:callout` carrying `body:[text,text]` with a split combining sequence REJECTed before and loads clean now. A spec erratum is required and is tracked separately; until it lands the reference under-enforces §4.3.2 for extension blocks whose text array is not one of the six keys. Strict `children` is the specific wrong answer, and it is measurable: it drops the ROOT `blocks` array that `reject-content-split-combining-sequence-mid-array` uses AND `figure.caption`, and now breaks **eight** published cases (four vectors, four fixtures) where before this slice it broke one. **Six walks** were scoped, four in `scripts/lib/canonicalize.ts` (`canon`, `walkContentNodes`, `validateStoredByteInvariants`, `collectStoredTextViolations`) and two in `conformance/oracles/document_oracle.py` (`_walk_text_violations`, `_walk_content`). `dropAbsorbedTextNodes` and `_drop_absorbed_text_nodes` — the pair an earlier withdrawn attempt missed — are unchanged in body; what was scoped is their INVOCATION, inside `walkContentNodes` and `_walk_content` respectively. Those two model the merge for the RAW walk, and scoping the other four alone makes canon keep a node the raw walks drop. **The barrier gates the merge and the run rule, NOT id-namespace membership**, and that asymmetry is deliberate for now rather than settled. `definedId` still counts an id carried under `attributes`, so two `anchor` marks sharing one there is a REJECT even though the merge treats the same subtree as inert. Gating `definedId` as well was implemented and REVERTED: it removes ids from the namespace but `rewriteIds` still relabels them, so two documents differing only in an id under `attributes` canonicalized to identical bytes — a silent document-ID collision in place of a loud false REJECT, which is the worse trade in a content-addressed format. The barrier has to reach `definedId`, `rewriteIds` and the reference resolver together or not at all; that is tracked separately. **Scan and canon agree at every position they both reach, with one KNOWN divergence.** `canon` skips a text node's `marks`, and `content` (a `presentation:footnote` mark's content blocks) occurs only there, so at that position the run rule fires and the merge does not — split text in a footnote mark's content is reported, while two mark-less nodes there stay unmerged. That asymmetry is #121's sequel, not a licence to widen the scope, and it is stated in the published vector rather than left for an implementer to infer. `check:block-content-positions` re-derives the set from the published schemas and fails on drift, comparing each transcription against the SCHEMA rather than against the other. Its derivation is **recursive and resolves local `$ref`s**: a position declared behind a `$ref` to an array-typed `$def`, or on an inline nested object (the shape `academic:abstractBlock.sections[].children` already uses), was previously invisible — and under-derivation is the one failure the gate cannot report on itself, since a key it never saw is a key both sets agree about. Both shapes now fail it. It also compares `CONTENT_BARRIER_KEY` across the two implementations and against `blockBase`, which nothing checked before. The change is a pure relaxation with **zero id blast radius** in this repository: every id-verifying gate passes unchanged and no example or fixture exercised the path. - **The published `anchor-range` structural rule no longer false-positives on a well-ordered range above 2^53.** `checkAnchors` ordered a Content Anchor URI's bounds by converting them to numbers, so two distinct integers that round to the same double compared equal and the range was reported inverted — under a message naming the same number twice. The URI grammar admits an unbounded `[0-9]+` and §4.3.2's 2^53 limit governs hashed *numbers* rather than digits inside a string, so such a URI is conformant. This mattered beyond its own rule: the document layer's position check had the identical defect and was fixed in the same change, which left two implementations in this repo disagreeing about the exact URI a new fixture pins. The comparison is now shared between them (`scripts/lib/decimal-digits.ts`), which is also what forced the leading-zero case to be handled — `[0-9]+` admits `007`, so a length-first digit comparison would otherwise have called `#p1/007-10` inverted. Two `anchor-range` vectors pin both - Broken cross-reference in presentation layers spec (Section 5.1.2 → 5.4) diff --git a/conformance/fixtures/document/integrity-error-attributes-id-collision/case.cdx b/conformance/fixtures/document/integrity-error-attributes-id-collision/case.cdx new file mode 100644 index 0000000..2d4c811 Binary files /dev/null and b/conformance/fixtures/document/integrity-error-attributes-id-collision/case.cdx differ diff --git a/conformance/fixtures/document/integrity-error-attributes-id-collision/case.json b/conformance/fixtures/document/integrity-error-attributes-id-collision/case.json new file mode 100644 index 0000000..91898ef --- /dev/null +++ b/conformance/fixtures/document/integrity-error-attributes-id-collision/case.json @@ -0,0 +1,38 @@ +{ + "description": "A block `id` duplicated by an `id` carried under another block's `attributes.semantic`. INTEGRITY-ERROR, state-invariant (Anchors and References section 7.2 makes an id collision an Error in all states), escalating with the document state as the other collision cases do. An id there is preserved as authored rather than relabeled — it is third-party vocabulary data, where an id is content — but it remains a full member of the shared UNIQUENESS namespace, exactly as a `semantic:term` id does (Document Hashing section 4.3.1 item 5: \"a preserved id remains a full member of the namespace in every other respect… Only the renaming skips it\"). That is not a technicality: the material is reachable, since a `link` mark's `href` resolves into it, so two nodes carrying one id genuinely leave a reference with two candidate referents. This is the case that distinguishes \"preserved\" from \"outside the namespace\" — an implementation choosing the latter reports nothing here, and then relabels the id anyway, computing one document id for two documents whose citation keys differ.", + "layer": "document", + "requires": [ + "container", + "document" + ], + "clause": "Document Hashing section 4.3.1 item 5; Anchors and References section 4", + "recipe": { + "entries": [ + { + "name": "manifest.json", + "text": "{\"cdx\":\"0.1\",\"id\":\"pending\",\"state\":\"draft\",\"created\":\"2025-01-10T08:00:00Z\",\"modified\":\"2025-01-10T08:00:00Z\",\"content\":{\"path\":\"content/document.json\",\"hash\":\"sha256:446334981f62833557fe195c5d7a14d276b63967d7834bf3217599232123e0b7\"},\"metadata\":{\"dublinCore\":\"metadata/dublin-core.json\"}}" + }, + { + "name": "content/document.json", + "text": "{\"version\":\"0.1\",\"blocks\":[{\"type\":\"paragraph\",\"id\":\"dup\",\"children\":[]},{\"type\":\"paragraph\",\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"n\":{\"type\":\"heading\",\"id\":\"dup\",\"children\":[]}}},\"children\":[]}]}" + }, + { + "name": "metadata/dublin-core.json", + "text": "{\"version\":\"1.1\",\"terms\":{\"title\":\"Attributes Id Collision\",\"creator\":\"A. Author\"}}" + } + ] + }, + "expect": { + "documentDisposition": { + "atLeast": "INTEGRITY-ERROR", + "atMost": "REJECT" + }, + "findings": [ + { + "code": "CDX-E-ID-COLLISION", + "atLeast": "INTEGRITY-ERROR", + "atMost": "REJECT" + } + ] + } +} diff --git a/conformance/suite.json b/conformance/suite.json index 8d6d9ac..8cfbe13 100644 --- a/conformance/suite.json +++ b/conformance/suite.json @@ -2,7 +2,7 @@ "normative": false, "$comment": "NON-NORMATIVE. This is the entry-point manifest for the CDX conformance suite, a testing artifact that references the specification and never extends it. The specification (spec/) and the published schemas (schemas/) are the only normative artifacts. Where anything here appears to disagree with spec/, the specification governs.", "suite": "cdx-conformance", - "suiteVersion": "0.2.4", + "suiteVersion": "0.2.5", "specVersion": "0.1", "title": "CDX Conformance Suite", "description": "An independently-runnable suite that lets a third-party implementation check itself against the CDX specification. This manifest binds a suite version to the specification version its expectations were derived against.", diff --git a/conformance/vectors/canonicalize.json b/conformance/vectors/canonicalize.json index 406fcb2..2d97889 100644 --- a/conformance/vectors/canonicalize.json +++ b/conformance/vectors/canonicalize.json @@ -891,6 +891,17 @@ }, "expectedCanonicalJcs": "{\"content\":{\"blocks\":[{\"exercises\":[{\"children\":[{\"children\":[{\"type\":\"text\",\"value\":\"Q\"}],\"type\":\"paragraph\"}],\"type\":\"academic:exercise\"}],\"preamble\":[{\"type\":\"text\",\"value\":\"Show your working.\"}],\"type\":\"academic:exercise-set\"}],\"version\":\"0.1\"},\"metadata\":{\"creator\":[\"C\"],\"title\":\"T\"}}", "expectedId": "sha256:21b661a2486cef3c6fc7363e7fde5029b48facc1b4202ae6c5c15e9ac4d1eb26" + }, + { + "name": "attributes-id-preserved-not-relabeled", + "description": "An `id` carried under `attributes` is preserved as authored, not relabeled, and consumes no index — the sibling block takes `b0` (§4.3.1 item 5). Item 5's alpha-equivalence exists so two documents differing only in author-chosen LABELS share an id; `attributes` is a closed set of authored values (Content Blocks §3.1), so an id there is a vocabulary key a JSON-LD or CSL consumer resolves against, and relabeling it makes two documents whose keys differ entirely hash identically. The id sits on a SINGULAR sub-object deliberately. Item 5 separately leaves as authored an id in an extension block's own DATA ARRAY, so a case putting this id in an array would be reproduced by an implementation that never applied the `attributes` rule at all — it would pin the outcome and not the rule. Nothing but the `attributes` rule reaches this position. Note what is NOT withheld: the id stays in the shared uniqueness namespace, because the material is reachable — a `link` mark's `href` resolves against it — so a duplicate there is a real collision.", + "parts": { + "manifest": "{}", + "content": "{\"version\":\"0.1\",\"blocks\":[{\"type\":\"paragraph\",\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"note\":{\"type\":\"heading\",\"id\":\"keepme\",\"children\":[]}}},\"children\":[]},{\"type\":\"paragraph\",\"id\":\"real\",\"children\":[]}]}", + "dublinCore": "{\"version\":\"1.1\",\"terms\":{\"title\":\"T\",\"creator\":\"C\"}}" + }, + "expectedCanonicalJcs": "{\"content\":{\"blocks\":[{\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"note\":{\"children\":[],\"id\":\"keepme\",\"type\":\"heading\"}}},\"children\":[],\"type\":\"paragraph\"},{\"children\":[],\"id\":\"b0\",\"type\":\"paragraph\"}],\"version\":\"0.1\"},\"metadata\":{\"creator\":[\"C\"],\"title\":\"T\"}}", + "expectedId": "sha256:15e27b0632507a2206be473dff14e151fdb4ec87a9ca8c96377058f06aa2a1e8" } ] } diff --git a/scripts/lib/canonicalize.ts b/scripts/lib/canonicalize.ts index c5dd5c0..3e4a389 100644 --- a/scripts/lib/canonicalize.ts +++ b/scripts/lib/canonicalize.ts @@ -705,6 +705,14 @@ export interface CollectedIds { * wants `ids`; only `alphaRenameIds` wants this. */ preserved: Set; + /** + * The subset of `preserved` held back by TYPE (`PRESERVED_ID_BLOCK_TYPES`) rather than by + * position. Only these carry item 5's canonical-name prohibition: an id under `attributes` is + * third-party vocabulary data the author does not choose — `b1`, `b2` is the CrossRef/JATS + * reference convention — and rejecting it would fail the documents this scoping protects, + * while the identical data in an `entries` array is accepted. + */ + preservedByType: Set; /** * Whatever `CollectOptions.classify` returned for each id, keyed by id. Empty when no * classifier was supplied. @@ -797,6 +805,7 @@ export function collectDefinedIds(content: unknown, options: CollectOptions = {} const ids: string[] = []; const duplicates: string[] = []; const preserved = new Set(); + const preservedByType = new Set(); const classified = new Map(); walkContentNodes( @@ -810,9 +819,9 @@ export function collectDefinedIds(content: unknown, options: CollectOptions = {} ids.push(id); // Recorded on the FIRST occurrence only, alongside `ids` — a repeat is a // duplicate and throws before any of this is consumed. - if (!ctx.inMarks && typeof node.type === 'string' && PRESERVED_ID_BLOCK_TYPES.has(node.type)) { - preserved.add(id); - } + const byType = !ctx.inMarks && typeof node.type === 'string' && PRESERVED_ID_BLOCK_TYPES.has(node.type); + if (ctx.blocked || byType) preserved.add(id); + if (byType) preservedByType.add(id); // Classified on the SAME first-occurrence branch as `ids`, so the two cannot // disagree about which ids exist or about which node defines one. if (options.classify !== undefined) classified.set(id, options.classify(node, ctx)); @@ -821,7 +830,7 @@ export function collectDefinedIds(content: unknown, options: CollectOptions = {} options, ); - return { ids, duplicates, preserved, classified }; + return { ids, duplicates, preserved, preservedByType, classified }; } /** Where a visited node sits, which is what decides namespace membership (`definedId`). */ @@ -840,6 +849,14 @@ export interface WalkContext { * it models a rewrite the canonicalizer never performs. */ inTextMarks: boolean; + /** + * The node sits at or below `attributes` (CONTENT_BARRIER_KEY) — a closed set of authored + * values (Content Blocks §3.1) whose members are data. An id there is content, not a label: + * it stays in the uniqueness namespace but is never relabeled, exactly as a + * PRESERVED_ID_BLOCK_TYPES id is. Removing it from the namespace instead relabels it via + * `rewriteIds`, which maps it onto a generated name. + */ + blocked: boolean; } /** @@ -874,7 +891,7 @@ export function walkContentNodes( return; } if (!isPlainObject(value)) return; - visit(value, { inMarks, inArray, parentKey, inTextMarks }); + visit(value, { inMarks, inArray, parentKey, inTextMarks, blocked }); for (const key of Object.keys(value).sort()) { if (raw && !inTextMarks && isDerivedField(value, key)) continue; // canon deletes it before relabeling const child = value[key]; @@ -1095,7 +1112,7 @@ function alphaRenameIds(content: unknown): unknown { // Canonical content: `canon` has already stripped derived fields and deduped each // text node's marks, so the default (non-raw) collection is exactly pass 1 of the // original in-place traversal. - const { ids, duplicates, preserved } = collectDefinedIds(content); + const { ids, duplicates, preserved, preservedByType } = collectDefinedIds(content); if (duplicates.length > 0) { throw new CanonicalizationError(`duplicate id "${duplicates[0]}" in the shared identifier namespace`); } @@ -1105,7 +1122,7 @@ function alphaRenameIds(content: unknown): unknown { // not collide with the `b0` the sequence below generates for some other block, and // both would reach the canonical output carrying the same id — with a `#b0` // reference silently binding to whichever the reader met first. - for (const id of preserved) { + for (const id of preservedByType) { if (CANONICAL_NAME.test(id)) { throw new CanonicalizationError( `id "${id}" uses the reserved canonical-name form: the id of a block whose type is left as authored ` + diff --git a/scripts/lib/fixture-corpus.ts b/scripts/lib/fixture-corpus.ts index 65a5d64..123ae44 100644 --- a/scripts/lib/fixture-corpus.ts +++ b/scripts/lib/fixture-corpus.ts @@ -2266,6 +2266,23 @@ export const FIXTURE_CORPUS: AuthoredCase[] = [ ), expect: rejectOnly('CDX-E-PART-STRING-NOT-NFC'), }, + { + name: 'integrity-error-attributes-id-collision', + description: 'A block `id` duplicated by an `id` carried under another block\'s `attributes.semantic`. INTEGRITY-ERROR, state-invariant (Anchors and References section 7.2 makes an id collision an Error in all states), escalating with the document state as the other collision cases do. An id there is preserved as authored rather than relabeled — it is third-party vocabulary data, where an id is content — but it remains a full member of the shared UNIQUENESS namespace, exactly as a `semantic:term` id does (Document Hashing section 4.3.1 item 5: "a preserved id remains a full member of the namespace in every other respect… Only the renaming skips it"). That is not a technicality: the material is reachable, since a `link` mark\'s `href` resolves into it, so two nodes carrying one id genuinely leave a reference with two candidate referents. This is the case that distinguishes "preserved" from "outside the namespace" — an implementation choosing the latter reports nothing here, and then relabels the id anyway, computing one document id for two documents whose citation keys differ.', + layer: 'document', + requires: ['container', 'document'], + clause: 'Document Hashing section 4.3.1 item 5; Anchors and References section 4', + recipe: documentArchive( + manifestJson({ content: { path: 'content/document.json', hash: 'sha256:446334981f62833557fe195c5d7a14d276b63967d7834bf3217599232123e0b7' } }), + contentBody('{"version":"0.1","blocks":[{"type":"paragraph","id":"dup","children":[]},{"type":"paragraph","attributes":{"semantic":{"@type":"Q","n":{"type":"heading","id":"dup","children":[]}}},"children":[]}]}', 'Attributes Id Collision'), + ), + // A RANGE, not a pin: section 5.4.2 assigns INTEGRITY-ERROR and the later states escalate + // it, exactly as the other id-collision cases express it. + expect: { + documentDisposition: { atLeast: 'INTEGRITY-ERROR', atMost: 'REJECT' }, + findings: [{ code: 'CDX-E-ID-COLLISION', atLeast: 'INTEGRITY-ERROR', atMost: 'REJECT' }], + }, + }, { name: 'positive-content-astral-text', description: 'A text node carrying an astral character (U+1F600, stored as the well-formed surrogate PAIR U+D83D U+DE00) in NFC. A reader that flagged any surrogate code unit rather than an UNPAIRED one would false-REJECT this — every non-BMP character in every conformant document is a surrogate pair.', diff --git a/scripts/test-canonicalize.ts b/scripts/test-canonicalize.ts index d8bde12..d56ae19 100644 --- a/scripts/test-canonicalize.ts +++ b/scripts/test-canonicalize.ts @@ -1403,10 +1403,30 @@ test('collectDefinedIds: rawInput reproduces canon, checked AGAINST canon', () = ['duplicate block id under attributes.semantic', { blocks: [{ type: 'paragraph', attributes: { semantic: { '@type': 'Q', children: [{ type: 'paragraph', id: 'dup', children: [] }, { type: 'paragraph', id: 'dup', children: [] }] } }, children: [] }] }], + // The two rows above both REJECT, so the `preserved` comparison below skips them. These do + // not, which is what lets it reach the barrier at all — without them two barrier mutations + // survive every gate in the repository. + ['unique id under attributes.semantic', { blocks: [ + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', n: { type: 'heading', id: 'kept', children: [] } } }, children: [] }, + { type: 'paragraph', id: 'real', children: [] }] }], + // The barrier must survive the raw walk's dedicated `marks` branch. It only bites when + // `blocked` is ALREADY set, so the mark has to sit INSIDE `attributes`, not carry it. + ['unique anchor id in marks INSIDE attributes', { blocks: [ + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', children: [ + { type: 'text', value: 'x', marks: [{ type: 'anchor', id: 'deep' }] }] } }, children: [] }, + { type: 'paragraph', id: 'real', children: [] }] }], ]; for (const [label, content] of cases) { - const { duplicates } = collectDefinedIds(content, { rawInput: true }); - assert.equal(duplicates.length > 0, rejectsForDuplicateId(content), `raw walk disagrees with canonicalization: ${label}`); + const raw = collectDefinedIds(content, { rawInput: true }); + assert.equal(raw.duplicates.length > 0, rejectsForDuplicateId(content), `raw walk disagrees with canonicalization: ${label}`); + + // `preserved` too, not only `duplicates`. The docstring above says this differential asserts + // the raw walk reproduces canon; comparing one field of three made that broader than what it + // checked, and two barrier mutations survived every gate in the repository because of it. + if (!rejectsForDuplicateId(content)) { + const canonical = collectDefinedIds((canonicalContent({ manifest: '{}', content: JSON.stringify(content), dublinCore: '{}' }) as any).content); + assert.deepEqual([...raw.preserved].sort(), [...canonical.preserved].sort(), `raw vs canonical preserved: ${label}`); + } } }); @@ -1444,6 +1464,67 @@ test("a text node's id must not spell a canonical name", () => { entries: [{ id: 'b0', type: 'text', value: 'x' }] }] }), 'sha256')); }); +test('an id under `attributes` is preserved, not relabeled', () => { + const parts = (content: unknown) => ({ manifest: '{}', content: JSON.stringify(content), dublinCore: '{}' }); + const canon = (content: unknown) => canonicalContent(parts(content)) as any; + const cited = (k1: string, k2: string) => ({ blocks: [{ type: 'paragraph', + attributes: { semantic: { citation: [{ id: k1, type: 'article-journal' }, { id: k2, type: 'book' }] } }, + children: [{ type: 'text', value: 'x' }] }] }); + + // Alpha-equivalence is for author-chosen LABELS. Applied to third-party vocabulary data it + // erases content: two bibliographies with different citation keys hashed identically, so a + // signature over one verified the other. + assert.notEqual( + computeDocumentId(parts(cited('smith2020', 'jones2019')), 'sha256'), + computeDocumentId(parts(cited('OTHER-KEY', 'DIFFERENT')), 'sha256'), + ); + assert.equal(canon(cited('smith2020', 'jones2019')).content.blocks[0].attributes.semantic.citation[0].id, 'smith2020'); + + // Preserved means preserved-and-still-counted: it consumes no index, so a real sibling takes + // the name the preserved occurrence would have had (§4.3.1 item 5). + const shifted = canon({ blocks: [ + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', note: { type: 'heading', id: 'ghost', children: [] } } }, children: [] }, + { type: 'paragraph', id: 'real', children: [] }] }); + assert.equal(shifted.content.blocks[0].attributes.semantic.note.id, 'ghost'); + assert.equal(shifted.content.blocks[1].id, 'b0', 'the preserved occurrence consumed no index'); + + // ... and still in the UNIQUENESS namespace. That material is reachable — a `link` href + // resolves into it — so a duplicate there is a real collision, not a false positive. + assert.throws(() => computeDocumentId(parts({ blocks: [ + { type: 'paragraph', id: 'dup', children: [] }, + { type: 'paragraph', attributes: { semantic: { '@type': 'Q', n: { type: 'heading', id: 'dup', children: [] } } }, children: [] }] }), 'sha256'), + /duplicate id/); + + // A reference INTO the barriered subtree keeps resolving, because the target keeps its name. + const linked = canon({ blocks: [{ type: 'paragraph', + attributes: { semantic: { '@type': 'Q', children: [{ type: 'text', value: 't', marks: [{ type: 'anchor', id: 'hidden' }] }] } }, + children: [{ type: 'text', value: 'x', marks: [{ type: 'link', href: '#hidden' }] }] }] }); + assert.equal(linked.content.blocks[0].children[0].marks[0].href, '#hidden'); + + // NOT subject to item 5's canonical-name prohibition, which is scoped to the TYPE-preserved + // blocks. `b1`, `b2` is the CrossRef/JATS reference convention, so CSL exported from such a + // source carries exactly these keys — rejecting them would fail the documents this scoping + // exists to protect, and the identical data in an `entries` array is accepted. + const cslKeys = (where: 'attributes' | 'entries') => where === 'attributes' + ? { blocks: [{ type: 'paragraph', attributes: { semantic: { citation: [{ id: 'b1', type: 'article-journal' }] } }, children: [] }, { type: 'paragraph', id: 'p', children: [] }] } + : { blocks: [{ type: 'semantic:bibliography', entries: [{ id: 'b1', type: 'article-journal' }] }, { type: 'paragraph', id: 'p', children: [] }] }; + for (const where of ['attributes', 'entries'] as const) { + const out = canon(cslKeys(where)) as any; + assert.doesNotThrow(() => computeDocumentId(parts(cslKeys(where)), 'sha256'), `${where}: a b vocabulary key is not an error`); + assert.equal(out.content.blocks[1].id, 'b0', `${where}: the real block still relabels`); + } + + // The prohibition still binds where item 5 scopes it — a TYPE-preserved block id. + assert.throws(() => computeDocumentId(parts({ blocks: [ + { type: 'semantic:term', id: 'b0', children: [] }] }), 'sha256'), /reserved canonical-name form/); + + // CONTROL. Ordinary content must be untouched by all of the above — without this the + // assertions are equally satisfied by a canonicalizer that stopped relabeling entirely. + const plain = canon({ blocks: [{ type: 'paragraph', id: 'p1', children: [{ type: 'text', value: 't', marks: [{ type: 'anchor', id: 'a2' }] }] }] }); + assert.equal(plain.content.blocks[0].id, 'b0', 'a real block still relabels'); + assert.equal(plain.content.blocks[0].children[0].marks[0].id, 'b1', 'and a real anchor mark still relabels'); +}); + // --- The asset map in the raw walk (B1b-3b-2) -------------------------------- // `normalizeMarks` resolves a `link` mark's href to its asset HASH before // `mergeAdjacentText` compares mark sets, so whether two adjacent text nodes merge — and diff --git a/spec/core/06-document-hashing.md b/spec/core/06-document-hashing.md index 5b790c6..9a5c656 100644 --- a/spec/core/06-document-hashing.md +++ b/spec/core/06-document-hashing.md @@ -161,13 +161,13 @@ The canonical content is a *transform* of the stored parts; the stored files are **Metadata projection.** From `metadata/dublin-core.json`, take the `terms` object and keep exactly `title`, `creator`, `subject`, `description`, and `language`, flattened to a `{term: value}` object (the `{version, terms}` wrapper and every other term — including the structured `creators` — are dropped). `title` and `description` are strings; `creator`, `subject`, and `language` are arrays (a scalar value is coerced to a one-element array). A term is omitted entirely when it is absent or wholly empty (`""` or `[]`); non-empty array elements are preserved verbatim, in authored order, and an empty (`""`) element inside an otherwise non-empty array is dropped (a term whose elements are all empty is likewise omitted). (`title` and `creator` are required and non-empty, so they always survive.) -**Content transforms.** Apply to `content/document.json`, across all block types including `namespace:type` extension blocks — uniformly, but for the position scoping item 4 names and the two id-relabeling exceptions item 5 names: +**Content transforms.** Apply to `content/document.json`, across all block types including `namespace:type` extension blocks — uniformly, but for the position scoping item 4 names and the id-relabeling exceptions item 5 names: 1. **Strip non-content fields**: remove `measurement.display` and `codeBlock.tokens` (presentational/regenerable; no canonical form), and any `crdt` field carried on a block or text node (transient collaboration synchronization state, excluded per section 4.1a). 2. **Resolve asset references**: replace every content-level asset *path* reference — an `image.src`, `svg.src`, or `signature.image`, or an archive-relative `href` on a `link` mark — with the referenced asset's content hash (`algorithm:hexdigest`). A reference resolves iff, after path normalization (no `.` or `..` segments; case-sensitive), it equals an asset's archive path — that asset's category directory (`assets/` followed by its category, i.e. the key under `manifest.assets` that registers it: the standard `images`, `fonts`, or `embeds`, or any additional category) joined with the asset's index `path` (Asset Embedding, section 3). The following are not asset paths and are left verbatim: an `href` beginning with `#` (an internal Content Anchor reference), a reference marked `external`, and any reference carrying a URL scheme (e.g. `https://`); an `svg` with inline `content` and no `src` has nothing to resolve; and an `image.fallback` path is left verbatim, being a backup rather than the bound resource. A reference beginning with `assets/` that matches no registered asset is a canonicalization error. The document ID thereby inherits the integrity of the asset index `hash` values, which MUST be verified against the asset bytes (Asset Embedding, section 8) before the ID can be trusted. 3. **Normalize marks**: within each text node, sort the `marks` array by each mark's JCS serialization (UTF-16 code-unit comparison — so bare formatting marks such as `"bold"` sort before structured marks such as `{"type":"link",…}`), remove marks with identical JCS serialization (deduplicate), and omit the `marks` key entirely when the array is empty (absent ≡ `[]`). Mark order is not semantic. 4. **Merge text nodes**: merge adjacent sibling text nodes that have no intervening non-text inline child and identical canonical mark-sets, concatenating their `value`s. Only a text node whose sole keys are `type`, `value`, and `marks` (after the step 1 stripping) is merge-eligible; a text node that also carries an `id`, `attributes`, or any other field is preserved unchanged and acts as a boundary, so no identifier or annotation is dropped. Offsets are defined over the concatenated text content (Anchors and References, section 3), so this moves no anchor. This rule and section 4.3.2 item 2 apply only within a **block-content array** — an array reached through one of the field names `blocks`, `caption`, `children`, `content`, `hints` or `preamble`, which is the exhaustive set. Membership is keyed on that field NAME and not on the owner's type: several such arrays hang off sub-objects carrying no `type` at all — a `subfigure`, an `academic:exercise` part, a `solution`, a structured abstract's section — and the content root, which is not a block, owns one. An implementation keying on the owner rather than the field misses them and computes a different document ID. Membership is keyed on the enclosing field, not on element shape: a text-shaped object an extension block carries in any other array is not block content and is left as authored, and `attributes` is a barrier at every depth beneath it — a closed set of authored values (Content Blocks, section 3.1) whose members are data even when shaped like inline content. An extension block therefore receives these transforms on a rich-text array exactly when it names that array with one of these keys; an array named anything else carries text the transforms do not reach, and a document containing one is still conformant. The two rules do not reach identical positions: this rule does not recurse into a text node's `marks`, so it never reaches a `presentation:footnote` mark's `content`, while section 4.3.2 item 2 does. -5. **Canonicalize identifiers**: relabel author-chosen identifiers to position-based canonical names, so that two documents differing only in their id *labels* produce the same document ID. A text node's `id` is outside this namespace entirely: it is never relabeled, occupies no index, and is not compared for duplicates — item 4 preserves it unchanged because it is content rather than a label. Because it survives into the canonical content, it MUST NOT spell a canonical name (`b` followed by digits); one that does is a canonicalization error, for the same reason the prohibition below applies to a preserved id — left as authored beside a generated `b0`, two identifiers reach the canonical output under one name. The relabeled namespace is the shared identifier namespace (Anchors and References, section 4): the `id` of every block other than a text node (including `namespace:type` extension blocks), the `id` of every `anchor` mark, and the `id` of every in-content sub-block element that carries one — an academic equation line and a `subfigure` — even though those carry no block `type`. These sub-block elements are the array items of an equation group's `lines` and a figure's `subfigures` — the exhaustive set of relabeled sub-block id arrays. Membership is keyed on WHERE an id sits (the enclosing field), not merely on whether the node has a `type` or is an array item: an id-bearing object an extension block carries in its own data array (any array other than `lines`/`subfigures`) is left as authored, even when shaped like a namespace member. In particular a `semantic:bibliography` entry's `id` — a CSL citation key in an `entries` array, referenced by a `citation` mark's `refs` (below) — is not relabeled, though the entry carries a CSL `type` that makes it resemble a block. Likewise an identifier carried on a singular named sub-object — such as a signature block's `signer` `id`, a Person identifier rather than a content-anchor target — is not part of this namespace. Two extension block types are an explicit exception to the *relabeling*, and are the only namespace members whose ids are left as authored: `semantic:term` and `semantic:footnote`. Each is the **defining** side of a reference that addresses a separate namespace and is itself never relabeled — a `glossary` mark's `ref` and a `semantic:term` `see` resolve against a `semantic:term` block's `id` when no external glossary is declared, and a `footnote` mark resolves against a `semantic:footnote` block (Semantic Extension sections 4.5.2 and 8.3) — so relabeling the definition while leaving the reference as authored left those rules unsatisfiable on the canonical form: the block became `b0` while the reference still named `term-algorithm`. A preserved id remains a full member of the namespace in every other respect: it occupies the shared uniqueness space, a duplicate is a canonicalization error exactly as above, and a Content Anchor URI resolves against it exactly as against a relabeled block id. Only the renaming skips it. Because it is left as authored, a preserved id MUST NOT itself spell a canonical name (`b` followed by digits); one that does is a canonicalization error. This mirrors the prohibition on unresolved references below and holds for the same reason: duplicate detection compares *authored* ids, so an authored `b0` does not collide with the `b0` generated for some other block, and both would reach the canonical output under one identifier. Membership in this exception is keyed on the block's `type` rather than on where the id sits — unlike the rest of this item, and of necessity, since a preserved id sits directly on its block, exactly where any other block id sits. Walk the canonical content depth-first — at each node assigning a name to the node's own `id` (if any) before recursing into its members (object members in JCS key order, array elements in index order) — and give each id-defining occurrence the next name in the sequence `b0`, `b1`, `b2`, …; a duplicate id within this namespace is a canonicalization error. An occurrence whose id is preserved takes no name and **consumes no index**: the sequence continues at the same name for the next relabeled occurrence, so a `semantic:term` followed by a paragraph yields `b0` on the paragraph. Reserving an index for the preserved occurrence instead is equally implementable and would give identical content a different document ID, so the choice is stated here rather than left to the implementation. Then rewrite every Content Anchor URI reference (`#id`, optionally followed by `/offset` or `/start-end`) whose id is one of these names — preserving the suffix — in the following reference fields: a `link` mark `href`; the `academic:theorem-ref`, `academic:equation-ref`, and `academic:algorithm-ref` mark `target`; the `academic:theorem` `uses`; the `academic:proof` `of`; and the `semantic:ref` and `presentation:reference` block `target`. A Content Anchor URI whose id is not a relabeled id — for example a cross-document anchor — is left unchanged, except that such an unresolved reference MUST NOT itself spell a canonical name (`b` followed by digits): an unresolved `#b0` is a canonicalization error, because left unchanged it would be indistinguishable from a reference that resolved to that block, collapsing two distinct documents onto one ID. Because relabeling an `anchor` id or a `link` `href` changes that mark's serialization, the marks of any affected text node are re-sorted and de-duplicated (item 3) after rewriting. +5. **Canonicalize identifiers**: relabel author-chosen identifiers to position-based canonical names, so that two documents differing only in their id *labels* produce the same document ID. A text node's `id` is outside this namespace entirely: it is never relabeled, occupies no index, and is not compared for duplicates — item 4 preserves it unchanged because it is content rather than a label. Because it survives into the canonical content, it MUST NOT spell a canonical name (`b` followed by digits); one that does is a canonicalization error, for the same reason the prohibition below applies to a preserved id — left as authored beside a generated `b0`, two identifiers reach the canonical output under one name. The relabeled namespace is the shared identifier namespace (Anchors and References, section 4): the `id` of every block other than a text node (including `namespace:type` extension blocks), the `id` of every `anchor` mark, and the `id` of every in-content sub-block element that carries one — an academic equation line and a `subfigure` — even though those carry no block `type`. These sub-block elements are the array items of an equation group's `lines` and a figure's `subfigures` — the exhaustive set of relabeled sub-block id arrays. Membership is keyed on WHERE an id sits (the enclosing field), not merely on whether the node has a `type` or is an array item: an id-bearing object an extension block carries in its own data array (any array other than `lines`/`subfigures`) is left as authored, even when shaped like a namespace member. In particular a `semantic:bibliography` entry's `id` — a CSL citation key in an `entries` array, referenced by a `citation` mark's `refs` (below) — is not relabeled, though the entry carries a CSL `type` that makes it resemble a block. Likewise an identifier carried on a singular named sub-object — such as a signature block's `signer` `id`, a Person identifier rather than a content-anchor target — is not part of this namespace. An id that would otherwise be a member of this namespace is left as authored when carried under `attributes`, keyed on POSITION: item 4 makes that a barrier at every depth, and Content Blocks section 3.1 closes it to authored values, so an id there is content — a vocabulary key a JSON-LD or CSL consumer resolves against — and not a label this rule may rename. Relabeling it applies alpha-equivalence to data the author did not choose as a label: two documents whose bibliographic citation keys differ entirely would otherwise share a document ID, so a signature over one verifies the other. This is the same reasoning that exempts a `semantic:bibliography` entry's `id` above, applied to the same data one field over. Two extension block types are a further exception, keyed on TYPE rather than position: `semantic:term` and `semantic:footnote`. Each is the **defining** side of a reference that addresses a separate namespace and is itself never relabeled — a `glossary` mark's `ref` and a `semantic:term` `see` resolve against a `semantic:term` block's `id` when no external glossary is declared, and a `footnote` mark resolves against a `semantic:footnote` block (Semantic Extension sections 4.5.2 and 8.3) — so relabeling the definition while leaving the reference as authored left those rules unsatisfiable on the canonical form: the block became `b0` while the reference still named `term-algorithm`. A preserved id — by position or by type — remains a full member of the namespace in every other respect: it occupies the shared uniqueness space, a duplicate is a canonicalization error exactly as above, and a Content Anchor URI resolves against it exactly as against a relabeled block id. Only the renaming skips it. Because it is left as authored, a TYPE-preserved id MUST NOT itself spell a canonical name (`b` followed by digits); one that does is a canonicalization error. This does not extend to an id preserved by POSITION: such an id is third-party vocabulary the author did not choose — `b1`, `b2` is a common reference-numbering convention — and rejecting it would fail exactly the documents the position rule exists to protect, while the same key in an extension block's own data array carries no such prohibition. This mirrors the prohibition on unresolved references below and holds for the same reason: duplicate detection compares *authored* ids, so an authored `b0` does not collide with the `b0` generated for some other block, and both would reach the canonical output under one identifier. Membership in the TYPE-keyed exception is keyed on the block's `type` rather than on where the id sits — unlike the rest of this item, and of necessity, since such an id sits directly on its block, exactly where any other block id sits. The `attributes` exception above is keyed on position, as the rest of this item is. Walk the canonical content depth-first — at each node assigning a name to the node's own `id` (if any) before recursing into its members (object members in JCS key order, array elements in index order) — and give each id-defining occurrence the next name in the sequence `b0`, `b1`, `b2`, …; a duplicate id within this namespace is a canonicalization error. An occurrence whose id is preserved takes no name and **consumes no index**: the sequence continues at the same name for the next relabeled occurrence, so a `semantic:term` followed by a paragraph yields `b0` on the paragraph. Reserving an index for the preserved occurrence instead is equally implementable and would give identical content a different document ID, so the choice is stated here rather than left to the implementation. Then rewrite every Content Anchor URI reference (`#id`, optionally followed by `/offset` or `/start-end`) whose id is one of these names — preserving the suffix — in the following reference fields: a `link` mark `href`; the `academic:theorem-ref`, `academic:equation-ref`, and `academic:algorithm-ref` mark `target`; the `academic:theorem` `uses`; the `academic:proof` `of`; and the `semantic:ref` and `presentation:reference` block `target`. A Content Anchor URI whose id is not a relabeled id — for example a cross-document anchor — is left unchanged, except that such an unresolved reference MUST NOT itself spell a canonical name (`b` followed by digits): an unresolved `#b0` is a canonicalization error, because left unchanged it would be indistinguishable from a reference that resolved to that block, collapsing two distinct documents onto one ID. Because relabeling an `anchor` id or a `link` `href` changes that mark's serialization, the marks of any affected text node are re-sorted and de-duplicated (item 3) after rewriting. The following are left as authored, so a document that differs only in one of these labels is not yet guaranteed to share a document ID: the reference fields that address other namespaces — the `footnote` mark `id`, the `glossary` mark `ref`, `semantic:term` `see`, `citation` `refs`, `entity` `uri`, `index` `term`, and the `academic:algorithm-ref` `line` (a label into an algorithm's line numbering, not an `id`). Equation-line ids and subfigure ids are not in this set: they share the relabeled uniqueness namespace described above. A `semantic:footnote` or `semantic:term` block's own `id` is left as authored as well, but for a different reason and with a different consequence: it does not address another namespace — it stays in the shared uniqueness-and-anchor namespace described above and merely receives no canonical name, so anchors still resolve to it and a duplicate is still an error. It is listed here only because a document differing in one of those labels does not share a document ID. (A single text node is expected to carry at most one `anchor` mark; canonical names for multiple anchors on one node depend on their canonical mark order.) 6. **Preserve everything else as authored**: no other field is added, removed, defaulted, or coalesced. In particular `null` and absent are distinct, and JSON-Schema `default`s (such as `colspan`) are never materialized into hashed content.