Preserve ids carried under attributes, closing a document-ID collision

CHANGELOG.md

@@ -53,6 +53,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - Template generation script
 
 ### Fixed
+- **An `id` carried under `attributes` is no longer relabeled, closing one path to a silent document-ID collision.** §4.3.1 item 5 relabels author-chosen identifiers so two documents differing only in their LABELS share an id. That alpha-equivalence was reaching third-party vocabulary data, where an id is content: measured, two bibliographies whose CSL citation keys differed entirely canonicalized to identical bytes, so **a signature over one verified the other**. Item 5 already exempts a `semantic:bibliography` entry's `id` for exactly this reason; the same data under `attributes` was not. **PRESERVED, not excluded**, and the distinction is the whole fix: excluding takes the id out of the namespace but leaves `rewriteIds` mapping it onto a generated name, trading a loud failure for a silent one — that approach was implemented and reverted on the previous branch. A preserved id stays in the map's INPUT and out of the map, so `renameId` returns it verbatim, reusing the mechanism `semantic:term` and `semantic:footnote` already use. It does NOT inherit item 5's canonical-name prohibition, which stays scoped to the type-keyed exception: `b1`, `b2` is a common reference-numbering convention, so CSL exported from such a source carries exactly those keys, and rejecting them would fail the documents this rule exists to protect — while the identical data in an extension's own data array is accepted. An earlier draft of this change did reject them. §4.3.1 item 5 is amended to state the position-keyed exception, which it previously contradicted by calling the two type-keyed ones “the only namespace members whose ids are left as authored”. **The collision CLASS is not closed, and this entry should not be read as closing it.** The same harm remains reachable through any extension data array not literally named `entries`: measured, a `vendor:refs` block carrying a `citation` array gives one document ID for two entirely different key sets. Item 5 already states the broader rule — an id in an extension block's own data array is left as authored — and the reference implements it as a one-element denylist. Membership also still turns on `type` versus `@type`, so well-formed JSON-LD is out of the namespace and CSL-JSON is in. Both are the same cause, `definedId` keying membership on element shape where item 5 says position, and both are tracked separately. **The duplicate cases still REJECT, and that is correct.** The premise that nothing can reference material under `attributes` is false as implemented: a `link` mark's `href` RESOLVES against such an id — it is not rewritten, since a preserved id takes no canonical name — so a duplicate there leaves a reference with two candidate referents. One consequence is deliberate: a preserved occurrence **consumes no index**, so a real sibling block now takes the name the preserved one would have had, changing the document id of any document carrying an id under `attributes`. Item 5 states that rule and records that reserving an index instead “would give identical content a different document ID”. Zero documents in the corpus carry one, measured.
+- **Conformance suite `suiteVersion` 0.2.4 → 0.2.5.** 215 → 216 vectors, 163 → 164 fixtures, 378 → 380 checks. The vector and the fixture kill DIFFERENT mutants and both are needed: relabeling returning is caught only by the vector, while excluding ids from the namespace rather than preserving them is caught only by the fixture (disposition IGNORE where INTEGRITY-ERROR is required). Nothing in the published suite caught that second case before, which is how the reverted approach reached a branch. The vector puts its id on a SINGULAR sub-object, not in an array, and that placement is the point: item 5 separately leaves as authored an id in an extension block's own data array, so an array case is reproduced by an implementation that never applies the `attributes` rule — it would pin the outcome and not the rule. The fixture expects a disposition RANGE rather than a pin, since an id collision is an Error in all states (Anchors and References §7.2) that escalates with the document state. The Python oracle needs no counterpart — measured, not assumed: it does not relabel, so preserved-versus-relabeled is invisible to it, and it confirms the new fixture independently.
 - **Canonicalization no longer REJECTs conformant documents carrying text-shaped data in a position that is not block content.** Document Hashing §4.3.1 item 4 (merge adjacent text nodes) and §4.3.2 item 2 (concatenated-NFC) were implemented on element **shape** — “is this object `{type:"text",value:string}`?” — so ANY array whose members merely resembled text nodes was merged and run-checked. Both rules are now keyed on **position**: the schema-declared block-content arrays `blocks`, `caption`, `children`, `content`, `hints`, `preamble`, plus `attributes` as a **sticky** barrier. The barrier is what the shape reading gets wrong in practice: `attributes.semantic` is a JSON-LD annotation with `additionalProperties: true` (semantic.schema.json), so an author may put a member named `children` there at any depth, and clearing the flag on the way down readmits the very false REJECT the scoping removes. Only the RUN rule is withheld beneath `attributes` — it is HASHED material, so both **per-string** invariants still bind there, pinned by two REJECT fixtures; without them the scope could be widened into a wholesale exemption with every gate green, only a non-NFC KEY having been covered. The set is **keys, not (type, key) pairs**, because several content arrays hang off sub-objects carrying no `type` at all (`exercisePart`, `solution`, a phantom's content), which a type-keyed map cannot reach. **The scope is a NARROWING, and its licence is not yet in `spec/`.** §4.3 applies the transforms across all block types “uniformly” and §4.3.2 binds the rule to “a block” with no enclosing-field list, where §4.3.1 item 5 names `lines`/`subfigures` as exhaustive when it means to scope by position. Measured: a `vendor:callout` carrying `body:[text,text]` with a split combining sequence REJECTed before and loads clean now. A spec erratum is required and is tracked separately; until it lands the reference under-enforces §4.3.2 for extension blocks whose text array is not one of the six keys. Strict `children` is the specific wrong answer, and it is measurable: it drops the ROOT `blocks` array that `reject-content-split-combining-sequence-mid-array` uses AND `figure.caption`, and now breaks **eight** published cases (four vectors, four fixtures) where before this slice it broke one. **Six walks** were scoped, four in `scripts/lib/canonicalize.ts` (`canon`, `walkContentNodes`, `validateStoredByteInvariants`, `collectStoredTextViolations`) and two in `conformance/oracles/document_oracle.py` (`_walk_text_violations`, `_walk_content`). `dropAbsorbedTextNodes` and `_drop_absorbed_text_nodes` — the pair an earlier withdrawn attempt missed — are unchanged in body; what was scoped is their INVOCATION, inside `walkContentNodes` and `_walk_content` respectively. Those two model the merge for the RAW walk, and scoping the other four alone makes canon keep a node the raw walks drop. **The barrier gates the merge and the run rule, NOT id-namespace membership**, and that asymmetry is deliberate for now rather than settled. `definedId` still counts an id carried under `attributes`, so two `anchor` marks sharing one there is a REJECT even though the merge treats the same subtree as inert. Gating `definedId` as well was implemented and REVERTED: it removes ids from the namespace but `rewriteIds` still relabels them, so two documents differing only in an id under `attributes` canonicalized to identical bytes — a silent document-ID collision in place of a loud false REJECT, which is the worse trade in a content-addressed format. The barrier has to reach `definedId`, `rewriteIds` and the reference resolver together or not at all; that is tracked separately. **Scan and canon agree at every position they both reach, with one KNOWN divergence.** `canon` skips a text node's `marks`, and `content` (a `presentation:footnote` mark's content blocks) occurs only there, so at that position the run rule fires and the merge does not — split text in a footnote mark's content is reported, while two mark-less nodes there stay unmerged. That asymmetry is #121's sequel, not a licence to widen the scope, and it is stated in the published vector rather than left for an implementer to infer. `check:block-content-positions` re-derives the set from the published schemas and fails on drift, comparing each transcription against the SCHEMA rather than against the other. Its derivation is **recursive and resolves local `$ref`s**: a position declared behind a `$ref` to an array-typed `$def`, or on an inline nested object (the shape `academic:abstractBlock.sections[].children` already uses), was previously invisible — and under-derivation is the one failure the gate cannot report on itself, since a key it never saw is a key both sets agree about. Both shapes now fail it. It also compares `CONTENT_BARRIER_KEY` across the two implementations and against `blockBase`, which nothing checked before. The change is a pure relaxation with **zero id blast radius** in this repository: every id-verifying gate passes unchanged and no example or fixture exercised the path.
 - **The published `anchor-range` structural rule no longer false-positives on a well-ordered range above 2^53.** `checkAnchors` ordered a Content Anchor URI's bounds by converting them to numbers, so two distinct integers that round to the same double compared equal and the range was reported inverted — under a message naming the same number twice. The URI grammar admits an unbounded `[0-9]+` and §4.3.2's 2^53 limit governs hashed *numbers* rather than digits inside a string, so such a URI is conformant. This mattered beyond its own rule: the document layer's position check had the identical defect and was fixed in the same change, which left two implementations in this repo disagreeing about the exact URI a new fixture pins. The comparison is now shared between them (`scripts/lib/decimal-digits.ts`), which is also what forced the leading-zero case to be handled — `[0-9]+` admits `007`, so a length-first digit comparison would otherwise have called `#p1/007-10` inverted. Two `anchor-range` vectors pin both
 - Broken cross-reference in presentation layers spec (Section 5.1.2 → 5.4)

conformance/fixtures/document/integrity-error-attributes-id-collision/case.json

@@ -0,0 +1,38 @@
+{
+  "description": "A block `id` duplicated by an `id` carried under another block's `attributes.semantic`. INTEGRITY-ERROR, state-invariant (Anchors and References section 7.2 makes an id collision an Error in all states), escalating with the document state as the other collision cases do. An id there is preserved as authored rather than relabeled — it is third-party vocabulary data, where an id is content — but it remains a full member of the shared UNIQUENESS namespace, exactly as a `semantic:term` id does (Document Hashing section 4.3.1 item 5: \"a preserved id remains a full member of the namespace in every other respect… Only the renaming skips it\"). That is not a technicality: the material is reachable, since a `link` mark's `href` resolves into it, so two nodes carrying one id genuinely leave a reference with two candidate referents. This is the case that distinguishes \"preserved\" from \"outside the namespace\" — an implementation choosing the latter reports nothing here, and then relabels the id anyway, computing one document id for two documents whose citation keys differ.",
+  "layer": "document",
+  "requires": [
+    "container",
+    "document"
+  ],
+  "clause": "Document Hashing section 4.3.1 item 5; Anchors and References section 4",
+  "recipe": {
+    "entries": [
+      {
+        "name": "manifest.json",
+        "text": "{\"cdx\":\"0.1\",\"id\":\"pending\",\"state\":\"draft\",\"created\":\"2025-01-10T08:00:00Z\",\"modified\":\"2025-01-10T08:00:00Z\",\"content\":{\"path\":\"content/document.json\",\"hash\":\"sha256:446334981f62833557fe195c5d7a14d276b63967d7834bf3217599232123e0b7\"},\"metadata\":{\"dublinCore\":\"metadata/dublin-core.json\"}}"
+      },
+      {
+        "name": "content/document.json",
+        "text": "{\"version\":\"0.1\",\"blocks\":[{\"type\":\"paragraph\",\"id\":\"dup\",\"children\":[]},{\"type\":\"paragraph\",\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"n\":{\"type\":\"heading\",\"id\":\"dup\",\"children\":[]}}},\"children\":[]}]}"
+      },
+      {
+        "name": "metadata/dublin-core.json",
+        "text": "{\"version\":\"1.1\",\"terms\":{\"title\":\"Attributes Id Collision\",\"creator\":\"A. Author\"}}"
+      }
+    ]
+  },
+  "expect": {
+    "documentDisposition": {
+      "atLeast": "INTEGRITY-ERROR",
+      "atMost": "REJECT"
+    },
+    "findings": [
+      {
+        "code": "CDX-E-ID-COLLISION",
+        "atLeast": "INTEGRITY-ERROR",
+        "atMost": "REJECT"
+      }
+    ]
+  }
+}

conformance/suite.json

@@ -2,7 +2,7 @@
   "normative": false,
   "$comment": "NON-NORMATIVE. This is the entry-point manifest for the CDX conformance suite, a testing artifact that references the specification and never extends it. The specification (spec/) and the published schemas (schemas/) are the only normative artifacts. Where anything here appears to disagree with spec/, the specification governs.",
   "suite": "cdx-conformance",
-  "suiteVersion": "0.2.4",
+  "suiteVersion": "0.2.5",
   "specVersion": "0.1",
   "title": "CDX Conformance Suite",
   "description": "An independently-runnable suite that lets a third-party implementation check itself against the CDX specification. This manifest binds a suite version to the specification version its expectations were derived against.",

conformance/vectors/canonicalize.json

@@ -891,6 +891,17 @@
       },
       "expectedCanonicalJcs": "{\"content\":{\"blocks\":[{\"exercises\":[{\"children\":[{\"children\":[{\"type\":\"text\",\"value\":\"Q\"}],\"type\":\"paragraph\"}],\"type\":\"academic:exercise\"}],\"preamble\":[{\"type\":\"text\",\"value\":\"Show your working.\"}],\"type\":\"academic:exercise-set\"}],\"version\":\"0.1\"},\"metadata\":{\"creator\":[\"C\"],\"title\":\"T\"}}",
       "expectedId": "sha256:21b661a2486cef3c6fc7363e7fde5029b48facc1b4202ae6c5c15e9ac4d1eb26"
+    },
+    {
+      "name": "attributes-id-preserved-not-relabeled",
+      "description": "An `id` carried under `attributes` is preserved as authored, not relabeled, and consumes no index — the sibling block takes `b0` (§4.3.1 item 5). Item 5's alpha-equivalence exists so two documents differing only in author-chosen LABELS share an id; `attributes` is a closed set of authored values (Content Blocks §3.1), so an id there is a vocabulary key a JSON-LD or CSL consumer resolves against, and relabeling it makes two documents whose keys differ entirely hash identically. The id sits on a SINGULAR sub-object deliberately. Item 5 separately leaves as authored an id in an extension block's own DATA ARRAY, so a case putting this id in an array would be reproduced by an implementation that never applied the `attributes` rule at all — it would pin the outcome and not the rule. Nothing but the `attributes` rule reaches this position. Note what is NOT withheld: the id stays in the shared uniqueness namespace, because the material is reachable — a `link` mark's `href` resolves against it — so a duplicate there is a real collision.",
+      "parts": {
+        "manifest": "{}",
+        "content": "{\"version\":\"0.1\",\"blocks\":[{\"type\":\"paragraph\",\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"note\":{\"type\":\"heading\",\"id\":\"keepme\",\"children\":[]}}},\"children\":[]},{\"type\":\"paragraph\",\"id\":\"real\",\"children\":[]}]}",
+        "dublinCore": "{\"version\":\"1.1\",\"terms\":{\"title\":\"T\",\"creator\":\"C\"}}"
+      },
+      "expectedCanonicalJcs": "{\"content\":{\"blocks\":[{\"attributes\":{\"semantic\":{\"@type\":\"Q\",\"note\":{\"children\":[],\"id\":\"keepme\",\"type\":\"heading\"}}},\"children\":[],\"type\":\"paragraph\"},{\"children\":[],\"id\":\"b0\",\"type\":\"paragraph\"}],\"version\":\"0.1\"},\"metadata\":{\"creator\":[\"C\"],\"title\":\"T\"}}",
+      "expectedId": "sha256:15e27b0632507a2206be473dff14e151fdb4ec87a9ca8c96377058f06aa2a1e8"
     }
   ]
 }