From ec5673f01f0c6902663c19da5794b7ed53d60f61 Mon Sep 17 00:00:00 2001 From: Greg von Nessi Date: Fri, 28 Aug 2026 20:50:03 +0100 Subject: [PATCH] Close the remaining #112 coverage gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four items left open on #112 shared one shape: a check that exists but that nothing can prove is doing any work. An arm of the document oracle's `confirm_clean` runs only on a CLEAN fixture, and a fixture carrying the defect that arm looks for is by definition not clean, so no corpus case ever reaches it with its defect present. Deleting any one of the four arms B1b-3c-1a added left every gate green. The oracle gains a selftest driving all 37 refusal arms from the malformed corpus, and — because a table restoring today's coverage adds nothing that keeps tomorrow's arm honest — the arms are enumerated from the function's own AST and the drivers traced, so a new arm with no driver fails rather than passing quietly. Inserting one was measured to leave every gate green before this. `check:fixtures` gains a hash-coherence pass over every {path, hash} pair a manifest declares. The fixture comparator is a subset check, so an unexpected hash mismatch is tolerated outright, and on a case already expecting WARNING or REJECT it cannot move the disposition either — so every hand-computed hash in the corpus was unverified on any fixture that expects a finding. Corrupting one non-clean fixture's asset-index hash was invisible to check:conformance, check:fixtures, check:document-oracle and check:document-id together. One fixture covers the Dublin Core array shape, which no case reached: §4.3.1 projects creator, subject and language as arrays, and every other DC part in the corpus authors a scalar. Four verdict tests cover what the corpus cannot express, including the negative control for the concatenation scan's run flush — the existing fixture pins that a run is reported at a non-text element, but not that the flush separates. A fix for a related reader defect — a config-slot reference that declares a hash and achieves no binding, reported by nothing — was written and then reverted from this branch. A three-lens panel found the reuse of CDX-E-MANIFEST-REFERENCE-MALFORMED has no precedent and no note, that the fix closed the two shapes that code's published summary covers least, and that REJECT is inferred across an asymmetry the specification draws deliberately in three separate places. Publishing a disposition is a one-way door, so the defect and all the measurements are recorded in #120 for a specification decision instead. errors.json stays at v0.14/86; suiteVersion 0.2.2 -> 0.2.3, the patch position being where added cases land pre-1.0. --- CHANGELOG.md | 2 + .../case.cdx | Bin 0 -> 762 bytes .../case.json | 38 +++ conformance/oracles/document_oracle.py | 292 +++++++++++++++++- conformance/suite.json | 2 +- scripts/check-fixtures.ts | 268 +++++++++++++++- scripts/lib/fixture-corpus.ts | 19 ++ scripts/test-document-verdict.ts | 59 ++++ 8 files changed, 669 insertions(+), 11 deletions(-) create mode 100644 conformance/fixtures/document/reject-metadata-term-non-nfc-array-element/case.cdx create mode 100644 conformance/fixtures/document/reject-metadata-term-non-nfc-array-element/case.json diff --git a/CHANGELOG.md b/CHANGELOG.md index ade5888..0655d80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - Precise layout example (`examples/presentation-document/presentation/layouts/letter.json`) #### Tooling +- Conformance suite `conformance/suite.json` `suiteVersion` 0.2.3: the four coverage gaps the B1b-3c-1a review panel proved were real but left open. No new defect codes — `conformance/errors.json` stays at **v0.14** — and one added case, which is where the patch position sits under the rule 0.2.2 recorded. `document_oracle.py` gains a `selftest()`, as `archive_oracle.py` already had: an arm of `confirm_clean` runs only on a CLEAN fixture, and a fixture carrying the defect that arm looks for is by definition not clean, so no corpus case ever reaches it with its defect present. Measured: deleting any one of the four arms B1b-3c-1a added — the root envelope, the stored-byte text scan, the anchor-position pass and the out-of-hash annotation position — left every gate green, and each deletion made `confirm_clean` return "clean" rather than being caught by a later arm. The same was true of every OTHER arm, so the selftest covers all of them: 52 drivers reaching all **37** refusal arms, each asserting the SPECIFIC refusal message, because `confirm_clean` returns on its first failing arm and a bare "refused it" assertion is satisfied by any later one. Where an arm returns a tagged set — the classifier's six block/mark tags, the four reference tags, the three position rows — each tag gets its own row. Thirty-six arms are driven from the malformed corpus; the thirty-seventh, the archive-unreadable guard, is driven from synthetic bytes, no document fixture being an unreadable zip. Crucially the table does not merely restore coverage but RATCHETS it: the arms are enumerated from the function's own AST and the drivers traced, so an arm added later with no row fails the selftest instead of being silent — without that, inserting a new arm was measured to leave every gate green, which is precisely how the four arms above came to be unmutatable. Selftest failures now accumulate rather than returning on the first, so one rotted fragment reports the others in the same run. Building the table surfaced a latent crash it also fixes: `confirm_clean` assumed a parsed manifest was an object and raised `AttributeError` on a JSON array, and since `check` does not wrap the call, a clean fixture with an array manifest would have aborted the whole oracle run with a traceback instead of reporting a refusal. `check:fixtures` gains a hash-coherence pass over every `{path, hash}` pair a document fixture's manifest DECLARES — content part, asset index, presentation layer and extension config slot alike, 143 hashes across 123 fixtures: each must agree with the bytes the recipe stores, and must DISAGREE exactly where the case expects the mismatch code for that reference's class. This cannot live in `check:conformance` — the fixture comparator is a subset check, so an unexpected hash-mismatch finding is tolerated outright, and on a case already expecting a WARNING or a REJECT it cannot move the document disposition either. Measured on `reject-content-key-non-nfc`: flipping one hex digit of its content hash leaves all 366 conformance checks green while the new gate names it. The claim is scoped deliberately — on the five cases expecting exactly [IGNORE, IGNORE] the extra WARNING DOES move the disposition and `check:conformance` catches it, so the hole closed is the WARNING and REJECT population, not literally every non-clean fixture. Scope beyond `content.hash` was not optional: corrupting one non-clean fixture's asset-index hash was measured invisible to `check:conformance`, `check:fixtures`, `check:document-oracle` and `check:document-id` together. The pass declines a digest this runtime cannot compute rather than throwing — blake3 is permitted by Container §3.2 and declared in `capabilities.json`, and an uncaught throw there aborted the gate and took both independent oracles with it — and it pins BOTH its skip sets by NAME — the fixtures carrying no comparable reference, and the individual references it declined — with a floor on what was compared, because a gate that reports a tick having compared nothing, or that folds an unverified hash into the count of those that agree, is the same silent pass it exists to remove. `manifest.security.accessControl` is carried deliberately with NO code: it is the one security-block reference declared as a `{path, hash}` rather than a bare path and nothing in the reader verifies its hash, so the pass still requires it to match its bytes but does not attribute a disagreement to a code no implementation emits — which would make such a fixture unauthorable. The boundary is stated with what it leaves out rather than only what it covers: an asset whose bytes diverge from the hash listed INSIDE an index file is pinned by the oracle's asset arms on clean fixtures only, so it remains unpinned on a non-clean one, which is this same hole one level down. One document fixture (157 fixtures, 366 checks), `reject-metadata-term-non-nfc-array-element` is the coverage one: §4.3.1 keeps exactly five Dublin Core terms and projects `creator`, `subject` and `language` as ARRAYS, a scalar being coerced to a one-element array, while the AUTHORED value may be either shape — and every other Dublin Core part in the corpus authors the scalar, so no case had the array shape reaching §4.3.2's scan. Verified against both implementations separately: dropping the array terms from the reference's loop fails it on the document DISPOSITION, and dropping them from the oracle's makes it unable to confirm the defect. Four `test-document-verdict` cases for what the corpus cannot express: the array shape at the verdict layer for each of the three array-valued terms — pinning only `creator` was measured to leave `subject` and `language` unscanned in BOTH implementations with the entire corpus green — and the negative control for the concatenation scan's run flush. `reject-content-split-combining-sequence-mid-array` pins that a run IS reported at a non-text element but not that the flush SEPARATES; the control is a PAIR carrying the same four text-node values inside a `blockquote`, differing only by a paragraph between the second and third, each run NFC-stable alone and only their concatenation not. Measured: deleting the reset leaves all 366 checks and both oracles green and fails only the new test + - Conformance defect vocabulary `conformance/errors.json` **v0.14**: eight new codes for content-part validity. `CDX-E-CONTENT-ROOT-MALFORMED` (REJECT), `CDX-E-PART-STRING-NOT-NFC` and `CDX-E-PART-STRING-ILL-FORMED` (REJECT, both citing §5.4.3 as their siblings do), `CDX-E-ANNOTATION-ANCHOR-POSITION-INVALID` (WARNING, state-invariant), `CDX-E-ANCHOR-POSITION-INVALID` with **`disposition: null`** — §7.2's severity table covers only three anchor conditions and a structurally invalid position is none of them, so the gap is recorded rather than filled by analogy, and joins OQ-007's register — and **three** out-of-range codes rather than one, because an out-of-range position is disposed of by the §5.4.2 row of the FIELD carrying it and those rows do not agree. `CDX-E-ANCHOR-POSITION-OUT-OF-RANGE` is scoped to a `link` href, taking the dangling-core-anchor row, whose cited Anchors and References §7.2 tabulates exactly this condition and escalates it when frozen; `CDX-E-BLOCK-REFERENCE-POSITION-OUT-OF-RANGE` takes the extension-block-field row and inherits `CDX-E-BLOCK-REFERENCE-DANGLING`'s recorded contest; `CDX-E-CROSS-REFERENCE-POSITION-OUT-OF-RANGE` takes the extension-cross-reference row, WARNING in every state. One code across all three would publish a single disposition for rows that diverge, and post-B3 an `academic:theorem-ref` whose target was MISSING would stay a WARNING while one naming a merely stale offset became an INTEGRITY-ERROR. The vocabulary is append-only, so the arms had to be separated before a disposition was published, not after; the §7.2-versus-§5.4.2 conflict the cross-reference code records is new **OQ-011**, settled by neither. The structural arm is deliberately NOT split — a null disposition contributes nothing to a verdict and so cannot diverge between rows. The two Unicode codes are scanned over the part's **hashed material** — the stored bytes minus the fields canonicalization deletes — matching the "a **hashed** object key or string value" both rows and both summaries bind. A violation inside a deleted field (`crdt`, `codeBlock.tokens`, `measurement.display`) is therefore NOT reported: it is not hashed, and the split-view harm cannot reach it, since two readers cannot derive different canonical bytes from a field both delete. `positive-content-derived-field-non-nfc` is the published control for that boundary — a reader scanning the whole stored part rejects a conformant document in every state, and `crdt` carries opaque third-party state holding arbitrary user text. A field the transform *rewrites* rather than deletes IS reported, an asset path being replaced before validation runs. The third arm of the same clause, `CDX-E-PART-NUMBER-NON-REPRESENTABLE`, stays raw-scoped and so does not match: it was published that way, and narrowing it would flip an expectation an implementation may already meet. Twenty-nine document fixtures (156 total), two new `anchor-range` structural vectors (209 published vectors), independent Python confirmers for each new code a fixture can assert (seven of the eight; `CDX-E-ANCHOR-POSITION-INVALID` has none, having nothing to confirm) plus the matching `confirm_clean` arms, and forty `test-document-verdict` cases for the properties a fixture cannot express — the comparator requires every expected finding to carry a disposition interval, so no fixture can assert a null-disposition code, and being a subset check it cannot assert that a defect reported under one row is NOT also reported under another. Two of those fixtures pin properties of the position check that a single-text-node target cannot express: a range whose bounds are DISTINCT integers above 2^53, ordered by comparing the digit strings rather than the doubles they round to (`contentAnchorUri` admits an unbounded `[0-9]+`, and §4.3.2's 2^53 limit governs hashed *numbers*, not digits inside a string — so a double comparison finds the bounds equal and silently downgrades a tabulated WARNING to the null-disposition structural code); and a target whose text spans two text nodes, which is what makes §3's concatenation model observable at all — `join('')` to `join(' ')` previously survived in the reference AND the oracle, a shared misreading no gate could see. The derived text-bearing vocabulary is now pinned by assertion rather than only by the throw-on-empty guard, which measurement showed was not enough on its own: dropping `figcaption` from the derived set leaves the entire corpus green — every conformance check and every oracle confirmation — because no fixture anchors into one and a wrong set only produces the null-disposition structural code, which is invisible in both of the comparator's dimensions. The pin is written against the SPECIFICATION rather than against the derivation's output, which is what caught the next defect: `codeBlock` was MISSING from the derived set, because its schema reaches `textNode` through an `allOf` that narrows it (§4: exactly one marks-free text node) rather than through a direct reference. §3 names `codeBlock` text-bearing outright, so an anchor into a code block — the ordinary reason to anchor at all — was reported as targeting a non-text-bearing block on conformant content, and an out-of-bounds one was downgraded from the tabulated WARNING to the null-disposition code. The pin asserts two ONE-SIDED relations rather than an equality, which is the form §3 actually supports: it names its five types with "such as", so the list is illustrative and the definition is the children test. An equality assertion would fail on a CORRECT widening, and the cheapest route back to green from that failure is to narrow the derivation again — re-creating the defect. The lower bound is that every type §3 names is derived; the upper bound is that no container block is, §3 being explicit that a container "has no text content of its own". `conformance/suite.json` `suiteVersion` 0.2.1 -> 0.2.2, and its `versioning.rules` gains the rule both patch releases to date — 0.2.1 and this one — have followed: while pre-1.0 the minor position carries the breaking bump, so added cases and newly reported defect classes land in patch, which means a pre-1.0 patch bump CAN add an expectation an implementation is held to. v0.13's entry below stated it in prose; what it had never been given was a rule in the normative artefact. `conformance/capabilities.json` 0.2.0 -> 0.2.1, its `document` summary having claimed block/mark structure, NFC and the hash recomputes all still rode with a later phase, two of which had been false since B1b-2 and B1b-3a - Conformance defect vocabulary `conformance/errors.json` **v0.13**: two new codes for the footnote arm — `CDX-E-FOOTNOTE-REFERENCE-DANGLING` and `CDX-E-FOOTNOTE-REFERENCE-AMBIGUOUS`, both WARNING. The dangling code's disposition is transcribed from §5.4.2's extension-cross-reference row, which already named the footnote mark, and §4.5.2 restates it. The ambiguous one is weaker and the vocabulary says so: §5.4.2 assigns the ambiguity case no row at all, and §4.5.2 states the WARNING without a state qualifier and without citing §5.4 — so unlike its three siblings that entry does not claim state-invariance, and neither does this line. `CDX-E-CROSS-REFERENCE-DANGLING`'s "COVERAGE IS PARTIAL" note is rewritten: four of the row's five arms now have codes — citation, glossary, footnote, and the three `academic:*-ref` marks that code itself covers. The LEGAL arm remains uncovered, and the note now says so rather than passing over it: Legal Extension §5 states the disposition for a `legal:cite` whose `category` matches no Table of Authorities category, but it matches against a configuration rather than a content namespace, so it needs its own collector and its own answer on whether an absent configuration is indeterminate. Three document fixtures (127 total) and an independent Python confirmer that counts blocks per marker, since a flat id set cannot tell an ambiguous marker from a dangling one; `conformance/suite.json` `suiteVersion` 0.2.0 -> 0.2.1, the patch position being where added cases land now that the minor position carries the pre-1.0 breaking bump - Conformance defect vocabulary `conformance/errors.json` **v0.12**: `CDX-E-ID-COLLISION`'s note re-synced — it asserted the shared namespace "is exactly the one Document Hashing §4.3.1 item 5 relabels", which the preserved-id erratum makes false; the namespace is now a superset of what item 5 relabels, and the note distinguishes a `semantic:footnote` BLOCK id (a member) from the footnote MARK id that resolves to it (a separate namespace). `CDX-E-GLOSSARY-REFERENCE-DANGLING`'s recorded "specification tension" is retracted as resolved. Six `canonicalize` vectors added and one flipped (207 published vectors), covering the preserved id, the matching glossary `ref`, index compaction, and the three new rejects. `conformance/suite.json` `suiteVersion` 0.1.0 -> 0.2.0: the flipped vector replaces a published `expectedCanonicalJcs` and `expectedId`, which the suite's own rule calls a changed expectation (the breaking bump sits in the minor position while the suite is pre-1.0) diff --git a/conformance/fixtures/document/reject-metadata-term-non-nfc-array-element/case.cdx b/conformance/fixtures/document/reject-metadata-term-non-nfc-array-element/case.cdx new file mode 100644 index 0000000000000000000000000000000000000000..a202dce032cae28c4fb0e9d4f64873cb19396841 GIT binary patch literal 762 zcmaJ<%}&BV5H6y|i#I%QFv^|YZ!ofdU$T?l4#(a~3Kpq=wYg~#~ mi+rr_^J literal 0 HcmV?d00001 diff --git a/conformance/fixtures/document/reject-metadata-term-non-nfc-array-element/case.json b/conformance/fixtures/document/reject-metadata-term-non-nfc-array-element/case.json new file mode 100644 index 0000000..cb709e7 --- /dev/null +++ b/conformance/fixtures/document/reject-metadata-term-non-nfc-array-element/case.json @@ -0,0 +1,38 @@ +{ + "description": "A non-NFC string (`cafe` followed by U+0301 COMBINING ACUTE) carried as an ELEMENT of the array-valued `creator` term, while every string-valued term is conformant. Document Hashing section 4.3.2 binds the whole hashed basis, and section 4.3.1 keeps exactly five Dublin Core terms in it — projecting `creator`, `subject` and `language` as ARRAYS (a scalar value is coerced to a one-element array). It is the AUTHORED value that may be either a string or an array of strings, per dublin-core.schema.json, and this part authors the array form. So a reader that scanned only the terms authored as bare strings admits a non-NFC byte into the document identity through three of the five. The part is otherwise conformant — `title` is present and `creator` holds a non-empty member — so it projects cleanly and this is the section 4.3.2 scan arm rather than the malformed-metadata or missing-term rows. It carries the corpus's only array-shaped projected term, so it is the only fixture in which that shape reaches the scan. REJECT in every state.", + "layer": "document", + "requires": [ + "container", + "document" + ], + "clause": "Document Hashing sections 4.3.1, 4.3.2; State Machine section 5.4.3", + "recipe": { + "entries": [ + { + "name": "manifest.json", + "text": "{\"cdx\":\"0.1\",\"id\":\"pending\",\"state\":\"draft\",\"created\":\"2025-01-10T08:00:00Z\",\"modified\":\"2025-01-10T08:00:00Z\",\"content\":{\"path\":\"content/document.json\",\"hash\":\"sha256:5d69e5acc01e8b76df35531f9199eda7f594a972eec7d5718071842062fb39cc\"},\"metadata\":{\"dublinCore\":\"metadata/dublin-core.json\"}}" + }, + { + "name": "content/document.json", + "text": "{\"version\":\"0.1\",\"blocks\":[]}" + }, + { + "name": "metadata/dublin-core.json", + "text": "{\"version\":\"1.1\",\"terms\":{\"title\":\"T\",\"creator\":[\"A. Author\",\"cafe\\u0301\"]}}" + } + ] + }, + "expect": { + "documentDisposition": { + "atLeast": "REJECT", + "atMost": "REJECT" + }, + "findings": [ + { + "code": "CDX-E-PART-STRING-NOT-NFC", + "atLeast": "REJECT", + "atMost": "REJECT" + } + ] + } +} diff --git a/conformance/oracles/document_oracle.py b/conformance/oracles/document_oracle.py index c33504c..e3f7c37 100644 --- a/conformance/oracles/document_oracle.py +++ b/conformance/oracles/document_oracle.py @@ -29,6 +29,7 @@ Usage: document_oracle.py check """ +import ast import hashlib import io import json @@ -2245,6 +2246,13 @@ def confirm_clean(name, data): manifest = json.loads(mtext) except Exception as exc: # noqa: BLE001 return f"clean case manifest is not valid JSON: {exc}" + # Parseable JSON is not necessarily an OBJECT. Without this the arms below call + # `.get` on a list and raise AttributeError, and `check` does not wrap this call — + # so a clean fixture whose manifest was an array aborted the entire oracle run with + # a traceback rather than reporting a refusal. Found by driving this guard from + # `reject-manifest-not-object`, which is exactly what the selftest below does. + if not isinstance(manifest, dict): + return "clean case manifest is not a JSON object" path = (manifest.get("content") or {}).get("path") if isinstance(manifest.get("content"), dict) else None if not isinstance(path, str) or path not in cdm: return "clean case manifest does not reference an existing content part" @@ -2430,9 +2438,291 @@ def check(): return 0 +# Drives the clean-case guards in `confirm_clean` from the MALFORMED corpus. Each row is +# (fixture, message fragment, what the fixture carries). +# +# Why this table has to exist: an arm of `confirm_clean` only ever runs on a CLEAN fixture, +# and a fixture carrying the defect that arm looks for is by definition not clean. So no +# corpus case reaches the arm with its defect present, and deleting the arm outright left +# every gate green — the arms added by B1b-3c-1a were unmutatable in exactly that way. +# archive_oracle.py records the same trap — "Drive the REAL clean-case guard, not its +# parts" — though its own selftest is a single targeted probe rather than a table. +# +# The fragment matters as much as the fixture. `confirm_clean` returns on its FIRST failing +# arm, so `is not None` is satisfied by ANY later arm and would still pass with the arm +# under test deleted — the assertion has to be attributable to the arm it is pinning. +CLEAN_GUARD_CASES = ( + # Listed in the order `confirm_clean` runs them, so this table reads alongside it. + # --- the archive, the manifest and the content part --- + ("reject-manifest-absent", "has no readable manifest.json", + "no manifest.json at all"), + ("reject-manifest-duplicate-keys", "manifest has a duplicate key", + "a duplicate key in the manifest"), + ("reject-manifest-unparseable", "manifest is not valid JSON", + "a manifest that is not valid JSON"), + ("reject-manifest-not-object", "manifest is not a JSON object", + "a manifest that parses as an array rather than an object"), + ("reject-content-part-missing", "manifest does not reference an existing content part", + "a content reference naming a part the archive omits"), + ("reject-duplicate-json-keys", "content part is missing or has a duplicate key", + "a duplicate key in the content part"), + ("reject-content-unparseable", "content part is not valid JSON", + "a content part that is not valid JSON"), + # Two drivers, because the sweep is over EVERY declared part rather than the two + # named ones: a guard narrowed back to manifest+content still passes the first. + ("reject-dublincore-duplicate-keys", "part metadata/dublin-core.json has a duplicate key", + "a duplicate key in the Dublin Core part"), + ("reject-declared-part-duplicate-keys", "part metadata/doc.jsonld has a duplicate key", + "a duplicate key in a manifest-declared part that is not named .json by convention"), + + # --- B1b-3c-1a: content-part validity --- + ("reject-content-root-blocks-missing", "content root violates its envelope", + "a content root whose `blocks` is absent"), + ("reject-content-key-non-nfc", "stored-byte text invariants: ['non-nfc']", + "a non-NFC object KEY in the content half of the hashed basis"), + ("reject-content-text-ill-formed-and-non-nfc", "stored-byte text invariants: ['ill-formed']", + "an ill-formed UTF-16 sequence in the hashed basis"), + ("reject-metadata-term-non-nfc-array-element", "stored-byte text invariants: ['non-nfc']", + "a non-NFC Dublin Core term in the METADATA half of the hashed basis"), + # The three position rows assert the KIND as well as the arm. All three take the SAME + # arm, so without the kind any one of them would be satisfied by any position defect at + # all and would pin nothing about the field it names. That keeps these rows honest; it + # is not extra routing coverage, which CONFIRMERS already provides — misrouting `block` + # to `anchor` was measured to fail check() on its own, with the selftest disabled. + ("warn-anchor-position-out-of-range", "anchor-position finding(s): ['out-of-range:anchor']", + "an out-of-range position on a mark anchor"), + ("warn-block-reference-position-out-of-range", "anchor-position finding(s): ['out-of-range:block']", + "an out-of-range position on a block reference"), + ("warn-cross-reference-position-out-of-range", "anchor-position finding(s): ['out-of-range:xref']", + "an out-of-range position on a semantic cross-reference"), + ("warn-annotation-anchor-position-malformed", "out-of-hash annotation anchor carries a defective position", + "a structurally malformed position on an out-of-hash annotation anchor"), + + # --- B1b-2: the block/mark classifier, one row per tag it can return --- + # Two rows, because `block_bare` is emitted from two different places — a non-object in a + # block position (fail closed) and an unknown bare type (refused) — and one row cannot + # tell which path went silent. + ("reject-block-position-non-object", "classifier finding(s): ['block_bare']", + "a non-object in a block position"), + ("reject-block-type-unknown-bare", "classifier finding(s): ['block_bare']", + "an unknown BARE block type"), + ("warn-block-figure-cardinality", "classifier finding(s): ['block_malformed']", + "a known block type violating its own cardinality rule"), + ("ignore-block-namespaced-opaque-subtree", "classifier finding(s): ['block_ns']", + "an unknown NAMESPACED block type"), + ("reject-mark-type-unknown-bare", "classifier finding(s): ['mark_bare']", + "an unknown BARE mark type"), + ("warn-mark-anchor-no-id", "classifier finding(s): ['mark_malformed']", + "a known mark type missing a required field"), + ("ignore-mark-type-unknown-namespaced", "classifier finding(s): ['mark_ns']", + "an unknown NAMESPACED mark type"), + + # --- B1b-3b-1: metadata, out-of-hash parts and content references --- + ("reject-metadata-reference-malformed", "has no readable Dublin Core part", + "a Dublin Core reference that resolves to nothing"), + ("warn-metadata-part-unparseable", "Dublin Core part is not a JSON object", + "a Dublin Core part that is not a JSON object"), + ("warn-metadata-term-malformed", "Dublin Core is malformed or omits a required term", + "a Dublin Core part missing a required term"), + ("warn-custom-metadata-part-missing", "declares a path-only part reference that the archive omits", + "a path-only part reference the archive omits"), + ("warn-extension-data-part-unparseable", "unparseable out-of-hash extension data part", + "an unparseable out-of-hash extension data part"), + ("warn-annotation-anchor-dangling", "out-of-hash annotation anchored at no content id", + "an out-of-hash annotation anchored at no content id"), + # One row per reference tag, for the same reason the position rows carry their kind. + ("warn-anchor-dangling", "reference finding(s): ['anchor']", + "a dangling core anchor"), + ("warn-block-reference-dangling", "reference finding(s): ['block']", + "a dangling extension block reference"), + ("warn-cross-reference-dangling", "reference finding(s): ['xref']", + "a dangling semantic cross-reference"), + ("integrity-error-id-collision", "reference finding(s): ['collision']", + "two content nodes sharing an id"), + + # --- B1b-3b-2: hash-bound assets, presentation layers and config slots --- + ("warn-asset-index-missing", "missing a hash-bound part", + "a hash-bound part the archive omits"), + ("warn-asset-index-unusable", "asset index that cannot be used to resolve the category", + "an asset index that cannot resolve its category"), + ("reject-asset-index-path-divergent", "asset index path that differs from the derived location", + "an asset index declared at a path other than its derived location"), + ("warn-asset-index-hash-mismatch", "asset index does not match its declared hash", + "an asset index whose bytes do not match its declared hash"), + ("warn-asset-hash-mismatch", "asset or variant does not match its declared hash", + "an asset whose bytes do not match its declared hash"), + ("warn-asset-variant-missing", "lists an image variant the archive omits", + "a listed image variant the archive omits"), + ("warn-asset-reference-dangling", "references an asset that resolves to no registered asset", + "content referencing an unregistered asset"), + ("warn-presentation-hash-mismatch", "presentation layer does not match its declared hash", + "a presentation layer whose bytes do not match its declared hash"), + ("warn-presentation-undeclared", "ships a presentation file no manifest.presentation[] entry declares", + "a presentation file no manifest entry declares"), + ("warn-presentation-reference-dangling", "presentation targets a block id the content does not define", + "a presentation targeting an undefined block id"), + ("warn-config-part-missing", "config-slot side file that is absent or unparseable", + "a config-slot side file that is absent"), + ("warn-config-file-hash-mismatch", "config-slot side file does not match its declared hash", + "a config-slot side file whose bytes do not match its declared hash"), + + # --- semantic cross-references, footnotes, and the hash/id recomputes --- + ("warn-citation-and-glossary-dangling", "dangling cross-reference(s): ['citation', 'glossary']", + "dangling citation and glossary references"), + ("warn-footnote-reference-dangling", "footnote resolution defect(s): ['dangling']", + "a footnote mark resolving to no block"), + ("warn-footnote-reference-ambiguous", "footnote resolution defect(s): ['ambiguous']", + "a footnote marker resolving to more than one block"), + ("warn-file-and-id-mismatch", "content.hash does not match the stored content bytes", + "a content.hash that does not match the stored bytes"), + ("warn-asset-document-id-mismatch", "recomputed document id does not match manifest.id", + "a manifest.id that is not the recomputed canonical id"), +) + + +# The archive-unreadable arm is the one refusal `confirm_clean` has that NO document fixture +# can drive: every document fixture is a readable zip, and the only two archives in the corpus +# that are not — reject-truncated-no-eocd, reject-unsafe-overlong-utf8 — are container-layer, +# which the rows above resolve no path for. `confirm_clean` takes BYTES rather than a fixture, +# so it is driven directly, the same shape as archive_oracle.py's targeted probe. Without it +# the coverage assertion below could not reach 37 of 37 and would have to carry an exemption. +UNREADABLE_ARCHIVE_PROBE = ( + b"PK\x03\x04 not a readable archive", + "zipfile could not read it", + "bytes that are not a readable archive", +) + + +def _confirm_clean_arms(): + """{line number: static message text} for every REFUSAL return in `confirm_clean`. + + Derived from this file's own AST rather than listed, so the count cannot drift from the + function it describes. Nested callables are SKIPPED: `ast.walk` descends into them, so a + local helper's `return` would be enumerated as an arm — and the tracer below filters on + the frame's code name, so nothing could ever mark it reached. That combination produces a + selftest failure no row can satisfy, pointing the maintainer at a defect that does not exist. + """ + fn = next(n for n in ast.walk(ast.parse(io.open(__file__, encoding="utf-8").read())) + if isinstance(n, ast.FunctionDef) and n.name == "confirm_clean") + + def static_text(node): + if isinstance(node, ast.Constant) and isinstance(node.value, str): + return node.value + if isinstance(node, ast.JoinedStr): + return "".join(v.value for v in node.values + if isinstance(v, ast.Constant) and isinstance(v.value, str)) + return "" + + arms = {} + + def visit(node): + for child in ast.iter_child_nodes(node): + if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)): + continue # a nested helper's return belongs to the helper, not to this arm set + if isinstance(child, ast.Return) and not ( + child.value is None + or (isinstance(child.value, ast.Constant) and child.value.value is None) + ): + arms[child.lineno] = static_text(child.value) + visit(child) + + visit(fn) + return arms + + +def selftest(): + """Assert the clean-case guards REFUSE every document carrying what they look for. + + Drives the REAL `confirm_clean`, not the helper predicates behind it. Asserting the + helpers directly is the trap archive_oracle.py records: such a selftest passes with the + guard removed from the clean-case path entirely, which is the only place it does any work. + + Two properties, and the second is what keeps the first honest: + + 1. Every driver is refused BY ITS OWN ARM, checked on the refusal message rather than on + "was refused at all" — `confirm_clean` returns on its first failing arm, so a bare + is-not-None assertion is satisfied by any LATER arm and survives deleting the one + under test. + 2. Every refusal arm is reached by some driver. Without this the table is a snapshot: it + restores coverage for the arms present today while adding nothing that keeps a NEWLY + added arm from being silent — which is exactly how the four B1b-3c-1a arms came to be + unmutatable. The arms are enumerated from the AST and the drivers traced, so a new + arm with no row fails here instead of passing quietly. + + A CONSEQUENCE FOR FIXTURE AUTHORS, since nothing else states it: each driver below must + trip the arm its row names and no EARLIER one. Giving one of them a second, unrelated + defect makes an earlier arm fire, which both fails that row and leaves its own arm + unreached — so the two reports are correlated and the second is diagnosed as shadowing + rather than as a missing row. + + Failures accumulate rather than returning on the first, so one rotted fragment reports + every other rotted row in the same run instead of costing a run each. + """ + arms = _confirm_clean_arms() + reached = set() + failures = [] + shadowed_fragments = [] + + def trace(frame, event, arg): + if frame.f_code.co_name != "confirm_clean": + return None + if event == "line" and frame.f_lineno in arms: + reached.add(frame.f_lineno) + return trace + + def drive(label, data, fragment, why): + verdict = confirm_clean(label, data) + if verdict is None: + failures.append(f"the clean-case guards blessed {label}, which carries {why} " + f"— the guard for it is vacuous or absent") + elif fragment not in verdict: + failures.append(f"{label} carries {why}, but the clean-case guards refused it for a " + f"different reason: {verdict!r} (expected to contain {fragment!r})") + shadowed_fragments.append((fragment, label)) + + previous = sys.gettrace() # restore, do not clobber: a caller may be profiling or tracing + sys.settrace(trace) + try: + for fixture, fragment, why in CLEAN_GUARD_CASES: + path = os.path.join(FIXTURES, fixture, "case.cdx") + if not os.path.exists(path): + failures.append(f"fixture {fixture} is missing") + continue + drive(fixture, open(path, "rb").read(), fragment, why) + drive("", *UNREADABLE_ARCHIVE_PROBE) + finally: + sys.settrace(previous) + + for line in sorted(set(arms) - reached): + # An arm goes unreached for two very different reasons, and the remedies are opposite. + # If a row exists but its driver was refused by an EARLIER arm, the row is not missing + # — the fixture grew a second defect — and telling the maintainer to add a row makes + # them add a duplicate. Correlate the two reports rather than emitting both blind. + shadow = next(((frag, label) for frag, label in shadowed_fragments + if frag and frag in arms[line]), None) + if shadow is not None: + failures.append(f"confirm_clean line {line} went unreached because its driver " + f"{shadow[1]} was refused by an EARLIER arm — fix that fixture, " + f"do NOT add a row") + else: + failures.append(f"confirm_clean line {line} is a refusal arm no driver reaches: " + f"{arms[line] or '(dynamic message)'} — add a row to " + f"CLEAN_GUARD_CASES that carries that defect, or a probe if no " + f"fixture can") + + for failure in failures: + print(f"FAIL selftest: {failure}") + if failures: + print(f"{len(failures)} selftest failure(s).") + return 1 + print(f"selftest: {len(CLEAN_GUARD_CASES) + 1} drivers reach all {len(arms)} refusal arms " + f"in confirm_clean") + return 0 + + def main(): if len(sys.argv) >= 2 and sys.argv[1] == "check": - sys.exit(check()) + sys.exit(selftest() or check()) print("usage: document_oracle.py check", file=sys.stderr) sys.exit(2) diff --git a/conformance/suite.json b/conformance/suite.json index 47a4fb4..b28c1e0 100644 --- a/conformance/suite.json +++ b/conformance/suite.json @@ -2,7 +2,7 @@ "normative": false, "$comment": "NON-NORMATIVE. This is the entry-point manifest for the CDX conformance suite, a testing artifact that references the specification and never extends it. The specification (spec/) and the published schemas (schemas/) are the only normative artifacts. Where anything here appears to disagree with spec/, the specification governs.", "suite": "cdx-conformance", - "suiteVersion": "0.2.2", + "suiteVersion": "0.2.3", "specVersion": "0.1", "title": "CDX Conformance Suite", "description": "An independently-runnable suite that lets a third-party implementation check itself against the CDX specification. This manifest binds a suite version to the specification version its expectations were derived against.", diff --git a/scripts/check-fixtures.ts b/scripts/check-fixtures.ts index f7af411..6a3b838 100644 --- a/scripts/check-fixtures.ts +++ b/scripts/check-fixtures.ts @@ -1,7 +1,7 @@ #!/usr/bin/env npx tsx /** - * Gate for the Level-1 container fixture corpus (the document track's front door). + * Gate for the committed fixture corpus, container and document layers alike. * * This gate proves the committed corpus is coherent and the reference reader is * capable; the end-to-end VERDICT evaluation (does the reader disposition each @@ -13,21 +13,31 @@ * fixture fails. Mirrors the vector byte round-trip gate. * 2. Schema: every committed case.json validates against fixtures.schema.json. * 3. No orphans: every committed case directory maps to a corpus case. - * 4. Reader codes: every code zip-reader.ts can emit is registered in errors.json. - * 5. Deflate support: method 8 round-trips in-memory with a verified CRC (proving + * 4. Content-hash coherence: every document fixture's declared content.hash agrees + * with the bytes its recipe stores, and disagrees exactly where the case expects + * CDX-E-FILE-HASH-MISMATCH. check:conformance cannot assert this — see below. + * 5. Reader codes: every code zip-reader.ts can emit is registered in errors.json. + * 6. Deflate support: method 8 round-trips in-memory with a verified CRC (proving * the mandatory algorithm works WITHOUT committing a non-deterministic archive). - * 6. Independent oracle: archive_oracle.py confirms each malformation is actually - * present in the committed bytes. + * 7. Independent archive oracle: archive_oracle.py confirms each container-layer + * malformation is actually present in the committed bytes. + * 8. Independent document oracle: document_oracle.py does the same for the part layer, + * and its selftest proves its clean-case guards refuse what they check for. + * + * Plus one unnumbered check between 3 and 4: every authored disposition interval is + * well-formed (atLeast <= atMost), which the case schema does not constrain. */ import * as fs from 'fs'; import * as path from 'path'; import { execFileSync } from 'child_process'; +import { isValidContentHash, algorithmOf, hashBytes, CanonicalizationError } from './lib/canonicalize.js'; import { createAjv } from './lib/ajv-utils.js'; import { FIXTURE_CORPUS, caseJson } from './lib/fixture-corpus.js'; import { buildZip } from './lib/zip-writer.js'; import { readArchive, crcMatches, CODE } from './lib/zip-reader.js'; import { CODE as DOC_CODE } from './lib/document-verdict.js'; +import { ASSET_CODE } from './lib/asset-index.js'; import { CLASSIFIER_CODE } from './lib/content-classifier.js'; import { isRegisteredCode } from './lib/error-codes.js'; import { isWellFormedInterval } from './lib/disposition.js'; @@ -109,14 +119,254 @@ for (const c of FIXTURE_CORPUS) { } if (intervalOk) ok(`all ${FIXTURE_CORPUS.length} fixtures have well-formed disposition intervals`); -// 4. Reader + document-mapper + content-classifier codes registered. +// Hash coherence: every hash a document fixture's manifest DECLARES must agree with the bytes +// its recipe stores — and must DISAGREE exactly where the case expects the mismatch code for +// that reference's class. +// +// This cannot be delegated to check:conformance, which is why it lives here. The fixture +// comparator is a SUBSET check over findings, so an unexpected hash-mismatch finding is +// tolerated outright; and on a case whose expected disposition is already WARNING or REJECT it +// cannot move the document disposition either, since a hash mismatch is a WARNING in draft and +// review (every fixture is one or the other). Measured on `reject-content-key-non-nfc`: flipping +// one hex digit of its content hash leaves all 366 conformance checks green while this gate +// names it. The exception is the five cases expecting exactly [IGNORE, IGNORE], where the extra +// WARNING does move the disposition and check:conformance catches it — so the hole this closes +// is the WARNING and REJECT population, not literally every non-clean fixture. +// +// Scope is every {path|index, hash} pair reachable in the MANIFEST. That deliberately stops at +// the manifest: the asset entries listed INSIDE an index file are a second level whose paths are +// resolved relative to the index's own location, and re-deriving that here would duplicate +// asset-index.ts's resolution — the kind of second implementation that drifts and then hides the +// defect it was added to catch. Those hashes are pinned instead by the oracle's asset arms, but +// SAY WHAT THAT LEAVES: those arms run only on CLEAN fixtures, so an asset whose bytes diverge +// from the hash listed inside an index is unpinned on a non-clean one. Measured on +// `integrity-error-id-collision-distinct-assets`: corrupting its asset bytes leaves +// check:fixtures, check:conformance, check:document-oracle, check:document-id and +// check:structural all green. That is this same hole one level down, and the boundary is drawn +// here deliberately rather than because the residue is empty. +// 4. Hash coherence. +console.log('\nFixture hash coherence:'); +{ + const isObj = (v: unknown): v is Record => + typeof v === 'object' && v !== null && !Array.isArray(v); + + // The mismatch code a reference's class carries, decided by the manifest key it hangs under. + // `security` is NOT a config slot: manifest.security.accessControl is the one security-block + // reference carried as a {path, hash} (manifest-projection.ts:454), the config slots are + // exactly academic/semantic/legal/collaboration, and NOTHING in the reader verifies + // accessControl's hash against stored bytes. So no code names a disagreement there, and + // claiming one would make such a fixture unauthorable — the gate would demand a code the + // implementation never emits, and check:conformance would then fail on that same code. + // `null` keeps the forward direction (the hash must still match its bytes) without inventing + // an attribution for the reverse one. + const codeForTopKey = (key: string): string | null => + key === 'content' ? DOC_CODE.FILE_HASH_MISMATCH + : key === 'assets' ? ASSET_CODE.ASSET_INDEX_HASH_MISMATCH + : key === 'presentation' ? DOC_CODE.PRESENTATION_HASH_MISMATCH + : key === 'security' ? null + : DOC_CODE.CONFIG_FILE_HASH_MISMATCH; + + interface Ref { where: string; path: string; hash: unknown; code: string | null } + const refsOf = (manifest: unknown): Ref[] => { + const out: Ref[] = []; + if (!isObj(manifest)) return out; + const visit = (node: unknown, where: string, code: string | null): void => { + if (Array.isArray(node)) { node.forEach((el, i) => visit(el, `${where}[${i}]`, code)); return; } + if (!isObj(node)) return; + // `index` is the asset category's path key; `path` is every other reference's. A + // reference with no `hash` key is a path-only binding (metadata.dublinCore, provenance) + // and binds no bytes, so it is not this check's business. + const pathKey = typeof node.path === 'string' ? 'path' : typeof node.index === 'string' ? 'index' : null; + if (pathKey !== null && 'hash' in node) out.push({ where, path: node[pathKey] as string, hash: node.hash, code }); + // A node carrying a `hash` but whose path is MISTYPED would otherwise vanish here, which + // is the same silent hiding place the mistyped-`hash` branch below refuses to allow. + // Recorded with an empty path so it reaches that branch rather than dropping out. + else if (pathKey === null && 'hash' in node && ('path' in node || 'index' in node)) out.push({ where, path: '', hash: node.hash, code }); + for (const k of Object.keys(node)) visit(node[k], `${where}.${k}`, code); + }; + for (const k of Object.keys(manifest)) visit(manifest[k], k, codeForTopKey(k)); + return out; + }; + + const documentCases = FIXTURE_CORPUS.filter((c) => c.layer === 'document'); + let compared = 0; + let malformed = 0; + const declined: string[] = []; + const noComparableRef: string[] = []; + let coherent = true; + + for (const c of documentCases) { + const manifestEntry = c.recipe.entries.find((e) => e.name === 'manifest.json'); + let manifest: unknown = undefined; + if (manifestEntry?.text !== undefined) { + try { manifest = JSON.parse(manifestEntry.text); } catch { manifest = undefined; } + } + const expected = new Set(c.expect.findings.map((f) => f.code)); + let comparableHere = 0; + const disagreed = new Set(); + + for (const ref of refsOf(manifest)) { + if (ref.path === '') { + // Recorded by the walk with an empty path because `path`/`index` was present but not a + // string. Treated exactly as a mistyped `hash` is, rather than dropping out: the pair + // binds bytes and a mistyped path unbinds it just as effectively. + // + // EXCEPT for a config slot, where the reader reports NOTHING for this shape — see #120, + // which is open on whether it should and under which code. Demanding one here would make + // such a fixture unauthorable: this gate would require the code and check:conformance + // would then fail because the reader never emits it. The gap is recorded there, not + // asserted here, until the specification decision is taken. + if (ref.code === DOC_CODE.CONFIG_FILE_HASH_MISMATCH || ref.code === null) { + declined.push(`${c.name}:${ref.where} (mistyped path on a config-slot reference; #120)`); + continue; + } + if (!expected.has(ASSET_CODE.MANIFEST_REFERENCE_MALFORMED) && !expected.has(ASSET_CODE.MANIFEST_HASH_MALFORMED)) { + fail(`${c.name}: ${ref.where} carries a hash beside a MISTYPED path, but the case expects neither ${ASSET_CODE.MANIFEST_REFERENCE_MALFORMED} nor ${ASSET_CODE.MANIFEST_HASH_MALFORMED} — the hash binds nothing and nothing says so`); + coherent = false; + } + malformed++; + comparableHere++; + continue; + } + const entry = c.recipe.entries.find((e) => e.name === ref.path); + // An entry with no `text` is a real ZERO-BYTE part — zip-writer stores `e.text ?? ''` — + // so its declared hash is comparable against the digest of the empty string. Only a + // reference whose target the archive omits entirely is uncomparable. + if (entry === undefined) { declined.push(`${c.name}:${ref.where} (target absent)`); continue; } + // Forward-looking: no document fixture manipulates its stored bytes today (the three + // recipes that do are container-layer, which this loop never reaches). Kept so that + // adding one cannot silently start comparing a digest against bytes the recipe altered. + if (entry.truncateStoredBytes !== undefined || entry.methodCodeOverride !== undefined) { + declined.push(`${c.name}:${ref.where} (stored bytes manipulated)`); + continue; + } + if (typeof ref.hash !== 'string') { + // A present-but-mistyped `hash` is a malformed manifest reference, not a coherence + // question. Asserted rather than skipped, so a mistyped one cannot hide here. + if (!expected.has(ASSET_CODE.MANIFEST_HASH_MALFORMED) && !expected.has(ASSET_CODE.MANIFEST_REFERENCE_MALFORMED)) { + fail(`${c.name}: ${ref.where} declares a non-string hash ${JSON.stringify(ref.hash)} but the case expects neither ${ASSET_CODE.MANIFEST_HASH_MALFORMED} nor ${ASSET_CODE.MANIFEST_REFERENCE_MALFORMED}`); + coherent = false; + } + malformed++; + comparableHere++; + continue; + } + if (!isValidContentHash(ref.hash)) { + if (!expected.has(ASSET_CODE.MANIFEST_HASH_MALFORMED)) { + fail(`${c.name}: ${ref.where} declares ${JSON.stringify(ref.hash)}, not a well-formed algorithm:hexdigest, but the case does not expect ${ASSET_CODE.MANIFEST_HASH_MALFORMED}`); + coherent = false; + } + malformed++; + comparableHere++; + continue; + } + let actual: string; + try { + // Algorithm from the value's own prefix, as every other hash comparison in the repo + // does. This THROWS for an algorithm the runtime cannot compute — blake3 is permitted + // by Container §3.2 and declared in capabilities.json, and document-verdict.ts:676 + // deliberately tolerates it rather than reporting. An uncomputable digest is a limit of + // this runtime, not an incoherent fixture, so it is declined here too. Uncaught it + // would abort the whole gate and take checks 5-8, including BOTH oracles, with it. + actual = hashBytes(algorithmOf(ref.hash), Buffer.from(entry.text ?? '', 'utf8')); + } catch (err) { + // ONLY an uncomputable algorithm is declinable. Swallowing every throw would let a + // future fault here be silently reclassified as one — the sibling call sites at + // document-verdict.ts:538 and :545 rethrow for the same reason. + if (!(err instanceof CanonicalizationError)) throw err; + declined.push(`${c.name}:${ref.where} (${algorithmOf(ref.hash)} not computable here)`); + continue; + } + compared++; + comparableHere++; + if (actual !== ref.hash) { + if (ref.code === null) { + // No published code names a disagreement on this reference — see codeForTopKey. The + // hash is still wrong, and saying so is the whole point of the forward direction. + fail(`${c.name}: ${ref.where} declares ${ref.hash} but the stored bytes digest to ${actual}; no defect code names a mismatch on this reference, so it is simply incoherent`); + coherent = false; + } else { + disagreed.add(ref.code); + if (!expected.has(ref.code)) { + fail(`${c.name}: ${ref.where} declares ${ref.hash} but the stored bytes digest to ${actual}, and the case does not expect ${ref.code} — recompute it, or the case carries an unnamed extra hash defect`); + coherent = false; + } + } + } + } + + // The other direction: a case naming a hash-mismatch code must actually carry one of that + // class. Without this a fixture can expect a mismatch it does not have and prove nothing. + for (const code of [DOC_CODE.FILE_HASH_MISMATCH, ASSET_CODE.ASSET_INDEX_HASH_MISMATCH, + DOC_CODE.PRESENTATION_HASH_MISMATCH, DOC_CODE.CONFIG_FILE_HASH_MISMATCH]) { + if (expected.has(code) && !disagreed.has(code)) { + fail(`${c.name}: expects ${code}, but every ${code === DOC_CODE.FILE_HASH_MISMATCH ? 'content' : 'matching'} hash it declares AGREES with the stored bytes — the case cannot be testing what it names`); + coherent = false; + } + } + if (comparableHere === 0) noComparableRef.push(c.name); + } + + // A COUNT IS NOT ENOUGH, and this is the failure this block was itself guilty of. Without a + // floor the gate prints a tick having compared nothing: any refactor that moves every fixture + // onto the uncomparable path leaves "0 fixtures ... agree" and exit 0, which is literally true + // and completely empty. So the skip set is pinned by NAME, and every fixture must be accounted + // for. Growing it is a deliberate edit, as it is for the orphan check above. + const EXPECTED_NO_REF = [ + 'reject-content-part-missing', // manifest names a content part the archive omits + 'reject-content-path-traversal', // content.path is ../secret.json; no entry matches + 'reject-content-reference-malformed', // content is a bare string, so no {path, hash} pair + 'reject-manifest-absent', // no manifest.json at all + 'reject-manifest-not-object', // manifest parses to an array + 'reject-manifest-unparseable', // manifest is not valid JSON + ]; + const unexpected = noComparableRef.filter((n) => !EXPECTED_NO_REF.includes(n)); + const absent = EXPECTED_NO_REF.filter((n) => !noComparableRef.includes(n)); + if (unexpected.length > 0) { + fail(`fixtures newly carrying no comparable hash reference: ${unexpected.join(', ')} — if that is intended, add them to EXPECTED_NO_REF with the reason`); + coherent = false; + } + if (absent.length > 0) { + fail(`EXPECTED_NO_REF names fixtures that DO now carry a comparable hash reference: ${absent.join(', ')} — remove them`); + coherent = false; + } + if (compared === 0) { + fail('the hash-coherence pass compared nothing — it is vacuous, not clean'); + coherent = false; + } + // A DECLINED reference is a declared hash this pass did NOT verify. Counting its fixture among + // those that "agree" is precisely the empty tick this block exists to remove, so declines are + // pinned by name and reported separately rather than folded into the total. Every one today is + // a reference whose target the archive deliberately omits — there are no bytes to hash. + const EXPECTED_DECLINED = [ + 'reject-asset-index-path-divergent:assets.images (target absent)', + 'reject-content-part-missing:content (target absent)', + 'reject-content-path-traversal:content (target absent)', + 'warn-asset-index-missing:assets.images (target absent)', + 'warn-config-part-missing:semantic.bibliography (target absent)', + 'warn-presentation-part-missing:presentation[0] (target absent)', + ]; + const newDeclines = declined.filter((d) => !EXPECTED_DECLINED.includes(d)).sort(); + const goneDeclines = EXPECTED_DECLINED.filter((d) => !declined.includes(d)); + if (newDeclines.length > 0) { + fail(`declared hashes newly going unverified: ${newDeclines.join(', ')} — if that is intended, add them to EXPECTED_DECLINED with the reason`); + coherent = false; + } + if (goneDeclines.length > 0) { + fail(`EXPECTED_DECLINED names references that ARE now verified: ${goneDeclines.join(', ')} — remove them`); + coherent = false; + } + if (coherent) ok(`${compared} declared hash(es) across ${documentCases.length - noComparableRef.length} document fixtures agree with their stored bytes (${malformed} deliberately malformed, ${declined.length} declined and named, ${noComparableRef.length} fixtures carry no comparable reference)`); +} + +// 5. Reader + document-mapper + content-classifier codes registered. console.log('\nReader defect codes:'); const codes = [...Object.values(CODE), ...Object.values(DOC_CODE), ...Object.values(CLASSIFIER_CODE)]; const unreg = codes.filter((c) => !isRegisteredCode(c)); if (unreg.length === 0) ok(`all ${codes.length} reader/document/classifier codes registered in errors.json`); else fail(`reader/document/classifier codes not registered in errors.json: ${unreg.join(', ')}`); -// 5. Deflate support (in-memory; never committed, since deflate bytes are not +// 6. Deflate support (in-memory; never committed, since deflate bytes are not // guaranteed stable across zlib builds). console.log('\nCompression support:'); { @@ -130,7 +380,7 @@ console.log('\nCompression support:'); } } -// 6. Independent archive oracle: container-layer defect-presence in the bytes. +// 7. Independent archive oracle: container-layer defect-presence in the bytes. console.log('\nIndependent archive oracle (python3, defect-presence):'); try { const out = execFileSync('python3', [path.join(CONF, 'oracles', 'archive_oracle.py'), 'check'], { encoding: 'utf8' }); @@ -142,7 +392,7 @@ try { fail(`archive_oracle.py check failed${e.message ? `: ${e.message}` : ''}`); } -// 7. Independent document oracle: part-layer defect-presence in the bytes. +// 8. Independent document oracle: part-layer defect-presence in the bytes. console.log('\nIndependent document oracle (python3, defect-presence):'); try { const out = execFileSync('python3', [path.join(CONF, 'oracles', 'document_oracle.py'), 'check'], { encoding: 'utf8' }); diff --git a/scripts/lib/fixture-corpus.ts b/scripts/lib/fixture-corpus.ts index 443f592..019a242 100644 --- a/scripts/lib/fixture-corpus.ts +++ b/scripts/lib/fixture-corpus.ts @@ -2206,6 +2206,25 @@ export const FIXTURE_CORPUS: AuthoredCase[] = [ ), expect: CLEAN, }, + { + name: 'reject-metadata-term-non-nfc-array-element', + description: 'A non-NFC string (`cafe` followed by U+0301 COMBINING ACUTE) carried as an ELEMENT of the array-valued `creator` term, while every string-valued term is conformant. Document Hashing section 4.3.2 binds the whole hashed basis, and section 4.3.1 keeps exactly five Dublin Core terms in it — projecting `creator`, `subject` and `language` as ARRAYS (a scalar value is coerced to a one-element array). It is the AUTHORED value that may be either a string or an array of strings, per dublin-core.schema.json, and this part authors the array form. So a reader that scanned only the terms authored as bare strings admits a non-NFC byte into the document identity through three of the five. The part is otherwise conformant — `title` is present and `creator` holds a non-empty member — so it projects cleanly and this is the section 4.3.2 scan arm rather than the malformed-metadata or missing-term rows. It carries the corpus\'s only array-shaped projected term, so it is the only fixture in which that shape reaches the scan. REJECT in every state.', + layer: 'document', + requires: ['container', 'document'], + clause: 'Document Hashing sections 4.3.1, 4.3.2; State Machine section 5.4.3', + recipe: documentArchive( + manifestJson(), + [ + { name: 'content/document.json', text: CLEAN_CONTENT }, + // Written as a raw JSON literal rather than through `dublinCoreJson`, for the reason + // the content fixtures above give: the `\u0301` escape keeps the STORED bytes ASCII, so + // the decomposed sequence cannot be silently normalized away by an editor and leave the + // fixture passing for the wrong reason. + { name: 'metadata/dublin-core.json', text: '{"version":"1.1","terms":{"title":"T","creator":["A. Author","cafe\\u0301"]}}' }, + ], + ), + expect: rejectOnly('CDX-E-PART-STRING-NOT-NFC'), + }, // === B1b-3c-1a: anchor POSITION validity (03a §2.2, §3, §7.3) ====================== // diff --git a/scripts/test-document-verdict.ts b/scripts/test-document-verdict.ts index d352ca0..37c8129 100644 --- a/scripts/test-document-verdict.ts +++ b/scripts/test-document-verdict.ts @@ -950,6 +950,34 @@ test('a non-NFC PROJECTED Dublin Core term IS reported', () => { assert.ok(codes.has('CDX-E-PART-STRING-NOT-NFC'), '`title` projects into the document ID'); }); +for (const term of ['creator', 'subject', 'language'] as const) { + test(`a non-NFC element of the ARRAY-valued projected Dublin Core term \`${term}\` IS reported`, () => { + // §4.3.1 keeps exactly five terms and projects `creator`, `subject` and `language` as + // ARRAYS — "a scalar value is coerced to a one-element array". It is the AUTHORED value + // that may be a string or an array of strings (dublin-core.schema.json makes each a + // `oneOf`); after projection all three are arrays. So a violation inside an ELEMENT is + // inside the hashed basis exactly as a bare string is. + // + // All three are asserted, not just one: narrowing the scan loop to + // PROJECTED_STRING_TERMS + 'creator' was measured to leave the entire corpus green — + // 366/366, both oracles, every verdict test — so pinning one term leaves the other two + // unscanned in BOTH implementations, which is the shared-misreading case this suite exists + // to catch. + const terms: Record = { title: 'T', creator: 'A. Author' }; + terms[term] = term === 'creator' ? ['A. Author', 'cafe\u0301'] : ['ok', 'cafe\u0301']; + const dc = JSON.stringify({ version: '1.1', terms }); + const content = '{"version":"0.1","blocks":[]}'; + const hash = 'sha256:' + createHash('sha256').update(Buffer.from(content, 'utf8')).digest('hex'); + const codes = codesFor([ + { name: 'manifest.json', text: JSON.stringify({ cdx: '0.1', id: 'pending', state: 'draft', created: '2025-01-10T08:00:00Z', modified: '2025-01-10T08:00:00Z', content: { path: 'content/document.json', hash }, metadata: { dublinCore: 'metadata/dublin-core.json' } }) }, + { name: 'content/document.json', text: content }, + { name: 'metadata/dublin-core.json', text: dc }, + ]); + assert.ok(codes.has('CDX-E-PART-STRING-NOT-NFC'), 'an array ELEMENT is hashed basis exactly as a bare string is'); + assert.ok(!codes.has('CDX-E-METADATA-PART-UNPARSEABLE'), 'a string array is projectable, so this is the scan arm, not the malformed one'); + }); +} + // The three absence/presence properties the fixture comparator cannot express: it is a // SUBSET check, so "this document reports NOTHING" is unassertable there. Each pairs the // scan against `canonicalContent`, because the scan's scope claim is a claim ABOUT canon — @@ -986,6 +1014,37 @@ test('a split run in a `figure` `caption` rich array IS reported', () => { assert.throws(() => canonicalContent({ manifest: '{}', content: doc, dublinCore: '{}' }), 'and canon throws, via its merge arm'); }); +test('a non-text element BREAKS the run, so the strings either side are never concatenated', () => { + // The NEGATIVE control for the mid-array flush. `reject-content-split-combining-sequence-mid-array` + // pins that a run IS flushed and reported at a non-text element, but nothing pinned that the + // flush actually SEPARATES: deleting the `run = ''` reset leaves that fixture green, because + // the longer merged run still violates NFC. Only a document whose runs are individually clean + // can tell the two apart. + // + // The PAIR is the control. Both documents carry the same four text-node values in the same + // order and differ only by a paragraph between the second and third. Each run is NFC-stable + // alone ('ae', and U+0301 followed by 'c'); only the concatenation across the boundary is not. + // So a scan that fails to reset reports `separated`, and one that resets too eagerly stops + // reporting `adjacent` — the second assertion is what stops "never concatenate" passing. + // + // Both sit in a `blockquote`, whose `children` take `#/$defs/block` and so admit a text node + // beside a paragraph. Verified with the repo's own validator against a negative control (a + // paragraph's children holding a paragraph is INVALID, Content Blocks §7.2 rule 1). The + // shape is deliberately NOT at the root: the root array would work too, but its admissibility + // for a bare text node is settled only by the schema, and a control does not need to rest on + // that. `blockquote`, `listItem`, `tableCell`, `admonition` and `definitionDescription` all + // admit it. + const separated = '{"version":"0.1","blocks":[{"type":"blockquote","children":[{"type":"text","value":"a"},{"type":"text","value":"e"},{"type":"paragraph","children":[{"type":"text","value":"x"}]},{"type":"text","value":"\\u0301"},{"type":"text","value":"c"}]}]}'; + const adjacent = '{"version":"0.1","blocks":[{"type":"blockquote","children":[{"type":"text","value":"a"},{"type":"text","value":"e"},{"type":"text","value":"\\u0301"},{"type":"text","value":"c"}]}]}'; + assert.equal(collectStoredTextViolations(JSON.parse(separated)), null, 'the non-text element ends the run'); + assert.equal(collectStoredTextViolations(JSON.parse(adjacent))?.code, 'CDX-E-PART-STRING-NOT-NFC', 'and the same values adjacent DO violate'); + // Canon agrees in both directions, by its merge arm: with no marks to differ, the adjacent + // nodes merge into one value and the per-STRING check fires on it, while the separated pair + // merges into two individually clean values. + assert.doesNotThrow(() => canonicalContent({ manifest: '{}', content: separated, dublinCore: '{}' }), 'canon accepts the separated runs'); + assert.throws(() => canonicalContent({ manifest: '{}', content: adjacent, dublinCore: '{}' }), 'and throws on the adjacent ones'); +}); + test('the derived text-bearing set contains every block §3 names, and no container', () => { // TWO ONE-SIDED ASSERTIONS, NOT AN EQUALITY, and the difference is the point. §3 defines a // text-bearing block as "a block whose `children` are text nodes" and then illustrates it —