XSD: pin accepted schema dependency sets and gate parser-contract drift

[LouisDeconinck]

Context

#286 now provides XSD-derived XML parser contracts and records the exact XSD bytes used in run provenance.

Recent work in 4ca1cc6 also resolves multi-file schema sets through:

xs:include
xs:import
xs:redefine

and records every resolved schema document as a hashed input artifact in the signed run manifest.

That gives excellent after-the-fact provenance.

The remaining gap is before-the-run enforcement.

Problem

A publisher can replace the bytes behind an unchanged path or URL:

https://registry.example/schema/company.xsd

while the Duckle pipeline itself remains unchanged.

The next run then derives a different parser contract.

Today the run manifest can tell us afterwards:

previous XSD hash != current XSD hash

but the production ingestion may already have parsed and published data using the new contract.

For recurring registry feeds, I would like a mode where changing the authoritative parser contract requires explicit acceptance.

Desired behavior

Treat the complete resolved XSD dependency set as one versioned parser contract.

Conceptually:

root XSD
    ↓
resolve include/import/redefine closure
    ↓
canonical dependency set
    ↓
hash/fingerprint
    ↓
compare with accepted parser contract

A useful policy could be:

xsd:
  uri: schemas/company.xsd
  changePolicy: fail

with possible values:

allow
warn
fail

Backward-compatible default behavior can remain allow.

Dependency-set identity

The fingerprint should cover the resolved schema closure, not only the root document.

For example:

canonical URI/path
SHA-256
resolved namespace, where relevant

for every document.

This catches:

root schema changed
import changed
include changed
dependency added
dependency removed
dependency graph changed

even when the configured root URI remains identical.

Fail before parsing the large source

With:

changePolicy: fail

the check should happen after the schema set is resolved and hashed but before Duckle starts parsing the large XML input or performs downstream side effects.

For a 30 GB registry feed the desired flow is:

resolve XSD set
    ↓
compare contract fingerprint
    ↓
changed?
    yes -> stop
    no  -> start large XML parse

A parser-contract mismatch should not advance watermarks, baselines, publication state, or other deferred-success state.

Explain the change

Machine-readable output should identify what changed.

For example:

{
  "code": "xsd_contract_changed",
  "root": "schemas/company.xsd",
  "previous": "...",
  "current": "...",
  "changes": [
    {
      "uri": "schemas/common-types.xsd",
      "change": "modified",
      "previous_sha256": "...",
      "current_sha256": "..."
    },
    {
      "uri": "schemas/new-types.xsd",
      "change": "added",
      "sha256": "..."
    }
  ]
}

Where Duckle can cheaply derive it, also showing the resulting column/schema difference would be useful:

+ legal_form_code VARCHAR
turnover DECIMAL(18,2) -> DECIMAL(20,2)
address required -> optional

but dependency-level drift detection is sufficient for v1.

Explicit acceptance

Provide an explicit way for an operator or authorized agent to make a changed parser contract the new accepted version.

Conceptually:

duckle schema inspect <pipeline>
duckle schema accept <pipeline>

or XSD-specific equivalents.

The exact command naming is not important.

Important semantics:

• acceptance is explicit;
• acceptance is audited;
• a failed run never automatically accepts the observed schema;
• production policy can forbid agents from accepting a new parser contract;
• the previously accepted fingerprint remains available for comparison.

This is similar in spirit to manually accepting a new qa.baseline.

Run provenance

A run should record both:

accepted parser-contract fingerprint
observed/resolved XSD dependency hashes

so it is possible to prove that the run used exactly the contract that was approved.

The existing signed artifact records can remain the source of the individual XSD hashes.

Remote/offline behavior

This should compose with #288 and the network policy.

In a pinned/offline production environment:

accepted XSD dependency set
    -> exact known bytes

should be usable without silently fetching a new version from the publisher.

A remote schema changing should never be automatically accepted merely because the URL remained the same.

Scope

This does not require:

• full XSD validation;
• implementing the complete XML Schema standard;
• semantic compatibility analysis between arbitrary XSD versions;
• a hosted schema registry.

A first implementation can simply detect:

accepted dependency fingerprint != observed dependency fingerprint

and apply allow | warn | fail.

That already provides the important safety property.

Why this matters

For recurring official-register ingestion, the XSD is effectively part of the parser implementation.

The production invariant should therefore be:

Pipeline code and parser contract cannot change independently without the change being visible and, when configured, explicitly approved.

That makes registry ingestion considerably more reproducible and safer for unattended or agent-managed pipelines.

[SouravRoy-ETL]

Implemented on main (1116620). The first implementation you scoped is what shipped: fingerprint the resolved dependency set, compare against the accepted one, apply allow | warn | fail.

The contract is the set, not the root. An xs:include three levels down decides a column's type as much as the root does, so a root whose bytes never moved is no evidence that anything held. The fingerprint covers every document in the closure. It is canonical (entries sorted), so a schema that merely reorders its own imports is not a change, while editing any document, or adding or dropping one, is. Both properties are pinned by tests, including the one that matters most here:

fn a_changed_import_changes_the_contract_even_when_the_root_did_not()
fn membership_is_part_of_the_contract()
fn the_fingerprint_ignores_resolution_order()

Where it is remembered. .duckle/xsd_contracts, one <uri> <fingerprint> per line, greppable, deletable by hand. That is deliberately the same trust-on-first-use shape as this repo's SFTP known_hosts file, down to the escape hatch being "delete the line", because it is the same problem: something that legitimately rotates and also looks exactly like an attack.

The policy, as a node field (xsdChangePolicy, in the XSD section of src.xml):

value	on a change
warn (default)	says so once, accepts the new set, run continues
fail	refuses, and does not record the new fingerprint
allow	does not look, and does not record

fail not recording is the part worth calling out: accepting has to be a deliberate act, or the next run passes silently and the gate was theatre. Same reason allow does not write a contract nobody asked for.

I defaulted to warn rather than fail. fail is right for the unattended registry ingestion you are describing, but as a default it would turn the first legitimate reissue of any schema into a broken pipeline for people who never opted in. warn makes the drift visible - which is the actual complaint, that today it is invisible until after publication - without breaking a working feed.

Checked before parsing, not after. The check runs in the connector at schema-derivation time, before any document is read with the derived contract. It lives there rather than in the parser because only the connector knows every document that was actually loaded.

On the remote/offline half: this composes with the network policy as it stands (a local schema set may not fetch over the network at all, and a remote one goes through the shared agent where the allowlist applies), and a remote schema changing is now never silently accepted under fail. What it does not yet do is serve the accepted set from known bytes without contacting the publisher. That is the #288 offline-bundle shape rather than this one, and I would rather build it on this fingerprint than alongside it.

Not in v0.7.1, which was cut before this landed.

[LouisDeconinck]

The shipped dependency-set fingerprint and fail-before-parse gate solve the main parser-safety property very well.

I would keep #315 open only for the explicit acceptance workflow.

For production/agent use, deleting the entry from .duckle/xsd_contracts is manual, but it is not yet an audited and authorizable operation meaning "accept exactly this observed contract."

I would consider #315 complete once Duckle can:

• inspect the accepted versus currently observed fingerprint/dependency changes;
• explicitly accept the observed fingerprint;
• record actor, time and ideally a reason for that acceptance;
• allow server policy/RBAC to prevent an agent or restricted API key from accepting a new parser contract;
• retain the accepted contract fingerprint in run provenance.

I don't think semantic XSD compatibility analysis is required for closure.

Serving the exact accepted XSD bytes without contacting the publisher can also stay in #288's offline/reproducible-runtime scope.