Contracts: detect breaking producer-schema changes across downstream pipelines before deployment

[LouisDeconinck]

Current state

Duckle already has:

• per-pipeline schema/contracts and drift checks;
• Git diffs/review;
• a cross-pipeline asset catalog and downstream impact graph;
• inferred schemas and lineage.

The remaining gap is using those together to determine whether a producer change will break downstream consumers.

Problem

A pipeline can validate in isolation while still breaking another pipeline.

Examples:

remove or rename a column
change VARCHAR to INTEGER
make a previously non-null field nullable
change decimal precision/scale
change a nested JSON shape
change the semantic meaning while retaining the name

For a large country/source portfolio, manually opening every downstream pipeline is not reliable.

Proposed solution

Add a Git/release-time compatibility check.

Example:

duckle contracts check --base main --head HEAD --json

For each changed produced asset:

1. compare prior and candidate output contracts/inferred schemas;
2. find direct and transitive downstream consumers;
3. determine which consumers reference affected columns/types;
4. report breaking, potentially breaking, and additive changes.

Suggested compatibility rules

Configurable defaults:

add nullable column                 compatible
remove referenced column           breaking
rename referenced column           breaking
type widening                      usually compatible
type narrowing                     breaking
non-null -> nullable               potentially breaking
nullable -> non-null               breaking for historical data unless validated
decimal precision/scale change     rule-based

Output

Example:

{
  "producer": "source.company",
  "change": "column_removed",
  "column": "legal_form_code",
  "consumers": [
    {
      "pipeline": "normalize.company",
      "node": "map_legal_form",
      "reference": "legal_form_code"
    }
  ]
}

Waivers

Allow a reviewed waiver with:

reason
owner
expiry
affected contract/change hash

A generic permanent “ignore all” switch would be unsafe.

Runtime use

The first implementation can be CI/release-only.

Optional runtime enforcement can later block publishing an incompatible asset.

Relationship to release promotion

Release activation should run this check against the currently active release and show the affected downstream graph.

Scope

This does not require a central hosted schema registry. Git + Duckle’s catalog/contracts are sufficient.

Acceptance criteria

1. removing a consumed column identifies the exact downstream node;
2. additive safe changes do not block by default;
3. results are machine-readable;
4. waivers are explicit, scoped, and auditable;
5. the check can run without executing full production data.

[SouravRoy-ETL]

Shipped on main (a8976e6): duckle-runner contracts check --base <rev> [--format json|junit|sarif] [--strict].

Verified end to end on a real two-pipeline repo - producer writes /lake/orders.parquet, consumer filters on amt, commit, remove amt:

BREAKING               /lake/orders.parquet removes amt, read by consumer
possibly breaking      /lake/orders.parquet removes note, and nothing in this workspace reads it

1 breaking, 0 possibly breaking, against HEAD    (exit 1)

Severity depends on the reader, not the change, which is the part that makes this worth doing across pipelines rather than per pipeline. The same removal is additive in one workspace and an outage in another, and only the consumer graph tells them apart. A rename arrives as a removal plus an addition and is judged as the removal, because to a consumer binding the old name a rename and a deletion are the same event.

Your suggested rules, as implemented:

change	verdict
add a column	compatible - nothing can bind a column that did not exist
remove/rename a referenced column	breaking, and it names who
remove a column nothing here reads	possibly breaking, never "compatible"
type widening	compatible
narrowing a referenced column	breaking
previously non-null now nullable	possibly breaking

Three deliberate choices worth arguing with:

1. It compares against a git revision, not a stored contract. The previous schema already exists in the commit the change is proposed against. A separately stored contract is a second copy of the same fact that has to be kept in step, and the first time it drifts the check compares confidently against something nobody shipped. --base main is the whole configuration.

2. Only widenings that hold for EVERY value are called compatible. int32 -> int64, int32 -> decimal, anything -> text, date -> timestamp. Deliberately absent: int64 -> float64, decimal precision changes, anything that can round or overflow. Calling a conversion safe when it can lose a value is worse than saying nothing, because it gets believed - so those land as breaking-if-referenced instead.

3. "References" is over-broad, and uncertainty never resolves toward reassurance. A column counts as referenced if its name appears as a whole word in a consumer's declared schema or anywhere in its node properties - a predicate, a mapper expression, SQL. Whole-word matters: without it a column called id is referenced by every pipeline in the workspace and the report becomes noise nobody reads. It still over-reports, and that is the right direction: a false breaking costs someone thirty seconds, a false compatible costs an incident.

Relatedly, removing a column nothing in the workspace reads is reported as possibly breaking rather than compatible. The reference search only knows about pipelines, and a dashboard, a notebook or an external consumer is not one. --strict fails the build on those too, for a repository that would rather stop and look.

What it does not do yet:

• Only declared schemas. Where a producing node declares no schema on one side of the comparison it says nothing, rather than inferring one and comparing it against a declared one - that would manufacture changes nobody made. Inferred-schema comparison needs a run, which a pre-deployment gate does not have.
• Direct consumers only, not the transitive closure. The catalog can walk downstream, so this is a small extension, but a transitive report needs to distinguish "reads the changed column" from "reads something derived from it" or it will implicate half the workspace on every edit.
• Nested JSON shapes and semantic changes behind an unchanged name - your last two examples - are not detected. The first needs column-level structure the declared schema does not carry; the second is not visible to any mechanical check, and I would rather say so than pretend a diff catches it.
• No --head flag: it compares the working tree against --base, which is the CI shape. Comparing two arbitrary revisions is easy to add if you want it.

Reuses the --format work from #312, so SARIF puts each finding on the asset in Code Scanning.

[LouisDeconinck]

The direct-vs-transitive distinction makes sense. I agree that we should not label every downstream pipeline breaking when we cannot prove that the changed column propagates through the intermediate transformation.

I would still like the transitive closure to be surfaced, though, especially now that #308 has a reusable affected-pipeline graph.

Perhaps the semantics can be:

direct consumer + proven reference
    -> breaking

direct consumer + uncertain reference
    -> possibly breaking

transitive downstream consumer without column lineage proof
    -> affected / requires revalidation

So for:

producer -> normalized_company -> search_index -> API export

if producer removes a field used by normalized_company, the first edge is definitely breaking. search_index and API export do not need to be called breaking unless lineage proves it, but I would still want them surfaced as part of the blast radius.

That seems safer for larger workspaces without pretending Duckle has column-level lineage that it does not yet have.

[SouravRoy-ETL]

Shipped on main (57459d4) - your three tiers, in your words.

Your exact example, run against a real three-hop workspace:

BREAKING  lake/company.parquet removes vat, read by normalized;
          revalidate search_index (downstream, no column lineage to prove it either way)

{ "severity": "breaking",
  "affected":   ["normalized"],
  "revalidate": ["search_index"] }

• direct consumer + proven reference -> breaking
• direct consumer + no reference found -> possiblyBreaking
• transitive downstream, no lineage proof -> revalidate, surfaced and never failing the gate

You put your finger on why the third tier has to be its own thing rather than a weaker affected: proving a dropped column propagates through an intervening transform needs column lineage across that transform, and Duckle does not have it. Calling search_index breaking would be a confident wrong claim in front of the person deciding whether to merge; leaving it out entirely hides it from the blast radius. Naming it, and saying plainly why it is not being judged, is the only honest option.

It reuses #308's graph rather than walking again. The set comes from catalog::impact, which already returns a depth: depth 1 is the direct consumers the check was handling anyway, anything deeper is the transitive closure. One traversal, so the two cannot come to disagree about the shape of the workspace - which is the same reason validate --affected calls affected's own selection rather than a copy of it.

Two smaller calls:

• A direct consumer is not repeated at the lower tier. One pipeline shown twice reads as two problems, and the reviewer then has to work out that it is one.
• check_asset keeps its signature and its behaviour. The downstream set is an argument on a new function, so every existing caller and every existing test is unchanged - there is a test asserting the two-argument form still claims nothing about downstream.

What this still does not do: it says nothing about which downstream pipelines are more likely affected than others. Ranking them would need the column lineage the tier exists to admit is missing. When lineage.rs can resolve columns across a whole pipeline from a finished run rather than from a compiled plan, the top tier can absorb the ones it can prove and the third tier shrinks to what is genuinely unknown - which is the right end state and not something I can shortcut to now.