Runs: compare two executions across code, inputs, state, quality, outputs, and performance

[LouisDeconinck]

Problem

When a pipeline’s output or duration changes, operators need to distinguish:

code changed
parameters changed
source artifact changed
incremental watermark changed
runtime/dependency changed
cache behavior changed
data-quality results changed
the same code simply ran slower

Duckle records much of this information, but a direct run comparison is still useful.

Proposed solution

Add:

duckle runs diff <run_a> <run_b> --json

and an equivalent web/API view.

Compare:

Identity

pipeline/release hash
Git commit
runtime/lock versions
environment
trigger/actor
partition/backfill
normalized non-secret parameters

Inputs/state

source/artifact URIs and hashes
incremental state before/after
parent/item/chunk counts
upstream release/run references

Execution

node SQL/config hashes
executed/reused/cached nodes
status and retries
duration
rows/bytes
memory/spill
HTTP retries/rate limits
reject/skip counts

Quality/output

contracts/checks/anomalies
published assets/tables
publish generation/release
output schema
row-count/aggregate summaries where already available

Privacy

Do not include:

secret parameters
authorization headers
raw row values by default
signed URL secrets

A full row-level data diff is a separate explicit operation and should not be the default.

Explanation

Highlight likely causes, for example:

parse stage slower because cache miss
row count changed after source artifact hash changed
output schema changed because release changed
same release/inputs but HTTP retries increased

These can be deterministic heuristics rather than AI-generated explanations.

Acceptance criteria

1. the diff shows code/runtime/input differences separately;
2. node performance and quality changes are comparable;
3. output is available as JSON and UI;
4. secret values remain redacted;
5. both runs remain immutable.

[SouravRoy-ETL]

Shipped on main (307d8ab).

duckle-runner runs diff <run_a> <run_b> [--json]

Real output from two runs of the same pipeline over a CSV that grew between them:

execution:
  node.out.rows                          3  ->  7
  node.src.durationMs                    61  ->  103
  durationMs                             172  ->  198
output:
  rows                                   3  ->  7

* Identical code, engine and parameters, but the output differs - which points at
  the sources rather than at Duckle.
* A node produced a different number of rows, which is visible from the receipts alone.

not compared:
  source content hashes and data-quality results: not recorded per run yet.

Against your acceptance criteria:

1. Code, runtime and input differences separate - grouped as code / runtime / invocation / inputs / execution / output. That grouping is the answer; "seventeen fields differ" helps nobody.
2. Node performance and quality comparable - the receipt now carries per-node rows and durations, which it did not before.
3. JSON - yes, versioned. UI - not done.
4. Secrets redacted - see below.
5. Both runs immutable - nothing in the diff path opens either run for writing.

On secrets. A parameter the pipeline declared secret is recorded as ***. A parameter in a pipeline that declares nothing is the harder case: nobody has said whether it is sensitive, and one of them may well be a password. Those are recorded as a digest of the value, so "this parameter changed between the two runs" stays answerable without a credential ever reaching a file - the same trade the #308 walk makes for context values. The redaction comes from the #317 substitution boundary, which is the only place that knows both the effective values (declared defaults included) and which of them are secret; a caller rebuilding the map from its own inputs would get both wrong.

Explanations are rules over recorded facts. Each one rests on a difference in the list above it, so a reader who disagrees can point at the rule. The most useful is the negative: same code, same engine, same parameters, output moved - which points outward, and is the reading operators most often need and least often reach on their own.

Absent is not zero. A run that failed at its second node has counts for nothing after it, and reading those as zero would report a collapse in volume that never happened. Same for a node missing from one side entirely.

What could not be compared is listed. A comparison that quietly omits what it could not see reads as "these are the same".

Two things running it exposed, both fixed here:

• The bare CLI was the only run surface that never appended a run-history record. The console, the scheduler and the desktop all do. So a duckle-runner <pipeline> run was invisible to the Runs tab, to alerting, to asset freshness, and to this command - my first live comparison reported "at least one run has no history record" for both runs.
• Two runs whose node row counts moved 3 to 7 reported no readable difference, because every output rule depended on a history record neither of them had. A node's row count is a difference in what was produced and the receipt alone carries it.

One structural change worth flagging. There were eight copies of the receipt-node mapping - the runner, MCP, three in the console, two in the scheduler, one in the desktop - so adding two fields meant finding all eight. #259 already shipped once having missed two surfaces, and the eighth here was found by the workspace type check rather than by looking. They are now one constructor.

Not done:

• No UI. The JSON is stable and versioned; the web/API view is not built.
• Source artifact hashes are not recorded per run, so "the source file changed" is inferred from row counts rather than known. This is the biggest remaining gap and the one I would do next.
• Data-quality results, HTTP retries and rate limits, memory and spill, reject and skip counts are not on the receipt.
• Git commit, lock versions, environment, partition/backfill identity are not recorded either; pipelineHash covers "did the document change" and nothing wider.
• Output schema is not compared; the cache key stands in for "different content" where a node has one.