Runs: retry a failed run from a safe node/checkpoint without repeating successful side effects

[LouisDeconinck]

Current state

Duckle has durable run IDs, node outcomes, item checkpoints, content-addressed stage caching, partition/backfill primitives in progress, and cancellation.

A failed full run still generally requires starting the pipeline again and relying on each stage’s incremental/cache behavior.

Problem

For a long pipeline:

download -> parse -> normalize -> enrich -> validate -> publish

a transient failure near the end should not require repeating hours of deterministic upstream work.

At the same time, “resume from node” is unsafe if it silently skips or repeats side-effecting stages.

Proposed solution

Add an explicit retry command that creates a new linked run.

Examples:

duckle run retry <run_id>
duckle run retry <run_id> --from validate
duckle run retry <run_id> --failed-partitions

Safety checks

Before reuse, verify:

same release/pipeline hash, unless explicitly overridden
same normalized non-secret parameters
same partition
required upstream cache/artifacts still exist
content hashes match
runtime compatibility

If any prerequisite is unavailable, fail clearly or require the operator to choose an earlier start.

Side effects

Never implicitly reuse or repeat a side-effecting sink.

Classify stages as:

pure/deterministic and reusable
read-only but non-deterministic
side-effecting/idempotent
side-effecting/unsafe to repeat

For unsafe stages, require an explicit operator choice.

Atomic publish groups can safely rerun as a complete publication where supported.

Provenance

The new run should record:

parent_run_id
retry reason
requested start node/partition
reused stages and source run/cache keys
re-executed stages
operator/actor

The original failed run remains immutable.

Node output contract

A stage can be reused only when Duckle has a durable output reference and verified cache key, not merely because the old log says “succeeded.”

UI/API/MCP

Expose the same plan as a dry run before execution:

reuse nodes A-D
execute nodes E-G
refuse/require confirmation for sink H

Scope

This is not process-level checkpoint restoration.

It is a safe new run that reuses verified durable outputs.

Acceptance criteria

1. retrying a late failure can reuse valid deterministic upstream stages;
2. changing release/parameters invalidates reuse by default;
3. unsafe side effects are never repeated silently;
4. the retry is a new run linked to the original;
5. the dry-run plan explains every reuse/re-execution decision.

[SouravRoy-ETL]

Slice 1 is on main (4f746bf): duckle-runner retry <run_id> [--dry-run] [--allow-changed] [--rerun-sinks] [--json].

It delivers AC2, AC4, AC5 and the refusal half of AC3. It does not deliver AC1, and the reason is worth writing down, because I assumed otherwise until I read the code.

Why reuse cannot resume a chain yet. outcache::input_fingerprint forms a stage's cache key by running SELECT count(*), sum(hash(t::VARCHAR)) FROM <view> against the run's own temp database. The upstream relation must therefore already exist in this run before a downstream key can even be computed, and sources are refused outright (cache_input_view.as_deref()?). So the cache skips a stage's own compute given a materialised input; it never skips producing that input. For your download -> parse -> normalize -> enrich -> validate -> publish, a retry still re-downloads. Resuming a chain needs durable stage outputs addressable without re-deriving them, which is its own piece of work, not a flag on this one.

So the slice leads with the half that can bite rather than the half that is slow.

  run    extract                  it failed last time
  reuse  parse                    <ws>/cache/daily/parse/9f2c....parquet
  WRITE  publish                  a sink writes outside the run

It refuses before planning anything when the pipeline hash changed, the engine version changed, the run being retried actually succeeded, or it would write to a sink:

refused  : retry:would-rewrite-sinks
           this retry would write again to 1: publish. Nothing here can tell a sink
           that is safe to repeat from one that is not, so it will not be decided
           for you. Re-run with --rerun-sinks once you know repeating is safe.

A refusal plans nothing - it does not hand back decisions and rely on the caller checking the refusal first. There is a test pinning exactly that.

Reuse is verified by looking. A node is reused only when its recorded output is still on disk. The receipt saying a node succeeded is not evidence its output survived, and a cache pruned since is the ordinary way it does not. That is the test that keeps "verified" in AC1 from becoming decoration.

On provenance. Nothing recorded could answer "is this the same work?", so each run now writes a receipt under runs/receipts/ keyed by run id: pipeline path, pipeline hash, engine version, per-node status and kind, and the output-cache key each node's result was stored under. RunRecord could not do this job - it is capped at 50 entries, has no per-node detail and no pipeline identity, and RunResult is Serialize only, so a finished run was unreadable the moment the process ended. The hash is taken on the document as parsed, before apply_time_builtins, or a pipeline that stamps a dated path would look changed every day and reuse would never once apply.

One bug worth reporting, because it is the exact class this issue is about. My first version read sink-ness off the receipt. A run that dies at the first node records nothing for anything downstream, so every sink the failure never reached read as "not a sink" and the retry planned a quiet re-run of it. The tests passed - they had given the sink a recorded entry. Running it is what caught it. Sink-ness now comes from the pipeline being retried, not from what the broken run managed to record.

Not in this slice, deliberately:

• --from <node> does not exist, and is not accepted as a flag. compile_partial walks BFS backwards from a target and drops downstream sinks - the opposite of what --from means. Shipping a flag that always errors would be worse than not shipping it.
• Reuse only ever covers nodes with cacheOutput set - opt-in, and declared by six components out of ~382. A pipeline that never ticked the box reuses nothing, and every line of the plan says so.
• Only duckle-runner --pipeline runs write a receipt. The API, the scheduler, plan steps and the desktop app do not, so retry on their ids answers retry:no-receipt rather than guessing. RunRecord.run_id is only minted on POST /api/run/async today, so there is not much to attach to yet.
• Partition-aware retry (--failed-partitions) belongs with Feature: First-class partition definitions and durable, resumable backfill execution #295 rather than here.

Leaving this open for your read on the sink rule, which is the part I would most like to be wrong about in a useful direction.

[LouisDeconinck]

--rerun-sinks is the right default/escape hatch for me. I would not ask Duckle to infer sink idempotency in order to close this issue; refusing potentially side-effecting writes unless the operator explicitly opts in is the safer contract.

The important remaining blocker is AC1: a reusable stage/source output needs to be addressable without recreating all of its upstream relations in the new run.

I would consider the useful closure shape to be:

• a receipt/cache entry provides a durable output reference plus its hash/schema/cache key;
• the retry planner can bind that durable output directly as an upstream relation;
• when an expected reusable output is gone, the planner walks back to the earliest stage that must rerun and explains why;
• common run IDs/receipts across execution surfaces come from Feature: Durable asynchronous server runs with run IDs, status, logs and cancellation #259;
• partition-aware retry stays in Feature: First-class partition definitions and durable, resumable backfill execution #295.

That would let a late failure genuinely avoid repeating an expensive download/parse/OCR/enrichment chain while keeping all sink repetition explicit.

[LouisDeconinck]

There may now be a cleaner path to the remaining AC1 than when this was last discussed.

Since then, Duckle has converged further on durable run identity, and other features have introduced the useful pattern:

work completes
    ↓
durable output committed
    ↓
hash/reference recorded
    ↓
execution unit succeeds

That is now explicit for chunked extraction and external-component artifacts.

I wonder if the missing primitive for retry should therefore be a small generic durable node-output reference, rather than making retry understand each cache implementation independently.

Conceptually:

node output
    kind = relation | artifact
    uri/path
    sha256
    schema, where tabular
    cache/output key
    producing run_id
    node_id

Then a retry plan can do:

old node succeeded
    ↓
durable output reference exists?
    no  -> walk backwards and rerun from earliest required producer
    yes -> verify bytes/hash/schema
    ↓
bind durable output as this run's upstream relation
    ↓
skip upstream execution

For:

download -> parse -> normalize -> enrich -> validate -> publish

that finally allows a late failure to start from a verified durable normalize or enrich output without rerunning download.

I would keep the existing sink rule exactly as it is:

potential external write
-> refuse unless --rerun-sinks was explicitly supplied

There is no need to infer idempotency to solve AC1.

The main architectural property I would aim for is:

Retry reuses verified durable outputs; it never reuses a historical "succeeded" status by itself.

That also feels reusable outside retry later, rather than becoming a retry-specific checkpoint format.

[SouravRoy-ETL]

AC1 is in, on your shape. 87a7992.

The generic reference, not a retry checkpoint

nodeout::NodeOutput - kind (relation | artifact), uri, sha256, bytes, rows, columns, cache key, producing run id, node id - recorded on the receipt when a stage's output is durably stored.

Worth reporting what I found looking for it: three near-identical shapes already exist - backfill::SliceArtifact, plugin::Artifact, ArtifactRef - plus five separate whole-file sha256 helpers and four different atomic-write stories. None of the three says which node produced it in a way a planner can join on, and none carries a schema. So your instinct that this should be a shared primitive rather than a retry-private format is right by a wider margin than the issue implies. I have not converted the three onto it: introducing the type and moving one producer is reviewable, rewriting all three at once is not. It is the shape they can converge onto.

Reuse a verified output, never a historical status

Your architectural property, enforced literally. The old rule asked whether something was filed under a cache key - and a key is a hash of the inputs, so it says the work would produce the same answer, not that the answer is still on disk. The planner now reads the recorded uri, size and sha256 and checks that file. A node recorded by an older build, with a key and no durable output, is explicitly not evidence and says so.

cache_hit is gone from plan() rather than left as an ignored parameter.

The walk backwards, and a bug it exposed

I first wrote it forwards - "bind a reusable output if something that runs reads it". Running it printed:

run    download    nothing durable was recorded for it
reuse  parse       ...parse/c06b70...parquet (374 bytes, sha256 42f59a0d75e1)
run    normalize   it failed last time

The download was still running. The question is not "does anything read this node" but "is this node needed at all", and a node whose consumer is bound from a file reads that file, not its parents. So the walk starts at the ends and stops at every binding - which is the "walks back to the earliest stage that must rerun" you asked for, arrived at from the other direction. After the fix:

skip   download    nothing this retry has to produce depends on it
reuse  parse       ...parquet (374 bytes, sha256 42f59a0d75e1)
run    normalize   it failed last time
WRITE  publish     the previous run did not record this node
starts at: normalize (it failed last time)
bindings : 1 verified output(s) read instead of re-derived

A plan and a run that disagreed

The second bug, and the one I would most want a reviewer to check for. skip download was printed and the retry read the CSV anyway - binding rewrote bound stages and nothing removed the skipped ones. Skipped stages are now dropped from the compiled plan. A sink is never dropped or bound: it is an end, the walk always reaches it, and dropping one would silently skip the write it exists to do.

Demonstrated rather than asserted. With the source file deleted from disk:

$ ls in.csv
ls: cannot access 'in.csv': No such file or directory

$ duckle-runner retry run-manual-pipe-... --rerun-sinks
  normalize            error
  parse                ok (3 rows)

parse yields its rows from the verified parquet and download is never staged.

Unchanged, per your note

The sink rule is exactly as it was: a potential external write refuses unless --rerun-sinks is supplied, and nothing infers idempotency. A changed pipeline or engine still disables reuse entirely, so nothing is bound even when every file is still there.

1871 tests pass.

Two honest limits

Reuse still only reaches nodes that opted into cacheOutput - six components. That is where a durable relation output exists to record. The reference type is generic, so widening it is now a question of which producers record one rather than of the planner learning another cache; ordinary sinks record nothing hashed at all today, which is the largest surface it would newly cover.

One extra DuckDB query per cached node, to read the parquet's column names. It found its own bug: DESCRIBE names its first column column_name, and asking for name is a binder error that comes back as an empty list rather than a failure - so it silently recorded no columns until it was run against a real file.

[SouravRoy-ETL]

Two more of the safety checks are in, and one of them was a live gap rather than a missing nicety. 5307341.

"Same normalized non-secret parameters" was never implemented

RunReceipt carried parameters and plan() never read them. So a retry of a run bound with jurisdiction=BE would run with the pipeline's defaults while reusing a cached output computed for BE - an answer belonging to neither set, produced quickly and looking fine.

And my earlier caveat on this issue is stale, which is what makes it live. I wrote in August that only duckle-runner --pipeline writes a receipt, so there was "not much to attach to yet". That is no longer true: execute_one_with now calls retry::begin/write/finish with recorded_params on the receipt. Console, API and scheduled runs are retryable today - and they are precisely the runs that have parameters. The gap arrived when #259/#289 landed, without anything failing.

Fixed in two halves, because either alone leaves a hole:

• the retry replays the recorded set. That needed the headless path to bind parameters at all, which it never has - there is no --param on duckle-runner --pipeline. They now go through the same typed boundary the console uses, so an invalid value fails before the run rather than inside it.
• plan() refuses retry:parameters-changed when the effective set differs, naming the parameters that differ, including one dropped entirely.

The CLI cannot actually reach that refusal, because it always replays. The check is made anyway: a surface that supplies its own parameters must be refused rather than trusted, and "correct by construction" is only worth having if something notices when the construction changes.

The escape hatch was the one path where the check did not hold

Worth reporting because a test caught my own fix being incomplete. I first put parameters only in the refusal. --allow-changed lets a retry proceed, and reuse_allowed was still pipeline_hash == recorded && engine == recorded - so with --allow-changed and different parameters, the plan bound an output computed under the old values. Parameters are now part of reuse_allowed too: you may retry, never with reuse.

1904 tests pass. Verified by removing both halves: the differing-parameter retry is allowed through, and the escape hatch binds extract.parquet again.

Where #305 stands now

same release/pipeline hash        done
same normalized parameters        done
required upstream outputs exist   done
content hashes match              done
runtime compatibility             done
same partition                    NOT done

partition_key is on the receipt and still unchecked, and it is the same class of bug: a retry of one day's slice reusing an output computed for another. I did not fold it into this change because a partitioned retry is #295's --failed-partitions shape rather than this one's, and I would rather ask than guess: should retry <run_id> of a partitioned run replay the recorded partition the way it now replays parameters, or refuse and direct the operator to backfill retry? The second is less convenient and harder to get subtly wrong.

Still open from before: --from <node> does not exist, and reuse still only reaches the six components that opt into cacheOutput - an ordinary sink records nothing hashed, which remains the largest surface the generic output reference would newly cover.