Lineage: optionally emit OpenLineage run and dataset events from Duckle manifests/catalog

[LouisDeconinck]

Current state

Duckle already has internal run manifests, dataset/artifact lineage, column lineage, run IDs, and a workspace catalog.

An optional standard export would let organizations integrate Duckle with existing lineage/observability tools without replacing Duckle’s own catalog.

Proposed solution

Emit OpenLineage-compatible events for:

START
COMPLETE
FAIL
ABORT/CANCEL, where representable

Run/job identity

Include:

Duckle run ID
pipeline/job ID
release/Git identity
trigger
parent Plan/backfill run
partition

Dataset facets

Map known inputs/outputs:

database/catalog/schema/table
DuckLake table
object-store/file/artifact URI
schema
data-source type

Include column-lineage facets where Duckle can determine them.

Do not invent precise lineage for dynamic/opaque code; mark it unknown or omit it.

Delivery

Configure an HTTP endpoint with:

bounded retries
short timeout
local buffer/queue
redaction

Telemetry delivery must never fail the data run.

Security/policy

• apply URI/query-parameter redaction;
• never emit credentials;
• allow policy to disable or restrict external export;
• optionally hash internal dataset names.

Scope

This is export only. Duckle does not need to import/manage an external OpenLineage catalog.

Acceptance criteria

1. successful and failed Duckle runs emit correlated events;
2. known input/output datasets and schemas are included;
3. column lineage is emitted only when known;
4. export failure cannot fail the pipeline;
5. sensitive values are redacted.

[SouravRoy-ETL]

Shipped on main (628da2f).

// <workspace>/openlineage.json - absent means no events, no file, no network
{ "namespace": "prod-eu", "endpoint": "http://marquez:5000/api/v1/lineage" }

START    | prod-eu/nightly | runId 79f0c574-56ee-5025-b0f4-addee7acb28a
COMPLETE | prod-eu/nightly | runId 79f0c574-56ee-5025-b0f4-addee7acb28a
   outputs: file  ${workspace}/data/curated.parquet  rowCount 3  unresolved

Emitted from retry::begin and retry::finish, not from each caller. There are eight execution surfaces - desktop, console, CLI, MCP, two in the scheduler, Plans - and a feed covering six of them is one nobody can reason about, because the missing runs look like runs that never happened. I made that mistake on #259 and the fix is to wire it once.

Five judgement calls worth arguing with:

• The run id is derived, not generated. The spec requires a UUID and Duckle's are readable strings, so they are mapped by name (v5 over the run id). START and COMPLETE are emitted by different calls and must agree; a random id per event would produce two unrelated runs in the collector and no completed one at all. The Duckle id travels in a duckle run facet, so the mapping is reversible by looking rather than by computing.
• interrupted is ABORT, not FAIL. That is the Feature: Durable asynchronous server runs with run IDs, status, logs and cancellation #259 distinction preserved across the boundary: the run stopped being observed, it did not fail, and a consumer that treats those the same re-runs work that may well have completed.
• A run-time reference is marked, not asserted. You asked for dynamic references to be marked unknown rather than invented, and the case that actually occurs here is a dataset name still holding a ${...} - the catalog records what the pipeline says, and ${workspace}/data/orders.csv names different files in different workspaces. Those carry unresolved: true and a reason. This is the one I would have got wrong: it looked like a working export until I read the emitted event.
• Datasets come from the catalog joined to the receipt by node id, so the join is exact rather than a guess from names - and a node the run never reached is not reported as touched. Absent is not zero: a run that stopped at node one touched nothing after it.
• catalogAvailable is on every event. A workspace with no saved catalog still gets run identity, timing and outcome, but its empty inputs/outputs are unknown, not empty, and the event says which. catalog::load is used rather than load_or_rebuild: rebuilding scans every pipeline, and paying that on the path of every run start to decorate telemetry is the wrong trade.

On delivery and security. Events are appended to logs/openlineage.ndjson before the POST is attempted, so the file is the buffer you asked for - a collector that is down costs one short timeout and the events are already durable rather than held in memory and lost with the process. Nothing in the emission path returns an error. Catalog asset ids are already credential-free (they are built through catalog::public_address, which strips userinfo and DSN credential segments); query strings are stripped on top of that, so a signed object-store URL does not carry its signature off the machine. hashDatasetNames replaces the name with a digest and keeps the namespace - knowing a graph spans two Postgres instances is the point and it names no table. Off unless configured, and a config that cannot be parsed disables rather than falling back to defaults, which would export to a namespace nobody chose.

Not done:

• No column-lineage facet. lineage.rs resolves column-level lineage from the compiled SQL, but it works off a serialized plan rather than off a finished run, so wiring it here means resolving at emit time or carrying it on the receipt. It is the next piece and the one that makes this worth more than run status.
• No release or Git identity on the job facet - pipelineHash is there, a release is Feature: Promote and atomically activate immutable tested Duckle releases with rollback #297.
• No partition or backfill identity - that is Feature: First-class partition definitions and durable, resumable backfill execution #295; there is nothing to put in the facet yet.
• One POST attempt, no retry. The event is already on disk, and retrying on the path of a run finishing costs another timeout every time a collector is down. A shipper that drains the NDJSON would be the better place for retries.
• No dataSource or schema dataset facets. The catalog knows the declared columns for many assets, so the schema facet is cheap and worth adding.
• No policy switch. You asked that policy be able to restrict external export; today it is workspace config only, so a server policy cannot forbid it.