Parameters: add typed pipeline parameter schemas shared by every execution surface

[LouisDeconinck]

Context

#127 added runtime prompting for unresolved ${name} placeholders.

The current behavior is useful for interactive runs:

discover unresolved placeholder
    -> prompt for a string value
    -> merge with contexts
    -> substitute before execution

The remaining production requirement is an optional typed parameter contract that is shared by every execution surface.

Problem

Without declarations, Duckle cannot reliably determine:

• whether a value is missing;
• whether false, 0, a date and a string were intended;
• whether an unexpected parameter name is a typo;
• whether a parameter is safe to record in run history;
• how schedules, Plans, API clients, MCP and backfills should render and validate the same input;
• whether a secret value must never be persisted.

This becomes important for pipelines such as:

jurisdiction = BE | NL | GB
effective_date = DATE
full_refresh = BOOLEAN
api_token = SECRET

Proposed solution

Allow an optional pipeline-level parameter schema.

parameters:
  jurisdiction:
    type: string
    enum: [BE, NL, GB]
    required: true
    description: Registry jurisdiction

  effective_date:
    type: date
    required: true

  full_refresh:
    type: boolean
    default: false

  max_companies:
    type: integer
    minimum: 1

  api_token:
    type: secret
    required: true

Useful initial types:

string
integer
number
boolean
date
datetime
enum
secret

Useful constraints:

required
default
enum
minimum / maximum
pattern
description

One normalization boundary

Normalize and validate parameters once before compilation/execution.

Every execution surface should receive the same typed result:

desktop
web console
CLI
HTTP API
MCP
scheduler
Plans
partitions/backfills
retry

Reject:

unknown parameter
missing required parameter
invalid type
invalid enum/range/pattern
conflicting duplicate input

Errors should be machine-readable and identify the parameter and expected contract.

Secret handling

A secret parameter value must never be written to:

pipeline JSON
run history
logs
previews
manifests
MCP responses
audit payloads
release files

It can be supplied through an authorized runtime surface or secret reference.

Non-secret normalized parameters should be recorded in run metadata so a run can be reproduced and compared.

SQL safety

A typed parameter must not automatically become an arbitrary raw SQL fragment.

Where a parameter is used in SQL, prefer parameter binding or a deterministic type-aware literal representation. Raw SQL fragments should require a separate explicit mechanism.

Precedence

Define and document precedence, for example:

declared default
    < context/environment binding
    < Plan/schedule parameter set
    < explicit run input

The final normalized values should be visible before execution, with secrets redacted.

UI and machine-readable schema

The same declaration should drive:

• text, number, date and checkbox controls;
• enum dropdowns;
• required/default indicators;
• CLI help;
• HTTP/MCP input schemas;
• validation;
• generated documentation.

A command/API such as this would also be useful:

duckle parameters schema pipeline.json --json

Backward compatibility

Pipelines without a declaration should retain the current unresolved-placeholder discovery and string prompt behavior.

Migration to typed declarations can therefore be incremental.

Relationships

This should compose with:

• Feature: First-class partition definitions and durable, resumable backfill execution #295 partitions and backfills;
• Feature: Promote and atomically activate immutable tested Duckle releases with rollback #297 immutable releases;
• Runs: retry a failed run from a safe node/checkpoint without repeating successful side effects #305 safe retry;
• Runs: compare two executions across code, inputs, state, quality, outputs, and performance #309 run comparison;
• Developer experience: pipeline-aware SQL completion, binder diagnostics, and inferred output schemas #314 SQL tooling.

Acceptance criteria

1. A declared boolean/date/integer is normalized identically through desktop, CLI, API and MCP.
2. Unknown, missing and invalid values fail before execution.
3. Secret values never appear in persisted or inspection surfaces.
4. Non-secret normalized parameters appear in run provenance.
5. Schedules and Plans can bind validated named parameter sets.
6. Existing undeclared-placeholder pipelines remain compatible.

[SouravRoy-ETL]

Shipped on main (0c26f2a).

"parameters": {
  "jurisdiction":   { "type": "string",  "enum": ["BE","NL","GB"], "required": true },
  "effective_date": { "type": "date",    "required": true },
  "full_refresh":   { "type": "boolean", "default": "false" },
  "max_companies":  { "type": "integer", "minimum": 1 },
  "api_token":      { "type": "secret",  "required": true }
}

All eight types and all six constraints from your list, minus enum as a separate type - it is expressed as the enum constraint on a typed parameter, exactly as your own example writes it.

The one normalization boundary is the part I made structural rather than careful. Every surface reaches substitution through context::apply_params, so the contract is enforced there and nowhere else. That is deliberate: validating per surface is how the desktop ends up accepting a value the scheduler refuses, and the bug is then in neither of them. There is a test that removes the check and watches a bad enum value substitute happily.

Errors are machine-readable and complete. Stable codes - param:unknown, param:missing, param:type, param:enum, param:range, param:pattern - each naming the parameter and carrying an expected string describing the contract. Every problem is returned, not the first: a form that fills in one mistake per round trip is a guessing game, and an agent correcting itself should not need four attempts.

An undeclared name is refused rather than ignored, with the near names suggested. This is the one I would defend hardest: jurisdiciton is far more likely a typo than a new parameter, and ignoring it means the run silently used the default and produced plausible wrong output.

secret is a declared type, not a naming heuristic. Inferring it would protect api_token and miss credential, decided by whoever typed the name. A secret never reaches run history, and a constraint failure on one never echoes the value - there is a test asserting the literal value does not appear in the message. In history it is replaced rather than dropped, because a missing key reads as "never supplied" and "was this run given a token?" is worth being able to answer.

Two smaller calls:

• A refused set does not half-substitute. Validation happens before any node is touched, and the test asserts the placeholder is still intact after a refusal. A partially substituted document that then fails somewhere else would be much harder to reason about.
• A pattern the pipeline author wrote wrongly does not refuse the caller's value. That is the author's bug, and refusing the caller for it blames the wrong person.

Not done:

• No UI yet. The contract is enforced everywhere, but the desktop and console still render the Passing Input Parameters to Pipelines (Web Dashboard Integration) #127 string prompt rather than a typed control per parameter. The declaration carries everything a form needs - type, enum, range, description - so this is frontend work rather than another contract.
• Duplicate/conflicting input across sources (a value given on the CLI and in a context file, say) is not detected as a conflict; last one in wins, as before. Detecting it needs the surfaces to pass provenance in, not just values.
• Backfill and partition parameters are not yet routed through this, since they build their own value sets. Worth doing with Feature: First-class partition definitions and durable, resumable backfill execution #295.

A pipeline that declares no parameters behaves exactly as #127 did, and there is a test pinning that too.

[LouisDeconinck]

This looks like the right core implementation, especially having one normalization boundary in apply_params.

For our use case I would keep this open for two remaining correctness pieces:

1. conflicting duplicate input should not silently remain last-write-wins

For example, if a schedule binds:

jurisdiction = BE

and an explicit run input supplies:

jurisdiction = NL

I am fine with having a documented precedence rule, but I would like the normalized result to retain enough provenance to distinguish an intentional override from accidentally supplying the same parameter through multiple sources.

At minimum the effective parameter metadata could expose something like:

name
value/redacted
source
overrode_source

2. partitions/backfills should go through this exact same normalization boundary

This is especially important for parameters such as:

jurisdiction
effective_date
partition_date
full_refresh

I would rather not have #295 create a parallel parameter model and later have normal runs and backfills disagree.

The typed desktop/web controls are valuable as well, but I see those more as UX work on top of the contract than as a reason to redesign the underlying parameter model.

[SouravRoy-ETL]

Your point 1 is shipped on main (1bf5f48). Point 2 is not, and I want to be straight about why.

Provenance, in the four fields you named:

{ "name": "jurisdiction", "value": "NL", "source": "run input", "overrode": ["schedule"] }

The substitution boundary now takes sources rather than an already-flattened map. That was the actual problem: by the time values reached apply_params they had been merged into one HashMap, so the second source had already left no trace and no amount of care downstream could recover it. Sources are passed lowest-authority first, later wins - documented rather than clever - and every effective parameter carries where it came from and the sources whose different value it replaced. It lands on the run receipt, so the question is answerable after the run rather than only while it happens.

Only a differing value counts as an override. Two surfaces binding a parameter to the same value is a duplicate and harmless. Recording both alike would bury the case that matters in the noise of the one that does not - and the case that matters is precisely the one you described. conflicts() returns just those.

A secret is *** in the provenance record, built from the same redacted map run history uses. A record of where a value came from must not become the one place the value is written down.

Nothing reports an override yet, and that is honest rather than broken. There is exactly one source on the console path today - the request body - and schedules carry no parameters at all, so the conflict you describe cannot currently occur. I built the mechanism first on purpose: the difference between a deliberate override and an accidental double binding has to be recoverable the first time it happens, not after someone notices it was never recorded. When schedule-bound parameters land, they pass a second Supplied { source: "schedule" } and the reporting is already there.

apply_params keeps its signature and behaviour, labelling its single map as one source, so every existing caller is unchanged.

On point 2 - partitions and backfills through this same boundary. I have not done it, and I do not think I should do it as a bolt-on here. duckle-runner backfill today means watermark editing, not partitioned execution; there is no partition object, no partition_date, no backfill run identity. #295 asks for exactly those, and the scoping I did on it says the honest slice is a partition generator on top of the durable slice machinery in batch.rs, with the partition key becoming the batch item key. The right shape is for that generator to emit Supplied { source: "partition" } values into this boundary - which is a small piece of #295 rather than something to invent here first. Doing it the other way round would mean guessing at the partition model and then reworking it.

So: this boundary is ready for it, and I would rather wire it as part of #295 than build a second notion of a partition parameter next to the real one.