CI: add stable versioned JSON plus JUnit/SARIF output for validate, test, contract, policy, and review

[LouisDeconinck]

Problem

Duckle has strong CLI validation, tests, review, contracts, and policy findings.

CI systems and coding agents benefit when results use standard formats rather than requiring custom parsing of console text.

Proposed solution

For relevant commands, support:

--format json
--format junit
--format sarif

Commands:

duckle validate
duckle test
duckle contracts check
duckle policy check
duckle review

Stable JSON

Version the machine-readable schema:

{
  "schema_version": 1,
  "command": "validate",
  "findings": []
}

Document stable exit codes for:

success
validation/test failure
configuration/tool failure
policy refusal

JUnit

Represent tests/checks as test cases with:

pipeline
node/assertion
duration
failure message
captured safe diagnostics

SARIF

Use SARIF for file/node-level findings such as:

unknown property
invalid SQL
contract break
policy violation
unsafe component
schema drift

Map to pipeline/workspace file and node/property where possible.

GitHub annotations

SARIF is sufficient for GitHub Code Scanning; optional concise workflow annotations can be generated without embedding GitHub-specific behavior into the core engine.

Security

Generated reports must use the same secret/sensitive-value redaction as logs and API results.

Acceptance criteria

1. CI can display failed Duckle tests as native test results;
2. policy/validation findings point to the relevant file/node/property;
3. JSON schemas and exit codes are documented/versioned;
4. reports never expose secrets;
5. output remains consistent across platforms.

[SouravRoy-ETL]

Shipped for validate on main (d8958a2): --format json | junit | sarif.

Exit codes are documented and distinct, which I think matters more than the formats:

code	meaning
0	everything checked passed
1	a check failed - the thing being checked is wrong
2	the tool could not run - bad flag, unreadable file, no input

1 and 2 are deliberately separate. A job usually wants to fail differently on 2, because 2 means the gate never actually ran, and treating that as a pass is how a broken gate goes unnoticed for a month.

The stable JSON is versioned, and --json is unchanged rather than replaced: the envelope carries the old results array alongside the new findings, so an existing consumer reads ok/checked/failed/results and a new one reads schemaVersion/findings. Nobody has to migrate on a minor release.

The escaping is the part I would flag to anyone building this. It is hand-written on purpose. A SQL error message routinely contains < and &, and an unescaped JUnit file is one the CI system silently fails to parse - which renders as "no tests ran", the worst possible answer from a gate. Control characters are dropped rather than escaped, because XML 1.0 forbids them outright and a raw one makes the document unparseable. Both are tested, and the test parses the output with a real XML parser rather than matching substrings, because matching substrings is how you convince yourself malformed XML is fine.

SARIF reports only failures. A result for something that passed would annotate a file that is fine. URIs are forward-slashed so Code Scanning can match them to repository paths - a Windows separator never matches, which is a quiet way for a whole report to do nothing.

JUnit keeps the passes, since a report with two failures and no passes cannot be told from one where only two things ran.

Not yet: duckle test, contracts check, policy check and review. The report module is shared and takes a Finding { file, node, rule, message, ok }, so those are each a matter of building findings rather than another formatter - duckle test is the natural next one, since it already has per-case results and a node id to hang them on.

On redaction: compile_pipeline_sql already replaces secret values with named placeholders before the error text is produced, so validate findings inherit that. When duckle test and review are added, that needs re-checking per command rather than assumed.