Check artifact fingerprints and review a pinned replay pipeline

.github/workflows/ci.yml

@@ -39,3 +39,19 @@ jobs:
           "$RUNNER_TEMP/wheel-env/bin/evidence-gate" init-example wheel-example
           "$RUNNER_TEMP/wheel-env/bin/evidence-gate" validate --spec wheel-example/evidence-gate.yaml --run wheel-example/runs/demo-run --json-out status.json
           "$RUNNER_TEMP/wheel-env/bin/evidence-gate" packet --status status.json --md-out review-packet.md
+
+  replay-pipeline:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+      - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
+      - name: Verify the pinned companion tools and failure paths
+        working-directory: examples/replay-pipeline
+        run: |
+          uv sync --locked --group dev
+          uv run --locked pytest -q test_pipeline.py
+          uv run --locked python run.py --output .runs/ci --demonstrate-failure

.gitignore

@@ -6,5 +6,6 @@ __pycache__/
 dist/
 build/
 *.egg-info/
+examples/replay-pipeline/.runs/
 reports/evidence-status.json
 reports/review-packet.md

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 0.3.0 — 2026-10-02
+
+- Add a reproducible input-contract → replay-comparison → evidence-review example
+  with companion repositories pinned to immutable commits.
+- Exercise incomplete exports, duplicate input events, missing future markouts,
+  and refusal to overwrite earlier runs in a separate integration CI job.
+- Add optional output `sha256_field` checks, using chunked file hashing. Missing or
+  malformed expected digests, mismatches, and read errors fail validation.
+- Bind all four replay exports to their producer-recorded hashes, with regressions
+  for truncated CSVs and empty/tampered JSON, Markdown, and input-contract reports.
+- Keep the standalone checker's runtime dependencies unchanged.
+
+Compatibility: contract schema remains `1`. Outputs without `sha256_field` retain
+their existing presence/header behavior. Absent optional outputs still warn; when
+present, a configured digest must validate. An explicitly configured field must
+be a non-empty dotted path, and its report value must be 64 hexadecimal characters.
+Older package versions reject the new contract key; use 0.3+ for digest contracts.
+Matching bytes trust the producer's report and do not authenticate source data or
+validate artifact content/schema. CSV checks continue to inspect headers, not rows.
+
 ## 0.2.0
 
 ### Contract and status validation

README.md

@@ -24,11 +24,28 @@ After export:  exit 0; approved (10 checks)
 
 The example data and metrics are synthetic; the demonstration is of the gate's behavior.
 
+## Review a complete replay pipeline
+
+The [replay review example](examples/replay-pipeline/README.md) connects three tools:
+input ordering and identity checks, execution-assumption comparisons, and this evidence gate.
+It runs three profiles over two instruments with multiple book levels, produces JSON/CSV/Markdown
+artifacts, then demonstrates a missing export that blocks review despite passing metrics.
+The example pins its companion tools and checks the actual emitted reports.
+
+```bash
+cd examples/replay-pipeline
+uv run --locked python run.py --output .runs/complete --demonstrate-failure
+```
+
+Separate failure cases show duplicate inputs stopping before simulation and missing future
+evidence preventing promotion. The prices and decisions are synthetic; this is a pipeline
+verification example, not a strategy-performance estimate.
+
 ## Use it in your pipeline
 
 ```bash
-# Install from this checkout; uv users can keep using uv run instead.
-python -m pip install .
+# Install the versioned CLI. From a checkout, uv run or pip install . also works.
+uv tool install 'git+https://github.com/matchalatteguy/evidence-gate.git@v0.3.0'
 evidence-gate init-example scratch/toy-run
 
 evidence-gate validate \
@@ -71,6 +88,19 @@ reports:
 
 The report is a JSON object containing the named `status`, `summary`, `counts`, `metrics`, and `outputs` fields. Output paths are relative to the run root. Contracts reject empty report lists, unknown fields, and contradictory thresholds; paths that escape the run root, including escaping symlinks, fail validation.
 
+## Detect changed or truncated exports
+
+Optionally bind an output to the SHA-256 digest recorded in its producer's report:
+
+```yaml
+outputs:
+  predictions:
+    path_field: outputs.predictions
+    sha256_field: sha256.predictions
+```
+
+After finishing the export, the producer writes its 64-hex digest at `sha256.predictions` in the report. Validation hashes the artifact in chunks and fails for missing/malformed digests, changed bytes, or read errors. The [replay example](examples/replay-pipeline/README.md) applies this to all four exports, so a header-only CSV or emptied report cannot retain approval. [Digest contract details](docs/contract-schema.md#output-byte-integrity).
+
 ## Python API
 
 ```python
@@ -84,9 +114,9 @@ for failure in result.failures:
 
 ## What the decision means
 
-`approved` means the configured file and value checks passed. It does not verify that a report is truthful, that an experiment is sound, or that a model is useful. CSV checks inspect headers, not rows. Contracts and files should come from trusted local pipelines; containment checks are not a sandbox for hostile processes changing files during validation.
+`approved` means the configured file and value checks passed. It does not verify that a report is truthful, that an experiment is sound, or that a model is useful. CSV checks inspect headers, not rows. A matching digest establishes byte integrity against the trusted producer's report; it does not authenticate sources or validate artifact content/schema. Contracts and files should come from trusted local pipelines; containment checks are not a sandbox for hostile processes changing files during validation.
 
-The tool supports JSON reports, finite numeric thresholds, local file artifacts, and CSV headers. It has no scheduler, hosted state, network calls, or plugin execution. Version 0.2 tightens contract and status validation; see the [migration notes](CHANGELOG.md).
+The tool supports JSON reports, finite numeric thresholds, local file artifacts, optional SHA-256 integrity checks, and CSV headers. It has no scheduler, hosted state, network calls, or plugin execution. Version 0.3 adds opt-in artifact digests while preserving unconfigured contracts; see the [migration notes](CHANGELOG.md).
 
 ## Documentation and development
 

docs/contract-schema.md

@@ -55,7 +55,7 @@ The report list must be non-empty and report names must be unique. Unknown or du
 
 ## Dot paths
 
-`required_fields` and output `path_field` values use simple dot paths through JSON objects:
+`required_fields`, output `path_field`, and output `sha256_field` values use simple dot paths through JSON objects:
 
 ```yaml
 required_fields:
@@ -110,6 +110,35 @@ outputs:
 
 The referenced report value must be a relative path string under the run root. Required missing artifacts fail; optional missing artifacts warn. Each output spec must be a mapping with a `path_field`; malformed output specs raise a clear `ValueError` during spec loading. When `csv_columns` is set, Evidence Gate reads only the CSV header and checks that every listed column is present. Missing columns fail with `table.column_missing`; unreadable or non-UTF-8 CSV files fail with `table.invalid_csv` instead of crashing validation. The old `columns` spelling is accepted as a backward-compatible alias, but new contracts should use `csv_columns` so the CSV-specific behavior is explicit.
 
+### Output byte integrity
+
+Set `sha256_field` to the dotted field containing the producer's expected file digest:
+
+```yaml
+outputs:
+  predictions:
+    path_field: outputs.predictions
+    sha256_field: sha256.predictions
+    csv_columns: [id, label, score]
+```
+
+The field name must be a non-empty string with non-empty dotted components and no surrounding component whitespace. The expected report value must contain exactly 64 hexadecimal characters; uppercase and lowercase are accepted. A producer can record the digest after completing its output:
+
+```python
+import hashlib
+
+# report already contains outputs.predictions; run_root is the run directory.
+with (run_root / report["outputs"]["predictions"]).open("rb") as source:
+    report["sha256"] = {
+        "predictions": hashlib.file_digest(source, "sha256").hexdigest()
+    }
+# Write this report after the export is closed and the digest is recorded.
+```
+
+Validation reads the artifact in 1 MiB chunks. A matching digest adds `artifact.sha256_match`. Missing fields, malformed digests, changed bytes, and read errors fail with `artifact.sha256_missing`, `artifact.sha256_invalid`, `artifact.sha256_mismatch`, and `artifact.sha256_read_error`, respectively.
+
+Without this option, existing presence/header checks are unchanged. A missing optional artifact still warns without requiring a digest; a present optional artifact must satisfy its configured digest. The hash binds bytes to a trusted producer report. It does not authenticate that report or its sources, establish semantic correctness, or validate the artifact's content/schema. Preserve exports and their producer report together, and stop writers before validating.
+
 ## Path policy
 
 Paths must be non-empty strings. Output artifacts must be files; a directory fails with `artifact.not_file` even when no CSV columns are requested. Unreadable or non-UTF-8 reports fail with `report.invalid_json`.

docs/python-api.md

@@ -6,6 +6,7 @@ Evidence Gate exposes a small API for scripts, notebooks, and LLM agents that ne
 
 - `EvidenceSpec`: normalized contract containing report specifications and `schema_version`.
 - `ReportSpec`: expected JSON report, required fields, thresholds, and output artifacts.
+- `OutputSpec`: artifact path field, optional CSV columns, and optional producer digest field.
 - `RunBundle`: safe wrapper around a run root directory.
 - `ValidationResult`: pass/fail result, recommendation, counts, checks, warnings, and failures.
 - `Check`: one machine-readable check with a stable code and severity.
@@ -53,6 +54,8 @@ Common failure codes include:
 - `metric.invalid`, `metric.below_min`, `metric.above_max`
 - `numeric.invalid`, `numeric.below_min`, `numeric.above_max`
 - `artifact.missing`
+- `artifact.sha256_missing`, `artifact.sha256_invalid`
+- `artifact.sha256_mismatch`, `artifact.sha256_read_error`
 - `path.absolute`, `path.escape`
 - `table.column_missing`
 

examples/replay-pipeline/.gitignore

@@ -0,0 +1,4 @@
+.venv/
+.runs/
+__pycache__/
+.pytest_cache/

examples/replay-pipeline/README.md

@@ -0,0 +1,111 @@
+# Review an execution-assumption pipeline
+
+Check that a replay run has valid input ordering, internally consistent comparisons,
+and all its promised output files before handing it to a reviewer.
+
+This example connects:
+
+1. **Replay Contract Kit**: check event identities and ordering before simulation.
+2. **Replay Realism Kit**: compare named fee, latency, and queue assumptions.
+3. **Evidence Gate**: require the resulting report bundle and complete markout evidence.
+
+The thirteen input events describe two invented instruments and multiple price levels.
+Three profiles change the simulated fills and costs. All amounts use the same declared
+synthetic quote unit; no exchange, external data service, credentials, or live orders are used.
+Replay `timestamp` values use milliseconds. The auxiliary ISO-8601 `observed_at` field
+expresses those same offsets from a fixed UTC epoch for the dataset contract's timestamp parser.
+
+## Run it
+
+Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/). From the repository root:
+
+```bash
+cd examples/replay-pipeline
+uv run --locked python run.py --output .runs/complete --demonstrate-failure
+```
+
+The example's lock file pins the two companion repositories to immutable commits.
+The first installation downloads those public sources and dependencies. The pipeline
+itself subsequently reads local files and makes no network calls.
+
+Expected result:
+
+```text
+Input contract: passed (13 events)
+Scenario comparison: 3 profiles; 0 gate failures
+Missing export: needs_work; artifact.missing
+Complete bundle: approved
+Review packet: <output directory>/review.md
+```
+
+The missing-export step removes a generated CSV, verifies failure, restores that CSV,
+and verifies the same contract again. Each invocation requires a new output directory.
+It refuses to overwrite existing results.
+
+## Inspect the evidence
+
+| Output under the run directory | Purpose |
+| --- | --- |
+| `input/` | Exact local event and scenario inputs used for this run |
+| `reports/input-contract.json` | Input validation results and stable failure codes |
+| `artifacts/comparison.json` | Profiles, quantities, costs, fill traces, and gate results |
+| `artifacts/comparison.csv` | Scenario summary for spreadsheet inspection |
+| `artifacts/comparison.md` | Human-readable comparison and model limits |
+| `reports/pipeline.json` | Small consumer envelope connecting metrics to output paths |
+| `reports/incomplete-status.json` | The deliberate missing-export failure |
+| `reports/evidence-status.json` | Final evidence decision |
+| `review.md` | Review packet listing each configured check |
+
+The envelope's values are read from the comparison's emitted reports. They are not
+hardcoded success labels. It records SHA-256 digests of all four completed exports.
+The contract in `evidence.yaml` requires at least three scenarios, zero replay-gate
+failures, complete markout coverage among filled decisions, all four declared files,
+and matching export bytes. A truncated CSV with its header intact or an emptied
+JSON/Markdown/input-contract report fails with `artifact.sha256_mismatch`.
+
+## Deliberately fail each stage
+
+```bash
+# Exit 1: duplicate event identity and sequence; replay never runs.
+uv run --locked python run.py --output .runs/duplicate --case duplicate-event
+
+# Exit 1: a complete metrics report cannot compensate for an unfinished CSV export.
+uv run --locked python run.py --output .runs/missing-export --case missing-export
+
+# Exit 1: input ordering passes, but filled ALPHA decisions lack future markout evidence.
+uv run --locked python run.py --output .runs/missing-markout --case missing-markout
+```
+
+Output bundles from failed runs remain available for inspection. Exit `2` means that
+the command could not process its inputs or create its output directory.
+
+## Adapt the workflow
+
+- Map your event fields in `manifest.json`; keep input validation before simulation.
+- Describe explicit assumptions in `scenarios.yaml`. The `reference` profile is a
+  comparison baseline, not a selected profitable strategy.
+- Change evidence requirements in `evidence.yaml` to match the outputs you promise.
+- Preserve both failed and completed run directories so a reviewer can reproduce them.
+
+This consumer is an example, not a new orchestration framework. Its Python subprocess
+executes the installed replay CLI with explicit arguments and inspects the actual files.
+
+## Verify the example
+
+```bash
+uv sync --locked --group dev
+uv run --locked pytest -q test_pipeline.py
+```
+
+Tests exercise the complete run, missing export, duplicate inputs, missing future evidence,
+truncated/tampered exports, and refusal to overwrite earlier results. The repository's CI runs this example separately
+from the standalone evidence checker, preserving its small dependency surface.
+
+## What approval establishes
+
+Approval establishes that this fixture's configured input and output checks passed.
+Matching hashes establish byte integrity against the producer's envelope; they trust
+that envelope and do not authenticate sources or validate artifact content/schema.
+It does not authenticate source prices, establish statistical adequacy, validate venue
+matching behavior, or demonstrate trading profitability. Execution decisions remain
+independent, and the synthetic scenarios do not describe observed market conditions.

examples/replay-pipeline/events.jsonl

@@ -0,0 +1,13 @@
+{"event_type":"book","timestamp":0,"sequence":1,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","bids":[["99.90","2"],["99.80","4"]],"asks":[["100.10","1"],["100.20","3"]],"observed_at":"2026-01-01T00:00:00.000Z"}
+{"event_type":"book","timestamp":0,"sequence":2,"instrument_id":"BETA","venue_id":"fixture","source":"synthetic","bids":[["49.90","1"],["49.80","4"]],"asks":[["50.10","2"],["50.20","4"]],"observed_at":"2026-01-01T00:00:00.000Z"}
+{"event_type":"decision","timestamp":10,"sequence":3,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","side":"buy","size":"2","limit_price":"100.20","order_type":"taker","observed_at":"2026-01-01T00:00:00.010Z"}
+{"event_type":"decision","timestamp":10,"sequence":4,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","side":"buy","size":"1","limit_price":"99.90","order_type":"maker","observed_at":"2026-01-01T00:00:00.010Z"}
+{"event_type":"decision","timestamp":12,"sequence":5,"instrument_id":"BETA","venue_id":"fixture","source":"synthetic","side":"sell","size":"2","limit_price":"49.80","order_type":"taker","observed_at":"2026-01-01T00:00:00.012Z"}
+{"event_type":"trade","timestamp":30,"sequence":6,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","side":"sell","price":"99.90","size":"1","observed_at":"2026-01-01T00:00:00.030Z"}
+{"event_type":"trade","timestamp":60,"sequence":7,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","side":"sell","price":"99.90","size":"2","observed_at":"2026-01-01T00:00:00.060Z"}
+{"event_type":"book","timestamp":170,"sequence":8,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","bids":[["100.50","2"],["100.40","4"]],"asks":[["100.70","2"],["100.80","4"]],"observed_at":"2026-01-01T00:00:00.170Z"}
+{"event_type":"book","timestamp":180,"sequence":9,"instrument_id":"BETA","venue_id":"fixture","source":"synthetic","bids":[["49.10","2"],["49.00","4"]],"asks":[["49.30","2"],["49.40","4"]],"observed_at":"2026-01-01T00:00:00.180Z"}
+{"event_type":"book","timestamp":250,"sequence":10,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","bids":[["100.40","2"],["100.30","4"]],"asks":[["100.60","2"],["100.70","4"]],"observed_at":"2026-01-01T00:00:00.250Z"}
+{"event_type":"book","timestamp":250,"sequence":11,"instrument_id":"BETA","venue_id":"fixture","source":"synthetic","bids":[["49.00","2"],["48.90","4"]],"asks":[["49.20","2"],["49.30","4"]],"observed_at":"2026-01-01T00:00:00.250Z"}
+{"event_type":"book","timestamp":500,"sequence":12,"instrument_id":"ALPHA","venue_id":"fixture","source":"synthetic","bids":[["100.40","2"],["100.30","4"]],"asks":[["100.60","2"],["100.70","4"]],"observed_at":"2026-01-01T00:00:00.500Z"}
+{"event_type":"book","timestamp":510,"sequence":13,"instrument_id":"BETA","venue_id":"fixture","source":"synthetic","bids":[["49.00","2"],["48.90","4"]],"asks":[["49.20","2"],["49.30","4"]],"observed_at":"2026-01-01T00:00:00.510Z"}

examples/replay-pipeline/evidence.yaml

@@ -0,0 +1,27 @@
+schema_version: 1
+reports:
+  - name: replay-review
+    path: reports/pipeline.json
+    expected_status: passed
+    required_fields: [comparison_schema, counts, metrics, outputs]
+    counts:
+      input_events: {min: 1}
+      scenarios: {min: 3}
+      gate_failures: {max: 0}
+      missing_markouts: {min: 0, max: 0}
+    metrics:
+      minimum_markout_coverage: {min: 1, max: 1}
+    outputs:
+      input_contract:
+        path_field: outputs.input_contract
+        sha256_field: sha256.input_contract
+      comparison_json:
+        path_field: outputs.comparison_json
+        sha256_field: sha256.comparison_json
+      comparison_csv:
+        path_field: outputs.comparison_csv
+        sha256_field: sha256.comparison_csv
+        csv_columns: [name, decision_count, filled_count, markout_count, markout_coverage, fee_total, edge_after_fees, gate_failure_count]
+      comparison_review:
+        path_field: outputs.comparison_review
+        sha256_field: sha256.comparison_review

examples/replay-pipeline/manifest.json

@@ -0,0 +1,9 @@
+{
+  "schema_version": "1.0",
+  "dataset_id": "synthetic-execution-review",
+  "event_file": "events.jsonl",
+  "event_format": "jsonl",
+  "event_time_field": "observed_at",
+  "sequence_field": "sequence",
+  "entity_keys": ["instrument_id", "venue_id"]
+}