Skip to content

Compare evidence-window v2 in live-replay equivalence - #53

Merged
b1rdmania merged 1 commit into
b1rdmania:mainfrom
0xvigilant:fix/equivalence-window-v2
Oct 6, 2026
Merged

b1rdmania merged 1 commit into
b1rdmania:mainfrom
0xvigilant:fix/equivalence-window-v2

Conversation

@0xvigilant

Copy link
Copy Markdown
Contributor

What changed

groundnut-equivalence now accepts groundnut-evidence-window/v2 as well as v1. Built-in HTTP captures have emitted v2 since 0.2.0a3, so the projection reader refused every new HTML, PDF or JSON run with unsupported evidence-window schema.

The complete window object is still compared in both versions. A v2 producer-identity change (extractor, extractor library or runtime) between live and replay is therefore reported as a difference, not hidden. Any other window schema still fails as unsupported. The change is documented in docs/LIVE-REPLAY-EQUIVALENCE.md.

Evidence

Found while adding JSON sources (a separate PR stacked on this one). Three new tests in tests/test_equivalence.py drive a built-in HttpResolver with an injected opener for an HTML fixture and a text/plain fixture, then compare live against two replays. On main they fail with unsupported evidence-window schema; with this change they pass. A further test archives a replay snapshot whose v2 runtime differs and checks that it is reported as different.

Lane

  • IC product path
  • Canonical core
  • Active experiment
  • Legacy compatibility

Contract impact

  • No schema, policy, gate, metric, holdout, or public API changed.
  • Any intentional contract change is versioned and documented. The equivalence reader admits evidence-window v2, documented in LIVE-REPLAY-EQUIVALENCE.md. No compared field is excluded.
  • No quality claim is borrowed from a different domain or evaluation.
  • No metric or threshold was selected after viewing protected results.
  • Experimental behaviour is not described as a current product stage.

Determinism and disclosure

  • Deterministic tests do not access the network. They use an injected opener, and the socket block stays in place.
  • New model-backed behaviour is replaceable and benchmark-gated. None was added.
  • Outputs retain source, policy, component, and engine identity.
  • The diff contains no credentials, private source text, protected holdout data, or local filesystem paths.

Validation

python3.12 -m pytest -q   # 483 passed (main: 479; the 3 new v2 tests fail on main)

A review suggested one possible follow-up: share the accepted window-schema set between equivalence.py and EvidenceWindow.from_mapping, so the two cannot drift.

Built-in HTTP captures have emitted evidence-window v2 since 0.2.0a3, but the
projection reader still accepted only v1, so groundnut-equivalence refused any
new HTML, PDF or JSON run. Admit v1 and v2 and keep comparing the complete
window, including producer identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 6, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0939c752-1024-4d11-93ba-3336371b8def
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@0xvigilant 0xvigilant mentioned this pull request Oct 6, 2026
9 of 13 tasks

@b1rdmania b1rdmania left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete diff and tested the exact head 0abc27a. 483 tests pass locally and CI is green. The reader now accepts the v2 windows the current resolver emits while still comparing the full window object, including producer/runtime identity; unknown schemas remain rejected. No blocking findings.

@b1rdmania
b1rdmania merged commit f756bca into b1rdmania:main Oct 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants