Skip to content

Add deterministic replay gate for agent tests#268

Open
ZackMitchell910 wants to merge 1 commit intoinngest:mainfrom
ZackMitchell910:runledger/replay-gate
Open

Add deterministic replay gate for agent tests#268
ZackMitchell910 wants to merge 1 commit intoinngest:mainfrom
ZackMitchell910:runledger/replay-gate

Conversation

@ZackMitchell910
Copy link

@ZackMitchell910 ZackMitchell910 commented Dec 19, 2025

Summary

  • add a replay-only RunLedger eval suite (suite/case/schema/cassette + stub agent)
  • add a baseline file for regression gating
  • add a GitHub Actions workflow using runledger/Runledger@v0.1
  • add a small README note + ignore runledger_out/

How to run locally

runledger run evals/runledger --mode replay --baseline baselines/runledger-demo.json

Notes

  • no external calls; replay-only cassette
  • feel free to remove the suite/workflow if it is not desired
  • GitHub Actions note: workflows from first-time contributors/forks may require a maintainer to click “Approve and run” before checks will execute.

@changeset-bot
Copy link

changeset-bot bot commented Dec 19, 2025

⚠️ No Changeset found

Latest commit: 03c5111

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant