Repository navigation
test(web-benchmark): re-record agent recordings for @e2e-dev/web 0.12 - #833
Merged
Merged
Conversation
#774 bumped @e2e-dev/web to 0.12.0. The cache key carries the engine minor, so every committed entry sat under an old hash and the strict web agent job failed 25 steps with REPLAY_STALE. Re-recorded live; old entries deleted.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No issues found across 31 files
Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.
Re-trigger cubic
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.17.0 · 1m 49s · web · run artifacts |
Contributor
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 76.1k tokens (92% cached) · $0.0034 All 33 tests in 5 files
e2e 0.17.0 · 2m 52s · web · run artifacts |
okwasniewski
added a commit
that referenced
this pull request
Oct 4, 2026
okwasniewski
added a commit
that referenced
this pull request
Oct 5, 2026
okwasniewski
added a commit
that referenced
this pull request
Oct 5, 2026
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
web agentjob fails on main and on every open PR with 25REPLAY_STALE. #774 (chore: version packages) bumped@e2e-dev/webfrom 0.11.2 to 0.12.0. The cache key carries the engine minor (packages/e2e/src/cache/identity.ts:185), so every committed entry sat under an old hash, and--strict-cacherefused each step whose recording was stored under another key. No cache code changed.tests-agent/live: 30 new entries.Steps not replayed are the same ones as before for the same reasons as #796: the cart totals and wishlist acts end in planted bugs (missed by design), shadow-dom-form and iframe-form miss at a gap, and lying-labels and async-states hand off at a gap.
pick Express delivery, left out in #796, now records and replays.Verified
Ran it locally: yes
e2e run --config e2e.agent.config.ts --strict-cacheagainst the built packages, the same command CI runs.REPLAY_STALE)Recording run: 33 passed, 32 missed, 142 model calls, $0.048.
Follow-up
Every minor release of an engine invalidates all committed recordings this way, and the release PR does not catch it. Separate change.