Repository navigation
test(web-benchmark): re-record agent recordings for @e2e-dev/web 0.13 - #925
Merged
Merged
Conversation
#825 bumped @e2e-dev/web to 0.13.0. The cache key carries the engine minor, so every committed entry sat under an old hash and the strict web agent job failed 30 tests with REPLAY_STALE. Re-recorded live; old entries deleted.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 31 files
Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.
Turn on auto-fix | Re-trigger cubic
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.18.0 · 1m 53s · web · run artifacts |
Contributor
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 76.6k tokens (92% cached) · $0.0036 All 33 tests in 5 files
e2e 0.18.0 · 3m 4s · web · run artifacts |
…traces rich-text-editor bolded via Control+Shift+ArrowLeft then Shift+ArrowRight x8: on macOS the chord selects to line start, on Linux one word, so CI collapsed the selection and bolded nothing. New trace selects with Shift+ArrowLeft from the end, same result on both. playbook-cleanup dropped two redundant taps on an unrelated menu.
SudoScientist161
pushed a commit
to SudoScientist161/e2e
that referenced
this pull request
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
web agentjob fails on main and every open PR with 30REPLAY_STALE. chore: version packages #825 (chore: version packages) bumped@e2e-dev/webfrom 0.12.0 to 0.13.0. The engine reports its own package version (packages/web/src/engine.tsownVersion()), and the cache key keeps its major.minor (engineCompatibilityVersioninpackages/e2e/src/cache/identity.ts). So every committed entry sat under an old hash, and--strict-cacherefused each step whose recording was stored under another key. No cache code changed.tests-agent/live: 30 new entries, one per old entry (samerecordedForstep), and deleted the 30 old ones. Cache JSON only.Control+Shift+ArrowLeftthen 8xShift+ArrowRight. On macOS Chromium that chord selects to line start, so the presses leave "approved". On Linux it selects one word, so they collapse it and bold nothing. Re-recorded until the trace passes on Linux Chromium too: it now selects withShift+ArrowLeftfrom the end. No other entry uses a modifier chord.Verified
Ran it locally: yes
e2e run --config e2e.agent.config.ts --strict-cacheagainst the built packages, the same command CI runs.REPLAY_STALE)mcr.microsoft.com/playwright:v1.63.0-noble): old trace "Only the word approved must be bold", new trace "Document approved". Strict replay of that test 3x on macOS: 1 replayed each, plus a full strict suite: 33 passed, same counts as above.Recording run: 33 passed, 32 missed, 124 model calls, $0.028.
Follow-up
Same as #833: every minor release of an engine invalidates all committed recordings, and the release PR does not catch it. Separate change.
Checklist
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it..d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures.e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.