Repository navigation
fix(cache): drop the engine version from the replay cache key - #929
Merged
Merged
Conversation
Every engine minor re-keyed all committed recordings, so each `chore: version packages` merge failed the strict web agent job with REPLAY_STALE on main (#825 after #774). The key keeps the engine's name and SPI version; a release of the same engine is caught at replay, where each action re-finds its node. The 30 web benchmark entries move to their new hashes by a re-key verified against the old ones.
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.18.0 · 1m 55s · web · run artifacts |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
4 issues found across 39 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/e2e/tests/unit/trace-identity.test.ts">
<violation number="1" location="packages/e2e/tests/unit/trace-identity.test.ts:66">
P2: This compares `target` with an identical clone, so it no longer guards the PR’s key behavior. Compare otherwise-identical targets carrying different `engineVersion` values and require equal hashes.</violation>
</file>
<file name=".changeset/cache-key-engine-version.md">
<violation number="1" location=".changeset/cache-key-engine-version.md:5">
P2: `--strict-cache` identifies old-key entries only when the built-in file store can list them; custom `cache.store` runs cannot report them as `REPLAY_STALE`. Qualify this as file-store behavior.</violation>
</file>
<file name="docs/writing-an-engine.mdx">
<violation number="1" location="docs/writing-an-engine.mdx:100">
P2: This overstates how replay works: free actions and bare point actions replay without re-finding nodes. Qualify the rationale to node-targeted actions so engine authors do not assume every cached action is re-resolved.</violation>
</file>
<file name="docs/cache.mdx">
<violation number="1" location="docs/cache.mdx:304">
P2: A switch to another engine also makes `no-entry` fail with `REPLAY_STALE` when the store holds the same step under its old key. Add engine switches to these documented causes.</violation>
</file>
Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.
Turn on auto-fix | Re-trigger cubic
| buildTraceCacheKey({ ...parts, target: { ...target, engineVersion: '1.61.9' } }), | ||
| ), | ||
| ).toBe(hash); | ||
| expect(traceCacheKeyHash(buildTraceCacheKey({ ...parts, target: { ...target } }))).toBe(hash); |
Contributor
There was a problem hiding this comment.
P2: This compares target with an identical clone, so it no longer guards the PR’s key behavior. Compare otherwise-identical targets carrying different engineVersion values and require equal hashes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/e2e/tests/unit/trace-identity.test.ts, line 66:
<comment>This compares `target` with an identical clone, so it no longer guards the PR’s key behavior. Compare otherwise-identical targets carrying different `engineVersion` values and require equal hashes.</comment>
<file context>
@@ -71,14 +60,10 @@ describe('traceCacheKeyHash', () => {
- buildTraceCacheKey({ ...parts, target: { ...target, engineVersion: '1.61.9' } }),
- ),
- ).toBe(hash);
+ expect(traceCacheKeyHash(buildTraceCacheKey({ ...parts, target: { ...target } }))).toBe(hash);
});
</file context>
| "e2e": patch | ||
| --- | ||
|
|
||
| The replay cache key no longer carries the engine's version, so upgrading `@e2e-dev/web` or `@e2e-dev/mobile` keeps every committed recording. A recording re-finds its nodes at replay, which is where a node an engine release resolves differently is caught. Entries recorded before this release sit under the old keys: a `read-write` run records them again, and `--strict-cache` names them as `REPLAY_STALE` until it does. |
Contributor
There was a problem hiding this comment.
P2: --strict-cache identifies old-key entries only when the built-in file store can list them; custom cache.store runs cannot report them as REPLAY_STALE. Qualify this as file-store behavior.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .changeset/cache-key-engine-version.md, line 5:
<comment>`--strict-cache` identifies old-key entries only when the built-in file store can list them; custom `cache.store` runs cannot report them as `REPLAY_STALE`. Qualify this as file-store behavior.</comment>
<file context>
@@ -0,0 +1,5 @@
+"e2e": patch
+---
+
+The replay cache key no longer carries the engine's version, so upgrading `@e2e-dev/web` or `@e2e-dev/mobile` keeps every committed recording. A recording re-finds its nodes at replay, which is where a node an engine release resolves differently is caught. Entries recorded before this release sit under the old keys: a `read-write` run records them again, and `--strict-cache` names them as `REPLAY_STALE` until it does.
</file context>
| supported action kinds. `locate` answers `screen` queries; the exported | ||
| `version` is required: reports name it, and a saved session restores only | ||
| under the version that saved it. The replay cache does not key on it, since | ||
| a recording re-finds its nodes at replay. With `perform`, also declare |
Contributor
There was a problem hiding this comment.
P2: This overstates how replay works: free actions and bare point actions replay without re-finding nodes. Qualify the rationale to node-targeted actions so engine authors do not assume every cached action is re-resolved.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/writing-an-engine.mdx, line 100:
<comment>This overstates how replay works: free actions and bare point actions replay without re-finding nodes. Qualify the rationale to node-targeted actions so engine authors do not assume every cached action is re-resolved.</comment>
<file context>
@@ -95,9 +95,10 @@ export function hyperdrive(options: { device: string }): EngineHandle {
-supported action kinds. `locate` answers `screen` queries; the exported
+`version` is required: reports name it, and a saved session restores only
+under the version that saved it. The replay cache does not key on it, since
+a recording re-finds its nodes at replay. With `perform`, also declare
+`actions` to list the supported action kinds. `locate` answers `screen` queries; the exported
`resolveExpression` applies the standard query rules to a semantic tree.
</file context>
Suggested change
| a recording re-finds its nodes at replay. With `perform`, also declare | |
| node-targeted actions re-find their nodes at replay. With `perform`, also declare |
| same step under another key, from before an `e2e` or engine upgrade or a | ||
| change to the project id, the app's identity, or the agent's context. The | ||
| message names that file; delete it once nothing replays it. A custom | ||
| same step under another key, from before an `e2e` upgrade that re-keyed |
Contributor
There was a problem hiding this comment.
P2: A switch to another engine also makes no-entry fail with REPLAY_STALE when the store holds the same step under its old key. Add engine switches to these documented causes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/cache.mdx, line 304:
<comment>A switch to another engine also makes `no-entry` fail with `REPLAY_STALE` when the store holds the same step under its old key. Add engine switches to these documented causes.</comment>
<file context>
@@ -299,9 +301,9 @@ The reasons that fail are the ones that mean the app or the entry changed:
-same step under another key, from before an `e2e` or engine upgrade or a
-change to the project id, the app's identity, or the agent's context. The
-message names that file; delete it once nothing replays it. A custom
+same step under another key, from before an `e2e` upgrade that re-keyed
+entries or a change to the project id, the app's identity, or the agent's
+context. The message names that file; delete it once nothing replays it. A custom
</file context>
Contributor
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 76.6k tokens (92% cached) · $0.0036 All 33 tests in 5 files
e2e 0.18.0 · 2m 58s · web · run artifacts |
Merged
Contributor
🟢 e2e android: 54 passed, 1 skippedAll 55 tests in 7 files
e2e 0.18.0 · 8m 16s · android-emulator · run artifacts |
Contributor
🟢 e2e android agent: 20 passed24 agent steps · 123 model calls · 694.3k tokens (84% cached) · $0.0439 All 20 tests in 1 file
e2e 0.18.0 · 10m 29s · android-emulator · run artifacts |
Contributor
🟢 e2e ios: 52 passed, 1 skippedAll 53 tests in 7 files
e2e 0.18.0 · 8m 4s · ios-simulator · run artifacts |
Contributor
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 136 model calls · 821.5k tokens (81% cached) · $0.0563 All 22 tests in 1 file
e2e 0.18.0 · 12m 30s · ios-simulator · run artifacts |
okwasniewski
added a commit
that referenced
this pull request
Oct 9, 2026
… drop orphan web entries The 39 mobile recordings dated from Sep 23-24, before the Oct 3 replay policy bump (#732) and the key change in #929, so every CI run missed all of them and called the model. Both platforms are re-recorded on a Mac (iPhone 17 Pro on iOS 26.5, Pixel 7 API 36). The Android agentic job runs with --strict-cache like the web one; iOS stays lenient until the three recordings named in the README replay on a simulator. The web benchmark cache kept 30 entries under the pre-#929 key next to their re-keyed twins; the orphans go.
SudoScientist161
pushed a commit
to SudoScientist161/e2e
that referenced
this pull request
Oct 9, 2026
…-army#929) Every engine minor re-keyed all committed recordings, so each `chore: version packages` merge failed the strict web agent job with REPLAY_STALE on main (tester-army#825 after tester-army#774). The key keeps the engine's name and SPI version; a release of the same engine is caught at replay, where each action re-finds its node. The 30 web benchmark entries move to their new hashes by a re-key verified against the old ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
chore: version packagesmerge fails the strictweb agentjob on main withREPLAY_STALEon all 25 replayable steps (chore: version packages #825 now, chore: version packages #774 before it, re-recorded in test(web-benchmark): re-record agent recordings for @e2e-dev/web 0.12 #833). The cache key carried the engine'smajor.minor, and a 0.x engine bumps its minor on every release.relocate.ts). Keying on the version only cold-started recordings for engines that had not changed how a node resolves.recordedForand the agent config hashed to the committed file name for all 30, then the same parts hashed without the engine version. Entry contents are untouched (gitshows 30 renames).conservative/5and the pre-fix(cache): stop replays self-finalizing when the effect did not happen #732 key shape, so they have been stale since fix(cache): stop replays self-finalizing when the effect did not happen #732 and need a real re-record on a Mac, not a re-key (mobile CI does not run strict, so it spends model calls instead of failing).cache.mdx"What must match",writing-an-engine.mdx, the engine reference, and the AGENTS.md note on key changes.Verified
Ran it locally: yes
pnpm --filter @e2e-dev/web-benchmark run test:agent --strict-cacheagainst the built packages with a dummy gateway key, so a step that reaches the model fails and a replayed one cannot be mistaken for a live run:REPLAY_STALEThe 26/2/4 split is the one #833 recorded. The 7 failures are all
MODEL_PROVIDER_FAILEDfrom the dummy key on the handed-off and never-recorded steps; the cache directory was left unchanged by the run.pnpm --filter e2e exec vitest run --project unit: 2459 passed.lint,typecheck,docs:checkpass.Checklist
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it..d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures.e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.