You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(cache): replay-only runs that fail a step with no recording instead of calling the model #938
We gate every push on a replay of our committed recordings. The gate must make zero model calls:
a step whose recording is missing or stale should fail, not quietly spend tokens. Today cache: { mode: 'read-only', strict: true } gets close but not all the way:
A step with no entry at all still runs live under strict (docs/cache.mdx, "a never-recorded
step runs live"); no-entry is not in STALE_REASONS (packages/e2e/src/agent/step-cache.ts).
failIfRekeyed only recognizes an entry that records its whole step (paramsDigest, callIndex, agent). Entries written before 0.17 carry only testId, targetId and instructionDigest, so after a key change (0.13's engine minor, or fix(cache): drop the engine version from the replay cache key #929) strict treats them as
never recorded and the step goes live. We hit this with 22 committed recordings on the 0.18
upgrade.
agent.assert, waitFor and extract always run live, and maxModelCalls cannot be 0.
So a replay-only gate today needs three workarounds, the same trick #929's Verified section used:
a model that throws in every agent slot, stripping AI_GATEWAY_API_KEY and VERCEL_OIDC_TOKEN
(the gateway falls back to an OIDC token from .vercel/project.json), and an audit of report.json that every agent.act is self-finalized with metrics.modelCalls === 0.
Proposal: cache.replayOnly: true (and --replay-only) that implies strict and read-only, fails
any agent step without a replayable entry with a typed code (e.g. REPLAY_MISSING), refuses assert / waitFor / extract at collection or at the call with the same code, and never
resolves a model. Happy to send the PR once the shape is agreed (it touches the CLI, the config
schema and an error code, so it seems to want a maintainer's yes first). Related: #923.
Activity