Skip to content

feat(cache): replay-only runs that fail a step with no recording instead of calling the model #938

Open

Description

@outof-place

We gate every push on a replay of our committed recordings. The gate must make zero model calls:
a step whose recording is missing or stale should fail, not quietly spend tokens. Today
cache: { mode: 'read-only', strict: true } gets close but not all the way:

  • A step with no entry at all still runs live under strict (docs/cache.mdx, "a never-recorded
    step runs live"); no-entry is not in STALE_REASONS (packages/e2e/src/agent/step-cache.ts).
  • failIfRekeyed only recognizes an entry that records its whole step (paramsDigest,
    callIndex, agent). Entries written before 0.17 carry only testId, targetId and
    instructionDigest, so after a key change (0.13's engine minor, or fix(cache): drop the engine version from the replay cache key #929) strict treats them as
    never recorded and the step goes live. We hit this with 22 committed recordings on the 0.18
    upgrade.
  • agent.assert, waitFor and extract always run live, and maxModelCalls cannot be 0.

So a replay-only gate today needs three workarounds, the same trick #929's Verified section used:
a model that throws in every agent slot, stripping AI_GATEWAY_API_KEY and VERCEL_OIDC_TOKEN
(the gateway falls back to an OIDC token from .vercel/project.json), and an audit of
report.json that every agent.act is self-finalized with metrics.modelCalls === 0.

Proposal: cache.replayOnly: true (and --replay-only) that implies strict and read-only, fails
any agent step without a replayable entry with a typed code (e.g. REPLAY_MISSING), refuses
assert / waitFor / extract at collection or at the call with the same code, and never
resolves a model. Happy to send the PR once the shape is agreed (it touches the CLI, the config
schema and an error code, so it seems to want a maintainer's yes first). Related: #923.

Activity

  1. added
    P2Improvement / follow-up
    enhancementNew feature or request
    cachecache-1 locate/path caching
    on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Improvement / follow-upcachecache-1 locate/path cachingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions