[akram1089]

Closes #939.

What and why

• StepCacheInfo says how a step read the cache (mode, reason) but not what the attempt then did to the recording. A self-healing CI loop needs that to know which tests to run a second pass for, and today it has to diff .e2e/cache/ against git.
• Adds two optional fields, as proposed in the issue:
  • outcome: 'written' | 'kept' | 'evicted'
  • outcomeReason: kept is confirmed or unchanged; evicted is repaired, failed-after-replay, not-replaced or unconfirmed; absent on written.
• Additive: both fields are optional and only appear in read-write mode when the attempt changed or confirmed an entry. A read-only cache, an interrupted attempt, a step with no verdict, or nothing to record and nothing stored leaves outcome absent. No schema version bump.
• The report only claims what the store actually did: a custom store without delete, a rejected delete, or a failed write reports no outcome.

Each outcome comes from the place that already makes the decision:

flowchart LR
  S[step settles] --> C["StepTraceSession.conclude()"]
  C -- "end-mismatch repaired" --> E1["evicted: repaired"]
  C -- "failed after replay" --> E2["evicted: failed-after-replay"]
  C -- "passed, nothing to stage, entry read" --> E3["evicted: not-replaced"]
  C -- "staged write / keep" --> A[attempt ends]
  A --> F["flushStagedTraces()"]
  F -- "not verified, implicated" --> E4["evicted: unconfirmed"]
  F -- "keep, verified" --> K1["kept: confirmed"]
  F -- "write, same flow stored" --> K2["kept: unchanged"]
  F -- "write, verified" --> W[written]

Loading

• Step-time evictions: packages/e2e/src/agent/step-cache.ts, evict(reason) sets the outcome on the step's cache info once the delete has settled. conclude() runs before finish() attaches the details, so these reach both the report and agent.act()'s result.
• Attempt-end settlement: flushStagedTraces (packages/e2e/src/cache/context.ts) now returns { stepIndex, outcome, outcomeReason? }[], and packages/e2e/src/run/execute.ts merges them into record.steps[stepIndex].cache right after the flush. agent.act() returns before the attempt settles, so its result carries only the step-time evictions. The docs say so.
• Wire change: report-v1.schema.json gets both enums plus allOf rules (reason only with an outcome, no reason on written, each reason only with its own outcome). Both fixtures and a schema test pin those rules.
• Docs: a new "What a run did to a recording" table in docs/cache.mdx, plus docs/reference/agent.mdx and skills/e2e/references/agent.md.
• One existing unit test, evicts an entry that did not serve a pass with nothing to record in its place, recorded the replayed tap after begin(), so it was going down the repair branch instead. Its host now records the tap through the grammar, as the real dispatch does, and the test asserts not-replaced.

Verified

Ran it locally: yes

• A real e2e run against the browser fixture app with a scripted two-tap executor and cache: 'read-write'. Run 1 records, run 2 replays, and run 3 tightens the assertion so it fails. This is step.cache from each run's report.json:

run 1, record:          {"mode":"missed","reason":"no-entry","replayedActions":0,"totalActions":0,"outcome":"written"}
run 2, replay:          {"mode":"self-finalized","replayedActions":2,"totalActions":2,"outcome":"kept","outcomeReason":"confirmed"}
run 3, assertion fails: {"mode":"self-finalized","replayedActions":2,"totalActions":2,"outcome":"evicted","outcomeReason":"unconfirmed"}

• The integration suites trace-cache-replay and agent-trace-cache now assert the outcome end to end: kept/confirmed on whole replays, written on re-recordings, evicted/unconfirmed when a later assertion fails, and no outcome when the failure is a model that never answered or the cache is read-only in CI.
• Done on Windows. The full e2e package run has failures in 26 files on Windows (CRLF line endings, \ paths, POSIX file modes, git/telemetry environment). All 26 fail the same way on main, and none of them touch the cache or the report. pnpm docs:check was not run locally, because it needs the Mintlify docs workspace installed.

Checklist

• Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
• Breaking change: not one. Both fields are optional additions.
• Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
• Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures. (sdk-types.ts has no StepCacheInfo assertions, so nothing changed there.)
• Engine contract change (e2e/engine): not touched.

[cubic-dev-ai]

1 issue found across 16 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/e2e/src/run/steps.ts">

<violation number="1" location="packages/e2e/src/run/steps.ts:145">
P2: This exported type accepts outcome/reason combinations that the report schema rejects, and discriminating on `outcome` does not narrow `outcomeReason`. Model these fields as a discriminated union so SDK values match the schema.</violation>
</file>

<sub>Reply to a comment to ask cubic a question or push back. It learns from your replies.

Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This exported type accepts outcome/reason combinations that the report schema rejects, and discriminating on outcome does not narrow outcomeReason. Model these fields as a discriminated union so SDK values match the schema.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/e2e/src/run/steps.ts, line 145:

<comment>This exported type accepts outcome/reason combinations that the report schema rejects, and discriminating on `outcome` does not narrow `outcomeReason`. Model these fields as a discriminated union so SDK values match the schema.</comment>

<file context>
@@ -121,6 +121,28 @@ export interface StepCacheInfo {
+   * nothing a replay could check to record in the entry's place),
+   * `unconfirmed` (no verification passed after the step).
+   */
+  outcomeReason?: 'confirmed' | 'unchanged' | 'repaired' | 'failed-after-replay' | 'not-replaced' | 'unconfirmed';
   replayedActions: number;
   totalActions: number;
</file context>

[akram1089]

Leaving this as is for now, and leaving the thread open for a maintainer's call. Turning StepCacheInfo from an interface into a union would change a public type's shape: an interface X extends StepCacheInfo in user code stops compiling, which is a bigger contract change than this additive PR. The existing pairs on the same type are just as loose in TypeScript and are enforced only by the report schema: reason is optional on every mode, and derived is legal only with reason: 'gap'. outcome/outcomeReason follow that convention, and report-v1.schema.json rejects every invalid combination (pinned in schema-fixtures.test.ts). Happy to switch to a discriminated union if the maintainers prefer it.

[cubic-dev-ai]

All reported issues were addressed across 5 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic</sub>

[okwasniewski]

Thanks. This landed in a different shape in #869 as step.cache.write, which reports the outcome of each step's cache write in the report. Closing as superseded.