[siddhant573]

Builds on #753 (the screenshot mode); review that first. Until it merges this PR shows its commits too.

What and why

• Every run writes one sealed .evidence pack, <output>/evidence/<runId>.evidence, in the open, framework-agnostic format of @testmuai/evidence-cli (Apache-2.0, on npm). The same format other test tools produce, so an e2e run opens in the same viewers and tooling, as one file.
• On by default; evidence: false, E2E_EVIDENCE=0, or --no-evidence turn it off. Evidence takes no screenshot of its own and never changes how a run executes: the pack holds the frames the run already takes (on-failure by default). screenshot: 'every-step' (or --screenshot every-step, from feat(run): screenshot after every step with screenshot: 'every-step' #753) gives every step its frame.
• A built-in reporter (packages/e2e/src/report/evidence/), so it runs after the run and can never change its status or exit code; a pack that fails to write or validate is a line in the summary.
• The format's own library seals and validates the pack (seal.ts is the only importer), so this PR does not re-implement its rules.

flowchart LR
  R[report-1 + artifacts] --> M[map.ts: planPack] --> W[write.ts: live dir] --> S[evidence-cli finalize: seal zip] --> V[evidence-cli validate L1]
  V --> Sum[run summary line]

Loading

How a run maps:

Pack	From the run
run.yaml	run id, window, project, runner version, platforms, commit, models; token, cost, artifact-size metrics
tests/t-<id16>/	one per selected result, named from the result id, never a title (a test on two targets is two folders)
result.yaml	verdict, attempts, flaky, tags, final attempt's steps (kind = api, label, duration, cache mode)
steps/<n>-<attempt>-<index>/	step.json and screenshot.png
failure.yaml	on the failing step: error, expected/actual (matcher details, or an agent.assert instruction vs its explanation), locator, URL, failure frame, screen text
logs/	steps.ndjson, agent.ndjson (turns), the engine's trace when its redaction allows

• Report format change (report-1, additive): each step records argument, what it was given beside its target (typed text, a key, an assertion's expected value). A secret is <secret:name>. Text typed into a secure field is <withheld>: the locator engine reads the resolved node once before typing and records the text only when that read says the field is not secure, so it fails closed. Schema, both fixtures, and debugging.mdx/security.mdx change together.
• Report format change (report-1, additive): each step also records target, where it acted in CSS pixels: the box of the node a locator action resolved to (read from the node the locator already fetched, so no extra engine call), the point of a positioned tap or tapAt, and for an agent step its last action's node.
• The pack's step.json carries the fields the evidence viewer reads (summary, kind, status, duration_ms, url on app.open and failing steps, coordinates and element_rect for the cursor and outline) with the report's step record under e2e. Titles read Fill getByLabel("Username") with "admin", Expect getByRole("alert") to have text "Welcome back", Act: agree to the terms (check checkbox "Agree to terms" …).
• Engine contract (additive, e2e/engine): EngineArtifacts.screenshot may return { path, viewport } in place of a path. The session normalizes it, so every existing caller still gets a path; the step frame records the viewport and the pixel scale (read from the PNG header) on step.viewport, which the pack writes as environment.resolution. @e2e-dev/web reports the page's CSS viewport, @e2e-dev/mobile the screen in points, so a step's box lands on a device screenshot taken at a higher density.
• CI: the testbed (spec), web benchmark and web agent (benchmark), and the four mobile jobs (mobile) upload their pack as its own artifact (evidence-*, 7 days, SHA-pinned upload-artifact) and write a viewer link to the job summary. The hosted viewer opens an Actions artifact URL directly (after a GitHub sign-in), so every CI run on this PR has a clickable pack. Both steps are continue-on-error, like the existing report uploads.
• Each test folder's definition (the file the format hashes as "what was tested") is a generated test.json: the test's id, title path, file, line, and tags from the already-redacted report. It is never the test's source, which can hold a secret value as a plain string (the case fix(e2e): redact plain-string secret values in titles, labels, and agent input #727 now guards; plain-string-secrets.test.ts passes with the pack on).
• Verdicts: the format splits failed (the app was wrong) from broken (the check could not decide). Only ASSERTION_FAILED is failed.
• Secrets: only artifacts recorded complete or not-required are copied, so no video, binary download, command.log, or console output goes in; agent.extract values stay out. contentsUnder (the helper every secret-scanning integration test uses) now opens .evidence packs and the zips inside them, so engine-secrets, agent-policy, and the new evidence.test.ts all assert the secret is in no pack entry.
• The pack directory keeps the latest run's pack only (as artifacts/ keeps the latest run's files); evidence.outDir is refused inside the cache directory or a directory a run clears. e2e explore writes none. e2e init and the repo .gitignore ignore .e2e/evidence/.
• @testmuai/evidence-cli is imported lazily in seal.ts, so commands other than run never load it.

Verified

Ran it locally: yes

What was tested

Surface	How	Result	Pack
Unit + integration	pnpm test on the branch rebased on main	2,991 passed (one scheduler test flaked once, green 3/3 alone)	-
Web, deterministic	testbed suite, built CLI	71 passed	valid L1
Web, outcomes	scratch suite: pass, failed, broken, flaky, secret fill	each verdict mapped as designed	valid L1
Web, AI steps live	testbed agent suite, --no-cache, OpenAI gpt-6-luna	24 of 24 passed	valid L1
Web, AI judgment failure live	planted agent.assert that must fail	expected = instruction, actual = model's explanation	valid L1
Web, AI steps replayed	web-benchmark agent suite from committed recordings, no key	25 passed, 7 broken (no model)	valid L1
iOS	mobile-benchmark deterministic suite, iPhone 17 Pro simulator	53 to 54 of 55 (see below)	valid L1
CI path	fork run on GitHub-hosted runners: artifact upload + viewer link	2 artifacts, both open in the viewer	valid L1

Open the evidence (GitHub Actions artifacts from a run on GitHub-hosted runners; the viewer asks for a GitHub sign-in to read an artifact, kept 30 days):

• Testbed suite, 71 passed, tests named by title

• Web benchmark agent suite from committed recordings, 25 passed, 7 broken for lack of a model key

• testbed, pnpm --filter @e2e-dev/testbed run test with defaults: 71 passed, 1 skipped; the summary prints Evidence .e2e/evidence/<runId>.evidence. The format's own CLI accepts it:

$ evidence validate .e2e/evidence/01a0fb6c-…evidence --profile L1
ok valid (profile L1, evidence 0.1, status finalized)

• web-benchmark agentic suite from its committed recordings (CI=1 … --config e2e.agent.config.ts --strict-cache, no model key): the pack validates at L1; replayed agent.act steps map with their instruction as label and cache mode (self-finalized); steps that needed a model with none available map to broken with MODEL_PROVIDER_FAILED. Live agent.assert judgments (failed vs expected/actual) were not exercised: they need a model key.
• Live agent steps: the testbed agent suite with --no-cache on OpenAI gpt-6-luna (openai() from @ai-sdk/openai), 24 of 24 passed, pack valid at L1. Agent steps read Act: <instruction> (<what it did>), carry the cursor at the last action's element, keep turns in logs/agent.ndjson, and the run names the model. A planted failing agent.assert produced expected = the instruction and actual = the model's explanation ("The todo list shows one list item, not exactly three."). No secret value in any pack entry; extract keeps no extracted data.
• What it costs. Same machine, tests/forms.e2e.ts tests/todos.e2e.ts, --workers 1, 3 runs each:

Run	Median wall
--no-evidence	6.25 s
evidence on (default, pack only)	6.62 s

The pack (write, seal, validate) adds about 0.37 s here, and about 2 s for the full testbed's 34 MB pack with traces on. Asking for every-step frames adds about 32 ms a step on top (#753). The full parallel testbed suite runs in 27 s either way. evidence-cli seals in memory and synchronously, so a very large pack blocks the event loop while it seals; a streaming seal belongs in that library.

• iOS, mobile-benchmark deterministic suite on an iPhone 17 Pro simulator (iOS 26.5), built locally with expo run:ios --configuration Release:

Run	Result	Wall
--no-evidence (2 runs)	54 passed, 1 skipped	747 s, 795 s
evidence on (2 runs)	52 and 53 passed	872 s, 800 s

infinite scroll and sticky chrome fail with evidence off too (known simulator flakes). toBeFocused follows the focus failed in both evidence-on full runs and passed in both evidence-off ones, but passed 6 of 6 shorter evidence-on runs with the same tests ahead of it, so it is not reproduced on demand. It checks a field the app focuses one frame after mounting; a step frame's accessibility snapshot right after the tap is a plausible cause. Open, and worth CI's mobile job once approved. The pack validates at L1 with resolution: 402x874 on every test and cursor boxes in points.

Checklist

• Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
• Breaking change: not one by the API. It does change the default output and the default run time (see Verified); evidence: false restores both.

[cubic-dev-ai]

All reported issues were addressed across 85 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 23 files (changes from recent commits).

<sub>Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 6 files (changes from recent commits).

<sub>Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[siddhant573]

Hi @okwasniewski,

At TestMu AI we've built a framework-agnostic format for test-run evidence: the open .evidence pack, validated by @testmuai/evidence-cli, with a hosted viewer at evidence.lambdatest.com. This PR integrates it into e2e. Every run writes one sealed pack holding each test's steps, a screenshot per step with the cursor and element box, failure records, and agent turns, so a run can be opened and shared as a single file.

Packs from this branch, uploaded as Actions artifacts (the viewer asks for a GitHub sign-in to read an artifact):

• Testbed suite, 71 tests
• Web benchmark agent suite, replayed from the committed recordings

Locally we also ran the agent suite live on an OpenAI model (24 of 24 passed) and the mobile benchmark on an iOS simulator. Details are under Verified in the description.

This PR builds on #753 (the screenshot: 'every-step' option), so that's the natural place to start. The workflows are waiting for approval since the PR comes from a fork. Whenever you have a moment to approve them and take a look, we'd really appreciate your feedback. We're happy to adjust anything, including making evidence opt-in rather than on by default if you'd prefer.

Thanks!

[kvnloo]

Thanks for making the cost visible in the PR — that makes the tradeoff much easier to reason about.

One design question before making evidence default-on: should the invariant be that diagnostic/evidence settings do not materially perturb execution timing unless explicitly requested? Here the pack itself is cheap, but turning it on also changes the screenshot fallback to every-step, and the measured small suite moves from ~7.1s to ~11.0s.

Could the default pack consume whatever evidence the run already produced, while screenshot: 'every-step' remains an explicit completeness knob (or perhaps a future “changed/mutating steps only” mode)? That would keep the sealed pack default cheap and make timing-sensitive tests less likely to change behavior merely because observability is enabled. The richer per-step visual pack can still be one flag away.

[cubic-dev-ai]

All reported issues were addressed across 17 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[siddhant573]

Thanks @kvnloo, that's a good invariant and I've adopted it in d0dcfec.

Evidence is still on by default, but it no longer touches the screenshot default. screenshot stays on-failure whether evidence is on or off, so the pack holds only the frames the run would have taken anyway. A full per-step visual pack now has to be asked for, with screenshot: 'every-step' or --screenshot every-step. The docs and changesets say this.

The new numbers come from the same machine and the same specs (--workers 1, 3 runs each):

Run	Median wall
--no-evidence	6.25 s
evidence on (default, pack only)	6.62 s

That leaves about 0.37 s for writing, sealing and validating the pack at the end of the run, with no per-step cost. I've updated the description to match.

[okwasniewski]

Thanks for the PR and the thorough work here! We're headed in a different direction: run output stays in the e2e-native report and failure pages, and we won't take a default-on dependency on an external format or viewer. If you want to keep this going, it can ship as a standalone package through the reporters API. Closing for now.

[siddhant573]

Thanks @okwasniewski for taking the time to look at this, and for the clear direction. Keeping run output e2e-native makes sense, and the reporters API turned out to be all a standalone package needs.

We took your suggestion:

• The evidence reporter now ships on its own as @testmuai/e2e-evidence, built on the public reporters API with no changes to e2e. Users opt in with reporters: ['list', evidence()]: feat(integrations): @testmuai/e2e-evidence, an e2e reporter that writes each run as a pack LambdaTest/evidence-cli#18
• One piece is useful to e2e's own report regardless of evidence: steps recording what they were given and where they acted (argument and target). That's a separate, much smaller PR, with no new dependency and no screenshot settings: feat(report): record what each step was given and where it acted #864

Thanks again, and thanks @kvnloo for the review.