Skip to content

feat(report): record what each step was given and where it acted - #864

Closed
siddhant573 wants to merge 3 commits into
tester-army:mainfrom
siddhant573:feat/report-step-detail
Closed

siddhant573 wants to merge 3 commits into
tester-army:mainfrom
siddhant573:feat/report-step-detail

Conversation

@siddhant573

@siddhant573 siddhant573 commented Oct 5, 2026 •

Copy link
Copy Markdown

Report steps say what each step was given and where it acted. Today a step's label names its target (getByLabel("Username")), but the value it typed, the key it pressed, and the value an assertion expected are not in report.json at all, and neither is where on the screen it acted.

  • argument (string, ≤1024 bytes): the text a fill or pressSequentially typed, the key a press sent, the option a selectOption picked, the files a setInputFiles uploaded (basenames), and the value an expect matcher expected (not <expected> for a negated one). A secret is always <secret:name>, never its value.
  • Typed text and keys fail closed (packages/e2e/src/locator/screen.ts, recordedInto). The locator reads the node it resolved before acting, and records the input only when that node is not secure and takes it itself: a text field for typing, a field or control for a key. Anything else is <withheld>, because an engine can hand input on: a fill on a <label> goes to its control, which can be a password field. A failed read withholds too. The recorder redacts and sanitizes again before keeping it (packages/e2e/src/run/steps.ts, boundArgument). An assertion's expected value is withheld when it reads a secure field.
  • target ({ box?, point? }, CSS pixels like SemanticNode.rect): the box of the node a step resolved to, and the point of a positioned tap (tapAt, tap({ position })). An agent step records its last action's, from ActionDispatcher, and none when that action had no node or point (a scroll, back, a wait).
  • Schema: report-v1.schema.json step gains both optional fields; the valid fixture uses them, the invalid one types them wrong. Docs: debugging.mdx (what steps holds) and security.mdx (the redaction table).
{ "api": "locator.fill", "label": "getByLabel(\"Username\")", "argument": "\"admin\"",
  "target": { "box": { "x": 110.48, "y": 166.88, "width": 153, "height": 21 } } }
{ "api": "locator.fill", "label": "getByLabel(\"Password\")", "argument": "<secret:admin.password>", … }
{ "api": "expect.toHaveText", "label": "getByRole(\"alert\")", "argument": "text \"Invalid credentials\"" }

Reporters read both from run.report. It also makes a failing step's report readable without opening the trace: which value went in and which was expected are on the step.

Verified

Ran it locally: yes

  • testbed, pnpm --filter @e2e-dev/testbed test on the built packages: 71 passed, 1 skipped. Of 458 last-attempt steps in apps/testbed/.e2e/report.json, 225 carry argument and 157 target; every locator verb (check, clear, click at a position, doubleTap, dragTo, fill, focus, hover, longPress, press, pressSequentially, scrollIntoView, secondaryTap, selectOption) has its box, the positioned click its point.
  • Secrets: the testbed's login fills admin.password through a Secret, which records <secret:admin.password> and its box; a plain string typed into its password input records <withheld>. Keys pressed into ordinary fields keep the key (Tab, Shift+Tab, Control+a, Alt+Shift+Enter, Enter). The password value (admin-pass) appears 0 times in report.json and 0 times under .e2e/artifacts.
$ jq -c '... | select(.api|test("^locator\\.")) | {api, argument, target}' apps/testbed/.e2e/report.json
{"api":"locator.fill","argument":"\"admin\"","target":{"box":{"x":110.484375,"y":166.875,"width":153,"height":21}}}
{"api":"locator.press","argument":"Tab","target":{"box":{"x":512.1875,"y":311.46875,"width":153,"height":21}}}
{"api":"locator.pressSequentially","argument":"\"Wa\"","target":{"box":{"x":509.078125,"y":187.875,"width":153,"height":21}}}
{"api":"locator.selectOption","argument":"\"Blue\"","target":{"box":{"x":210.859375,"y":276.46875,"width":43.0625,"height":53}}}
{"api":"locator.click","argument":null,"target":{"box":{"x":32,"y":166.875,"width":320,"height":200},"point":{"x":32,"y":166.875}}}

New tests:

  • tests/integration/step-argument.test.ts: typed text, key, and expected value, and a secret by name with its box; plain text typed into a secure field withheld; a fill and a key on a <label>, and a key on a password field, withheld; the 1024-byte cap; a secure field's expected value withheld; a locator action's box and a positioned tap's point. The label and secure-key cases fail on the first commit of this branch.
  • tests/integration/agent-step-target.test.ts, real web engine and scripted executor: an agent tap's box equals the observation's box for that node; a point tap records the point and the box it landed in, and on a control the control's box as a node tap; an act whose last action had no node records no target.
  • tests/unit/schema-fixtures.test.ts: the schema rejects a non-string or 1025-character argument, and a target with a missing, negative, or non-numeric field or an unknown key.

A report step now carries argument: the text a fill typed, the key a press
sent, the option selected, the files uploaded, or the value an assertion
expected. It is redacted and capped at 1024 bytes; a secret is recorded
as <secret:name>, and text typed into a secure field as <withheld>,
decided by reading the field before typing, so plain text typed into a
password input is as private as a secret.

It also carries target: the CSS-pixel box of the node the step resolved
to, and the point of a positioned tap. An agent step records its last
action's.

The report-1 schema, both fixtures, and the debugging and security docs
change with it.
…olved

Typed text and a pressed key are recorded only when the node the locator
resolved was read, is not secure, and takes the input itself: a text field
for typing, a field or control for a key. Anything else is <withheld>,
since an engine may hand input on (a fill on a <label> goes to its
control, which can be a password field). A key pressed on a password
field was recorded before; it is withheld now.

A secret fill and a locator swipe record their box like every other
locator action. An agent step's target is its last action's, and none
when that action had no node or point. The docs say a secret is always
<secret:name>, and which input is withheld.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 15 files

Reply to a comment to ask cubic a question or push back. It learns from your replies.

Re-trigger cubic

Comment thread packages/e2e/src/expect/async.ts Resolved
Comment thread docs/debugging.mdx Resolved
Comment thread packages/e2e/src/locator/screen.ts Resolved
Comment thread docs/security.mdx Resolved
… before it goes in

A negated assertion's argument reads `not <expected>`, as its failure
does. An agent step forgets its target when a relocated node has no box
and when a mutating project tool runs. Typed text and keys are decided
from the inspected node before the input goes in, so a step that times out
midway keeps its argument.

The valid report fixture carries its target on an act step, the schema
says why it bounds the argument in characters, and the docs describe a
step's label as the call and the assertion rule apart from typed input.
The agent target test compares the box with the observation's own, and
covers a point tap and an action with no node.
@okwasniewski

Copy link
Copy Markdown
Member

hey, sorry for the delay on this one. I'm landing #869 which brings in e2e own's trace reporting system, once the design settles I'll ask you to rework it on top of that.

@okwasniewski

Copy link
Copy Markdown
Member

Automated review (Claude Code on behalf of @okwasniewski)

Thanks for the careful work here, especially the fail-closed handling of secure fields. Since #869 landed, steps record the node each action landed on, the values assertions read, and screen changes, and trace.md shows them, so most of this is covered. What's left is typed values and pixel boxes. Those would need a rewrite on the new step events, with a use case behind them. Closing for now. If you hit a concrete need for the typed value in reports, please open an issue and we can design it on top of the trace events.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants