Repository navigation
feat(report): record what each step was given and where it acted - #864
siddhant573 wants to merge 3 commits into
Conversation
A report step now carries argument: the text a fill typed, the key a press sent, the option selected, the files uploaded, or the value an assertion expected. It is redacted and capped at 1024 bytes; a secret is recorded as <secret:name>, and text typed into a secure field as <withheld>, decided by reading the field before typing, so plain text typed into a password input is as private as a secret. It also carries target: the CSS-pixel box of the node the step resolved to, and the point of a positioned tap. An agent step records its last action's. The report-1 schema, both fixtures, and the debugging and security docs change with it.
…olved Typed text and a pressed key are recorded only when the node the locator resolved was read, is not secure, and takes the input itself: a text field for typing, a field or control for a key. Anything else is <withheld>, since an engine may hand input on (a fill on a <label> goes to its control, which can be a password field). A key pressed on a password field was recorded before; it is withheld now. A secret fill and a locator swipe record their box like every other locator action. An agent step's target is its last action's, and none when that action had no node or point. The docs say a secret is always <secret:name>, and which input is withheld.
There was a problem hiding this comment.
All reported issues were addressed across 15 files
Reply to a comment to ask cubic a question or push back. It learns from your replies.
Re-trigger cubic
… before it goes in A negated assertion's argument reads `not <expected>`, as its failure does. An agent step forgets its target when a relocated node has no box and when a mutating project tool runs. Typed text and keys are decided from the inspected node before the input goes in, so a step that times out midway keeps its argument. The valid report fixture carries its target on an act step, the schema says why it bounds the argument in characters, and the docs describe a step's label as the call and the assertion rule apart from typed input. The agent target test compares the box with the observation's own, and covers a point tap and an action with no node.
|
hey, sorry for the delay on this one. I'm landing #869 which brings in e2e own's trace reporting system, once the design settles I'll ask you to rework it on top of that. |
|
Automated review (Claude Code on behalf of @okwasniewski) Thanks for the careful work here, especially the fail-closed handling of secure fields. Since #869 landed, steps record the node each action landed on, the values assertions read, and screen changes, and |
Report steps say what each step was given and where it acted. Today a step's
labelnames its target (getByLabel("Username")), but the value it typed, the key it pressed, and the value an assertion expected are not inreport.jsonat all, and neither is where on the screen it acted.argument(string, ≤1024 bytes): the text afillorpressSequentiallytyped, the key apresssent, the option aselectOptionpicked, the files asetInputFilesuploaded (basenames), and the value anexpectmatcher expected (not <expected>for a negated one). A secret is always<secret:name>, never its value.packages/e2e/src/locator/screen.ts,recordedInto). The locator reads the node it resolved before acting, and records the input only when that node is not secure and takes it itself: a text field for typing, a field or control for a key. Anything else is<withheld>, because an engine can hand input on: a fill on a<label>goes to its control, which can be a password field. A failed read withholds too. The recorder redacts and sanitizes again before keeping it (packages/e2e/src/run/steps.ts,boundArgument). An assertion's expected value is withheld when it reads a secure field.target({ box?, point? }, CSS pixels likeSemanticNode.rect): the box of the node a step resolved to, and the point of a positioned tap (tapAt,tap({ position })). An agent step records its last action's, fromActionDispatcher, and none when that action had no node or point (a scroll, back, a wait).report-v1.schema.jsonstep gains both optional fields; the valid fixture uses them, the invalid one types them wrong. Docs:debugging.mdx(whatstepsholds) andsecurity.mdx(the redaction table).{ "api": "locator.fill", "label": "getByLabel(\"Username\")", "argument": "\"admin\"", "target": { "box": { "x": 110.48, "y": 166.88, "width": 153, "height": 21 } } } { "api": "locator.fill", "label": "getByLabel(\"Password\")", "argument": "<secret:admin.password>", … } { "api": "expect.toHaveText", "label": "getByRole(\"alert\")", "argument": "text \"Invalid credentials\"" }Reporters read both from
run.report. It also makes a failing step's report readable without opening the trace: which value went in and which was expected are on the step.Verified
Ran it locally: yes
pnpm --filter @e2e-dev/testbed teston the built packages: 71 passed, 1 skipped. Of 458 last-attempt steps inapps/testbed/.e2e/report.json, 225 carryargumentand 157target; every locator verb (check, clear, click at a position, doubleTap, dragTo, fill, focus, hover, longPress, press, pressSequentially, scrollIntoView, secondaryTap, selectOption) has its box, the positioned click its point.admin.passwordthrough aSecret, which records<secret:admin.password>and its box; a plain string typed into its password input records<withheld>. Keys pressed into ordinary fields keep the key (Tab,Shift+Tab,Control+a,Alt+Shift+Enter,Enter). The password value (admin-pass) appears 0 times inreport.jsonand 0 times under.e2e/artifacts.New tests:
tests/integration/step-argument.test.ts: typed text, key, and expected value, and a secret by name with its box; plain text typed into a secure field withheld; a fill and a key on a<label>, and a key on a password field, withheld; the 1024-byte cap; a secure field's expected value withheld; a locator action's box and a positioned tap's point. The label and secure-key cases fail on the first commit of this branch.tests/integration/agent-step-target.test.ts, real web engine and scripted executor: an agent tap's box equals the observation's box for that node; a point tap records the point and the box it landed in, and on a control the control's box as a node tap; an act whose last action had no node records no target.tests/unit/schema-fixtures.test.ts: the schema rejects a non-string or 1025-characterargument, and atargetwith a missing, negative, or non-numeric field or an unknown key.