Skip to content

Cut the model-read ecosystem specifics from testing: run-e2e browser driver guidance and the README default #17743

Description

@kyle-sexton

Part 3 of 5 of #16755. Refs #15703.

Problem

testing:run-e2e ranks third-party browser tools and gives their command sequences, and the README ships playwright-cli as the bundled browser_driver default, which picks the consumer's browser tool (R4).

Rows (7: R11 6, R4 1; plus F0449)

Row Rule Kind Cited at (as of 5b66a7f) Fact
F0441 R11 K4 plugins/testing/skills/run-e2e/context/e2e.md:5-29 prerequisites matrix (@playwright/cli 0.1.x floor, npm install -g, Chrome DevTools MCP, Claude in Chrome)
F0442 R11 K4 .../run-e2e/context/e2e.md:51-108 browser-tool rubric and driver ranking
F0443 R11 K4 .../run-e2e/context/e2e.md:110-185 flow-suite and render-check sections (@playwright/test, Cypress, @axe-core/playwright, toHaveScreenshot)
F0444 R11 K4 .../run-e2e/context/e2e.md:225-300 curl/jq and playwright-cli -s=<session> command sequences
F0445 R11 K4 .../run-e2e/context/own-driving.md:27-40,114-135,192-201 playwright-cli session usage; lines about this marketplace's keys and check-browser-driver.sh are R6 and stay
F0448 R11 K5 plugins/testing/skills/run-e2e/SKILL.md:34-35,346,358,374-375 Aspire, docker-compose, tilt, Playwright CLI, axe as examples and pointers
F0453 R4 K6 plugins/testing/README.md:12-17,102-116,194-196 browser_driver default playwright-cli (the coverage claim at 12-17 is R7 and the e2e_driver ids are R6; both stay)

F0449: #16755 counts 26 rows (R4 2) but its table holds 25. F0449, the /playwright:playwright skill pointer in run-e2e/SKILL.md that F0448's note names, has no row of its own. Re-read that site and record F0449 in the pull request body as R4 (cut with F0453), R11 (cut) or GONE.

F0453 and the driver rows (F0441-F0445, F0448) stay together: the default and the guidance it selects go in one pull request.

Files

  • Cut: plugins/testing/skills/run-e2e/context/e2e.md, .../run-e2e/context/own-driving.md, plugins/testing/skills/run-e2e/SKILL.md, plugins/testing/README.md (and the config-resolution text the default lives in, if the README restates it).
  • Outside references at d46efa9: scripts/docs-anchor-baseline.txt:125 (e2e.md, count 2); scripts/em-dash-purged-paths.txt:479 (README; stale only if the file is deleted).

Rules this part follows (from #16755, rules revision 5)

  • R11 (shipped knowledge): model-read K2, K3, K4, K5 or K7 text. Generic text replaces it only at a routing site where the skill still takes a language-specific step; anywhere else it is deleted with no replacement. An item kept only because a removed item consumes it goes in the same pull request.
  • R4 (consumer default): the value picks a tool or library for the consumer. It goes: no bundled default survives (ADR 0101 decision 2, "Ship no bundled default"); the choice is resolved from the consumer through docs/contracts/ecosystem-resolution/.
  • R8 (measured fact): cut it like the other rows and mark it in the pull request body as pending its targeted case; the host session's eval (below) decides whether it comes back, annotated portability-ok: measured <case-id>.
  • AC1's named exceptions are the kept classes of ADR 0101 (docs/adr/0101-resolve-ecosystem-specifics-from-the-consumer-repository.md, Decision and the 2026-11-17 amendment). Bundled defaults that pick a tool, command or library are retired wherever they appear (Interview Q32, D4 of the Brief).
  • Line numbers below were read at 5b66a7fa and may have moved; find each row by its cited text.

After the pull request opens (host session, not the lane)

The lane opens the pull request with Closes this issue and stops; it adds no label. The pull request stays held for the eval:

  • Hold line: a sweep or host session adds do-not-merge and posts, from a listed human's account, a comment with <!-- pr-hold reason=eval --> on a line of its own.
  • Targeted case per changed skill: one before-and-after case on sonnet at 3 runs a side, --case, --runs 3, a --max-cost-usd ceiling; opus only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first; start no batch after a usage-limit error (owner decision of 2026-11-16).
  • Controls: the owner's Q53 decision of 2026-11-17 (owner-decisions.md L202) reads: "each cut pull request is gated on a no-loss compare alone, with no wrong-text side". It supersedes Cut the model-read ecosystem specifics from testing #16755's Controls line and its AC7 sentence ("each partial cut carries a detection-control result"). A whole-page deletion still states its predicted outcome in the pull request body before the first batch, and the record says the case cannot detect wrong text.
  • On a LOSS reading: a trace read of each dropped case and an opus confirm; only a loss traced to the changed text counts.
  • R8 rows: the row's targeted case with and without it; on a case drop or a mean fall of 0.15 or more without it, restore the row annotated portability-ok: measured <case-id>; a deterministic grader states before the run in how many of n runs it must fail.
  • Record: on a VALID no-loss run, post from a listed human's account <!-- eval-pair verdict=no-loss validity=VALID head=<40-hex head SHA> --> naming the current head, with no "do not merge" text (docs/contracts/loop-lane/README.md, "Cross-lane PR hold"). pr-merge's "Release eval holds whose record matches the head" step (.github/scripts/pr-merge/release-eval-hold.mjs) then removes do-not-merge; a later push needs a new record.

Out of scope

Rows outside this part's table, the parent's kept rows, every other plugin, toolchain's bundled defaults (#15733), and widening the portability gate (#15732).

Done when

  • Every row in this part's table is cut or replaced at a routing site, and the README sites F0453 cites no longer state a playwright-cli default; a row whose cited text is already gone on main is listed as GONE. The userConfig default at plugins/testing/.claude-plugin/plugin.json:38-39 is manifest config, not model-read text. It stays, and the pull request body says it stays. The pull request body says whether that manifest default is also R4.
  • The pull request body states this part's row count (7, plus F0449's outcome) and lists every row id with its rule and outcome.
  • Each R11 row's script-reader search is quoted in the pull request body; a row whose file a script reads (for example check-browser-driver.sh or resolve-config.sh) is not cut, and is listed in the pull request body under a Script readers heading.
  • The pull request body names, for run-e2e, the existing eval case that reads the cut text, or says none does.
  • The pull request body says, for each changed file, whether the cut deletes the whole page or part of it.
  • scripts/docs-anchor-baseline.txt:125 matches e2e.md's remaining anchors.
  • AC10: no new enforcement-placing skill is added.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 21, 2026
  2. github-actions commented on Nov 21, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none
    Blast radius: the testing plugin's run-e2e skill pages (e2e.md, own-driving.md, SKILL.md), its README, and the docs-anchor baseline line. Readers are the model running run-e2e and consumers of the plugin.
    Reversible: yes; revert the pull request. Any row the eval later shows to matter is restored with a portability-ok: measured <case-id> annotation.
    Open questions: none
    I understood this as part 3 of #16755: cut the seven listed rows (F0441-F0445, F0448, F0453) plus F0449's outcome. That means removing the browser-tool ranking and command sequences from run-e2e and the playwright-cli default from the README, and fixing the anchor baseline. The scope and done-when are exact. The host-session eval steps are outside what this lane can run. The sibling issues named "Cut the model-read ecosystem specifics from testing: ..." cover other rows, so they are not duplicates. I found no merged pull request that already does this.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 21, 2026
  4. melodic-automation-lanes commented on Nov 21, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"232a2aa878aec7d24215336986223aae83095d6c17604e4907bb4c07059d5b0d","body_sha256":"8fe0c1b33aba2e7bac4c271445fda27daecf28c22c3367a4c28d266b2ee87579"}

  5. github-actions commented on Nov 21, 2026

    @github-actions
    Contributor
    No description provided.
  6. github-actions commented on Nov 21, 2026

    @github-actions
    Contributor
    No description provided.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    needs-triage

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions