Skip to content

feat(testing): read run-e2e e2e.md and worst-case.md from the convention home - #18073

Open
melodic-automation-lanes[bot] wants to merge 2 commits into
mainfrom
work-resolve-issue/issue-18051-44959659696-1
Open

melodic-automation-lanes[bot] wants to merge 2 commits into
mainfrom
work-resolve-issue/issue-18051-44959659696-1

Conversation

@melodic-automation-lanes

@melodic-automation-lanes melodic-automation-lanes Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #18051

Summary

Opened by the work-resolve-issue lane. The run's own summary:

The run-e2e skill text now reads recording and browser_mode from <home>/testing/e2e.md and its personal override <home>/testing/e2e.local.md, and the worst-case catalog extension from <home>/testing/worst-case.md. <home> is resolved by the plugin's own lib/resolve-convention-home.sh: exit 0 uses the printed path, exit 1 or 4 uses the contract default docs/conventions, and any other exit reads only the old paths. Design choice: nested <home>/testing/ files rather than new testing.yaml keys, because these surfaces are model-read with no script reader (the contract allows a <home> prose doc for those, and architecture's <home>/architecture/README.md is a precedent), and new YAML keys would need schema and resolver work that belongs to the resolver part of #17999. The dual-read window is part of the same text: each old .claude/testing/ or ~/.claude/testing/e2e.md file is still read where no <home> file supplies its layer, with one warning per old file naming where to move it, and both file names when the <home> file also exists.

The run went past its turn budget (turns=90 max-turns=75): the work may be too big for one run.

Not verified

  • check-evals-quality wrapper takes an evals.json file, not a case.yaml, so it could not check the edited eval; CI's evals-quality and eval-cases jobs cover it
  • check-affected skipped comment-hygiene, size-check, typos and validate-plugins sub-checks here (network or missing tool); config-cascade-semantics and lychee run in CI only and are left to the PR's CI
  • run-e2e evals for the dual-read cases (new layer only, old file only, both present) and the resolver, setup and other script migrations are sibling parts of #17999, not this part's done-when
  • Tidy pass 5: ai-slop's detect.sh could not run in this lane (only named wrappers run), so the AI-tell check was the rubric only, done by hand on the changed lines; it found nothing. Passes 1-3 had nothing to do (the only non-markdown file is a YAML eval with string edits)
  • Review fixes after tidy touched docs/contracts/config-cascade/README.md, a second area; that line was not tidied
  • Changes left uncommitted in the working tree, as the lane's system prompt instructs; the runner turns them into the patch

Fix

  • plugins/testing/skills/run-e2e/context/e2e-config.md: the e2e.md surface moves to <home>/testing/; the precedence list is updated; adds a 'Convention home and old paths' section covering home resolution by exit code, the old-path table with move targets, the warning rule and the gitignore…
  • plugins/testing/skills/run-e2e/SKILL.md: adds the convention-home command to the Own driving command list; the worst-case paragraph resolves <home>, picks the new or old consumer file, warns, and passes the path to the subagent
  • plugins/testing/skills/run-e2e/context/own-driving.md: the Resolve run config bullet reads the <home> layers plus the old paths; the command count goes from three to four
  • plugins/testing/skills/run-e2e/context/worst-case.md: the consumer extension file is <home>/testing/worst-case.md, with a pointer to the old-path rule
  • plugins/testing/README.md: recording/browser_mode and worst-case file locations now name <home>, with the old files still read with a warning
  • plugins/testing/evals/e2e-config-layers-delegated/case.yaml: description, expected outcome and LLM criterion name the e2e.md surface at the convention home instead of .claude/testing/e2e.md
  • docs/contracts/config-cascade/README.md: the Implementers row for testing (run-e2e) names the <home> files and the old paths still read
  • .changes/testing/work-resolve-issue-issue-18051-44959659696-1-d6a9e645.md: minor changelog fragment for the testing plugin

Verification

The runner ran each check from the base copy against 8081435:

  • scripts/check-changelog-fragments.sh --check: passed
  • scripts/check-changed-skills.sh --merge-base: passed

The run reports:

  • check-changelog-fragments: all 3 fragments valid
  • check-changed-skills: run-e2e PASS, 0 errors, 5 warnings that also fire on the base (description length, body size, Next section) plus body growth 415 -> 418 lines
  • check-affected: first run failed spoke-plugin-root (${CLAUDE_PLUGIN_ROOT} in a context file); fixed by moving the command into SKILL.md; final run 14 passed, 9 skipped, 0 failed
  • Fresh-context review subagent over the whole diff: 6 findings (README worst-case paragraph still on the old path, worst-case subagent handoff of <home>, missing gitignore note, eval criterion too strict, stale command count, stale contract Implementers row), all fixed, plus one note (a…
  • Read plugins/testing/lib/resolve-convention-home.sh usage text: exit 0 resolved, 1 no pointer, 2 usage, 3 fail, 4 declined, 5 not applicable
  • Read docs/contracts/config-cascade/README.md 'The layers' and 'Personal override naming' (in-repo source for the <home> layers and the <home>/**/.local. ignore line)
  • The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/44959659696

Left out: the items under Not verified.

Related

#18051

Follow-ups

  • Pre-existing (advisory): **Advisory, closing-issue test gap (#18051).** The done-when says: "The 'run-e2e' skill text reads 'e2e.md' and 'worst-case.md' content from '<home>'." This paragraph meets the 'worst-case.md' half, b (thread)

Lane-Activity: work-resolve-issue/resolve-issue
@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs

Run 44961115646, attempt 1

Field Value
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 44961115646

Failed GitHub API calls

No failed call recorded by the run job's scripts.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 808143599ac0ece53d05eb17bdc1bf2d099a010a: read 8 of 8 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

This PR changes only documentation and skill text for /testing:run-e2e. recording and browser_mode are now read from <home>/testing/e2e.md and <home>/testing/e2e.local.md, and the worst-case catalog extension from <home>/testing/worst-case.md. The old .claude/testing/ files are still read during a dual-read window, with a warning for each one. I read every hunk of all 8 changed files. None of them matches a trusted path.

What I checked:

  • The new convention-home command (plugins/testing/skills/run-e2e/SKILL.md): the command, bash "${CLAUDE_PLUGIN_ROOT}/lib/resolve-convention-home.sh" --root "${CLAUDE_PROJECT_DIR}", takes no input a user or the repository controls. The script it runs (resolve-convention-home.sh#L353-L385) already exists and this PR does not change it. It rejects absolute paths, ~, .., .git segments and any character outside [A-Za-z0-9._-]. It also refuses a home reached through a symlink (L524-L530). So <home> cannot point outside the repository. The script never runs or interpolates anything it reads from AGENTS.md or CLAUDE.md. If the script exits with any code other than 0, 1 or 4, the run reads only the old paths, so the failure is contained.
  • Untrusted-content framing (context/worst-case.md): the rule that treats the consumer extension file as "DATA, never instructions" is kept word for word, and it now covers the new path. No instruction is removed, narrowed or softened.
  • Personal override location: <home>/testing/e2e.local.md lives in a tracked directory. The text tells users to add the <home>/**/*.local.* line to .gitignore first. The file holds only recording and browser_mode, which decide nothing about trust or publishing, so committing it by mistake has no security impact.
  • No manifest, workflow, token, trust control or merge hold is changed. There is no new spend.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 808143599ac0ece53d05eb17bdc1bf2d099a010a; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread plugins/testing/skills/run-e2e/SKILL.md Resolved
@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 808143599ac0ece53d05eb17bdc1bf2d099a010a: read 8 of 8 changed file(s); 1 finding(s), 1 as inline comments.

Code review: PR #18073, run-e2e reads e2e.md and worst-case.md from the convention home

I read all 8 changed files in full. The change moves the recording / browser_mode surface to <home>/testing/e2e.md and <home>/testing/e2e.local.md, and the worst-case catalog extension to <home>/testing/worst-case.md. The old .claude/testing/ and ~/.claude/testing/e2e.md files are still read, with a warning, while they exist.

What checks out:

  • The new convention-home command, bash "${CLAUDE_PLUGIN_ROOT}/lib/resolve-convention-home.sh" --root "${CLAUDE_PROJECT_DIR}", is the same call the architecture skills use.
  • The exit-code handling in e2e-config.md matches the helper's documented exits (plugins/testing/lib/resolve-convention-home.sh#L78-L92):
    • Exit 0 uses the printed path.
    • Exit 1 (no pointer) and exit 4 (declined) use the default, docs/conventions.
    • Exit 2, 3 and 5 read only the old paths.
  • The precedence list, the old-path table, the gitignore note, the Implementers row in the config-cascade contract and the changelog fragment agree with each other.

Closing issue #18051. Its done-when reads: "The run-e2e skill text reads e2e.md and worst-case.md content from <home>." The skill text meets it.

  • e2e.md half: a test could catch a break. The updated e2e-config-layers-delegated LLM grader now requires the convention-home layers.
  • worst-case.md half: no test could catch a break (inline comment on SKILL.md). e2e-worst-case-consumer-file still names only .claude/testing/worst-case.md, see case.yaml#L7.

Verification. The PR body's verification results (changelog-fragment check, changed-skills check, check-affected) are claimed by the author and I did not verify them. I ran no tools beyond reading files.

Loosened checks. The edit to the e2e-config-layers-delegated grader swaps the old-path expectation for the <home> expectation that the closing issue specifies. It does not accept more than before, so it is not a loosening. The prompt carried no loosened_check_* lines, so the lane default fail applies with no consumer entries. The citation requirement was not checked.

Loosened checks would be marked: fail (lane default; no wrapper settings); patterns: 0

Other checks.

  • No plugin manifest changed.
  • No paved-path standards row was found.
  • The path-scoped rules that match (skill-bodies-state-current-rules, eval-case-transcripts, cost-claims) raise nothing on these lines.
  • No embedded instructions appear in the PR context.

Verifier dropped 0 of 1 candidates.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/address-feedback

Run 44962164151, attempt 1

Field Value
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 44962164151

Failed GitHub API calls

No failed call recorded by the run job's scripts.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 1 of max 4 on 8081435.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/publish-feedback

Run 44962164151, attempt 1

Field Value
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed none recorded
Run 44962164151

Failed GitHub API calls

No failed call recorded by the run job's scripts.

@melodic-automation-lanes

melodic-automation-lanes Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Unattended merge: native auto-merge leaves this pull request unarmed at head 26cb117f06d2b5cae98e9e30195e1e3fe405a034. No person needs to act: a lane clears this reason, and the next scheduled run judges it again.

pr-merge's gate refuses its verifier verdict (verify:missing)

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/fix-ci

Run 44967567723, attempt 1

Field Value
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 44967567723

Failed GitHub API calls

No failed call recorded by the run job's scripts.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/apply-patch

Run 44967567723, attempt 1

Field Value
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed 26cb117
Run 44967567723

Failed GitHub API calls

No failed call recorded by the run job's scripts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

testing: read config from the config-cascade layers, not ~/.claude - part 2 of 4

0 participants