Repository navigation
feat(mutation-testing): read the personal override from docs conventions mutation-testing.local.yaml #18084
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
melodic-automation-lanes
wants to merge
4
commits into
main
Choose a base branch
from
work-resolve-issue/issue-18014-44964541536-1
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+418
−22
Open
feat(mutation-testing): read the personal override from docs conventions mutation-testing.local.yaml #18084
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
d2d6cb7
Resolve issue #18014
melodic-automation-lanes[bot] e26384a
Fix failing CI checks on work-resolve-issue/issue-18014-44964541536-1
melodic-automation-lanes[bot] 8a48072
Address review feedback on work-resolve-issue/issue-18014-44964541536-1
melodic-automation-lanes[bot] 5c3e439
Fix failing CI checks on work-resolve-issue/issue-18014-44964541536-1
melodic-automation-lanes[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
9 changes: 9 additions & 0 deletions
9
.changes/mutation-testing/work-resolve-issue-issue-18014-44964541536-1-f70024d0.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| bump: minor | ||
| --- | ||
|
|
||
| ### Changed | ||
|
|
||
| - **The personal override moves to `docs/conventions/mutation-testing.local.yaml`**, beside the team file, where the config-cascade contract puts the personal layer. `setup apply local` writes it there and offers the `docs/conventions/**/*.local.*` ignore line. | ||
| - **An old `.claude/mutation-testing.local.md` is still read**: with no new file it supplies the personal layer as before, and with both present the new file wins and each old key is named as overridden or not applied. Every `check`, `apply` and audit run prints one warning line naming the old file, the new location and `/mutation-testing:setup apply local`. | ||
| - **`setup apply` moves the old file** after a preview and a yes: it writes the new file, checks it resolves the same values, then deletes the old one. New eval cases cover the new file only, the old file only and both present. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
37 changes: 37 additions & 0 deletions
37
plugins/mutation-testing/evals/se-both-personal-files-new-wins/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| schema_version: "1.1" | ||
| name: se-both-personal-files-new-wins | ||
| description: 'Both personal files present. warns-both grades the warning line naming the old file and the new one and saying the new one won; new-wins (llm) checks the new file supplies operators and the old max-mutants is named as not applied; no-write and no-edit grade read-only. Needs --scaffold --allow-tools Bash Write Edit.' | ||
| tags: [setup, scaffold, no-network] | ||
| runs: 3 | ||
| expected_outcome: 'docs/conventions/mutation-testing.local.yaml is the personal layer, so operators is narrow; one WARN line names both files and says the YAML file won; max-mutants 3 from the old file is named as not applied. Nothing is written.' | ||
| context: | ||
| scaffold_script: scaffold.sh | ||
| execution: | ||
| prompt: '/mutation-testing:setup check' | ||
| allowed_tools: [Bash, Read, Glob, Grep, Write, Edit] | ||
| max_turns: 40 | ||
| timeout_seconds: 900 | ||
| graders: | ||
| - name: 'warns-both' | ||
| type: 'regex' | ||
| pattern: 'WARN[^\n]{0,200}\.claude/mutation-testing\.local\.md[^\n]{0,200}docs/conventions/mutation-testing\.local\.yaml[^\n]{0,200}\bwon\b' | ||
| flags: 'i' | ||
| target: 'last_message' | ||
| - name: 'new-wins' | ||
| type: 'llm' | ||
| criteria: 'PASS only if the reply reports the operator set as narrow from docs/conventions/mutation-testing.local.yaml, says that file won over .claude/mutation-testing.local.md, and names max-mutants 3 from the old file as not applied. FAIL if it reports operators as default, applies max-mutants 3, or does not mention the old file''s max-mutants.' | ||
| focus: 'last_message' | ||
| - name: 'no-write' | ||
| type: 'tool_used' | ||
| tool: 'Write' | ||
| input_match: '.' | ||
| min: 0 | ||
| max: 0 | ||
| arm: 'both' | ||
| - name: 'no-edit' | ||
| type: 'tool_used' | ||
| tool: 'Edit' | ||
| input_match: '.' | ||
| min: 0 | ||
| max: 0 | ||
| arm: 'both' | ||
23 changes: 23 additions & 0 deletions
23
plugins/mutation-testing/evals/se-both-personal-files-new-wins/samples/no-edit.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [], | ||
| "why": "no call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Edit", | ||
| "input": { | ||
| "file_path": "/ws/docs/conventions/mutation-testing.local.yaml", | ||
| "old_string": "a", | ||
| "new_string": "b" | ||
| } | ||
| } | ||
| ], | ||
| "why": "an Edit call" | ||
| } | ||
| ] | ||
| } |
22 changes: 22 additions & 0 deletions
22
plugins/mutation-testing/evals/se-both-personal-files-new-wins/samples/no-write.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [], | ||
| "why": "no call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/ws/docs/conventions/mutation-testing.local.yaml", | ||
| "content": "operators: narrow\nmax-mutants: 3\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "a Write call" | ||
| } | ||
| ] | ||
| } |
18 changes: 18 additions & 0 deletions
18
plugins/mutation-testing/evals/se-both-personal-files-new-wins/samples/warns-both.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "WARN: old personal override .claude/mutation-testing.local.md; move it to docs/conventions/mutation-testing.local.yaml with /mutation-testing:setup apply local (docs/conventions/mutation-testing.local.yaml won)", | ||
| "why": "names both, says the new file won" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "WARN: old personal override .claude/mutation-testing.local.md", | ||
| "why": "new file not named" | ||
| }, | ||
| { | ||
| "answer": "WARN: old personal override .claude/mutation-testing.local.md; move it to docs/conventions/mutation-testing.local.yaml with /mutation-testing:setup apply local", | ||
| "why": "the old-file-only line, without saying the new file won" | ||
| } | ||
| ] | ||
| } |
17 changes: 17 additions & 0 deletions
17
plugins/mutation-testing/evals/se-both-personal-files-new-wins/scaffold.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| fixtures="$(cd "$(dirname "${BASH_SOURCE[0]}")/../fixtures" && pwd)" | ||
| bash "${fixtures}/seed-shipping.sh" configured | ||
| printf '.claude/**/*.local.*\ndocs/conventions/**/*.local.*\n' >>.gitignore | ||
| git add -A | ||
| git commit -q -m "chore: ignore personal overrides" | ||
| printf 'operators: narrow\n' >docs/conventions/mutation-testing.local.yaml | ||
| mkdir -p .claude | ||
| cat >.claude/mutation-testing.local.md <<'MD' | ||
| # Personal mutation-testing override | ||
|
|
||
| ```yaml | ||
| operators: default | ||
| max-mutants: 3 | ||
| ``` | ||
| MD |
37 changes: 37 additions & 0 deletions
37
plugins/mutation-testing/evals/se-old-personal-file-dual-read/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| schema_version: "1.1" | ||
| name: se-old-personal-file-dual-read | ||
| description: 'Old personal file only. warns-old-file grades the warning line naming the old file and the new location; old-values-apply (llm) checks the old file still supplies operators and the warning names the move; no-write and no-edit grade read-only. Needs --scaffold --allow-tools Bash Write Edit.' | ||
| tags: [setup, scaffold, no-network] | ||
| runs: 3 | ||
| expected_outcome: 'The old .claude/mutation-testing.local.md still supplies operators narrow, and one WARN line names it, the new location docs/conventions/mutation-testing.local.yaml and /mutation-testing:setup apply local. Nothing is written.' | ||
| context: | ||
| scaffold_script: scaffold.sh | ||
| execution: | ||
| prompt: '/mutation-testing:setup check' | ||
| allowed_tools: [Bash, Read, Glob, Grep, Write, Edit] | ||
| max_turns: 40 | ||
| timeout_seconds: 900 | ||
| graders: | ||
| - name: 'warns-old-file' | ||
| type: 'regex' | ||
| pattern: 'WARN[^\n]{0,200}\.claude/mutation-testing\.local\.md[^\n]{0,200}docs/conventions/mutation-testing\.local\.yaml' | ||
| flags: 'i' | ||
| target: 'last_message' | ||
| - name: 'old-values-apply' | ||
| type: 'llm' | ||
| criteria: 'PASS only if the reply reports the operator set as narrow, supplied by the old personal file .claude/mutation-testing.local.md, and names /mutation-testing:setup apply local as the way to move it to docs/conventions/mutation-testing.local.yaml. FAIL if it reports operators as default, calls the old file ignored or retired, or omits the move.' | ||
| focus: 'last_message' | ||
| - name: 'no-write' | ||
| type: 'tool_used' | ||
| tool: 'Write' | ||
| input_match: '.' | ||
| min: 0 | ||
| max: 0 | ||
| arm: 'both' | ||
| - name: 'no-edit' | ||
| type: 'tool_used' | ||
| tool: 'Edit' | ||
| input_match: '.' | ||
| min: 0 | ||
| max: 0 | ||
| arm: 'both' |
23 changes: 23 additions & 0 deletions
23
plugins/mutation-testing/evals/se-old-personal-file-dual-read/samples/no-edit.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [], | ||
| "why": "no call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Edit", | ||
| "input": { | ||
| "file_path": "/ws/.claude/mutation-testing.local.md", | ||
| "old_string": "a", | ||
| "new_string": "b" | ||
| } | ||
| } | ||
| ], | ||
| "why": "an Edit call" | ||
| } | ||
| ] | ||
| } |
22 changes: 22 additions & 0 deletions
22
plugins/mutation-testing/evals/se-old-personal-file-dual-read/samples/no-write.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [], | ||
| "why": "no call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/ws/docs/conventions/mutation-testing.local.yaml", | ||
| "content": "operators: narrow\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "a Write call" | ||
| } | ||
| ] | ||
| } |
14 changes: 14 additions & 0 deletions
14
plugins/mutation-testing/evals/se-old-personal-file-dual-read/samples/warns-old-file.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "WARN: old personal override .claude/mutation-testing.local.md; move it to docs/conventions/mutation-testing.local.yaml with /mutation-testing:setup apply local", | ||
| "why": "warning line" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "| Config presence | PASS | operators: narrow (.claude/mutation-testing.local.md) |", | ||
| "why": "no warning" | ||
| } | ||
| ] | ||
| } |
15 changes: 15 additions & 0 deletions
15
plugins/mutation-testing/evals/se-old-personal-file-dual-read/scaffold.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| fixtures="$(cd "$(dirname "${BASH_SOURCE[0]}")/../fixtures" && pwd)" | ||
| bash "${fixtures}/seed-shipping.sh" configured | ||
| printf '.claude/**/*.local.*\ndocs/conventions/**/*.local.*\n' >>.gitignore | ||
| git add -A | ||
| git commit -q -m "chore: ignore personal overrides" | ||
| mkdir -p .claude | ||
| cat >.claude/mutation-testing.local.md <<'MD' | ||
| # Personal mutation-testing override | ||
|
|
||
| ```yaml | ||
| operators: narrow | ||
| ``` | ||
| MD |
37 changes: 37 additions & 0 deletions
37
plugins/mutation-testing/evals/se-personal-yaml-layer/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| schema_version: "1.1" | ||
| name: se-personal-yaml-layer | ||
| description: 'Personal layer at the new location only. narrow-from-personal grades the merged value and its layer; no-old-path-warning (llm) checks no old-file warning; no-write and no-edit grade read-only. Needs --scaffold --allow-tools Bash Write Edit.' | ||
| tags: [setup, scaffold, no-network] | ||
| runs: 3 | ||
| expected_outcome: 'Probe 6 reports operators narrow from the personal file docs/conventions/mutation-testing.local.yaml, prints no old-path warning, and writes nothing.' | ||
| context: | ||
| scaffold_script: scaffold.sh | ||
| execution: | ||
| prompt: '/mutation-testing:setup check' | ||
| allowed_tools: [Bash, Read, Glob, Grep, Write, Edit] | ||
| max_turns: 40 | ||
| timeout_seconds: 900 | ||
| graders: | ||
| - name: 'narrow-from-personal' | ||
| type: 'regex' | ||
| pattern: 'narrow[^\n]{0,160}mutation-testing\.local\.yaml|mutation-testing\.local\.yaml[^\n]{0,160}narrow' | ||
| flags: 'i' | ||
| target: 'last_message' | ||
| - name: 'no-old-path-warning' | ||
| type: 'llm' | ||
| criteria: 'PASS only if the reply reports the operator set as narrow, supplied by the personal override docs/conventions/mutation-testing.local.yaml, and prints no warning about an old personal file at .claude/mutation-testing.local.md. FAIL if it ignores the personal file, reports operators as default, or warns about an old personal file that does not exist.' | ||
| focus: 'last_message' | ||
| - name: 'no-write' | ||
| type: 'tool_used' | ||
| tool: 'Write' | ||
| input_match: '.' | ||
| min: 0 | ||
| max: 0 | ||
| arm: 'both' | ||
| - name: 'no-edit' | ||
| type: 'tool_used' | ||
| tool: 'Edit' | ||
| input_match: '.' | ||
| min: 0 | ||
| max: 0 | ||
| arm: 'both' |
14 changes: 14 additions & 0 deletions
14
plugins/mutation-testing/evals/se-personal-yaml-layer/samples/narrow-from-personal.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "| Config presence | PASS | operators: narrow (docs/conventions/mutation-testing.local.yaml) |", | ||
| "why": "value and its layer" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "| Config presence | PASS | operators: default (docs/conventions/mutation-testing.yaml) |", | ||
| "why": "personal layer ignored" | ||
| } | ||
| ] | ||
| } |
23 changes: 23 additions & 0 deletions
23
plugins/mutation-testing/evals/se-personal-yaml-layer/samples/no-edit.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [], | ||
| "why": "no call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Edit", | ||
| "input": { | ||
| "file_path": "/ws/x.py", | ||
| "old_string": "a", | ||
| "new_string": "b" | ||
| } | ||
| } | ||
| ], | ||
| "why": "an Edit call" | ||
| } | ||
| ] | ||
| } |
22 changes: 22 additions & 0 deletions
22
plugins/mutation-testing/evals/se-personal-yaml-layer/samples/no-write.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [], | ||
| "why": "no call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/ws/x.py", | ||
| "content": "x" | ||
| } | ||
| } | ||
| ], | ||
| "why": "a Write call" | ||
| } | ||
| ] | ||
| } |
8 changes: 8 additions & 0 deletions
8
plugins/mutation-testing/evals/se-personal-yaml-layer/scaffold.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| fixtures="$(cd "$(dirname "${BASH_SOURCE[0]}")/../fixtures" && pwd)" | ||
| bash "${fixtures}/seed-shipping.sh" configured | ||
| printf 'docs/conventions/**/*.local.*\n' >>.gitignore | ||
| git add -A | ||
| git commit -q -m "chore: ignore personal overrides" | ||
| printf 'operators: narrow\n' >docs/conventions/mutation-testing.local.yaml |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.