Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
bump: minor
---

### Changed

- **The personal override moves to `docs/conventions/mutation-testing.local.yaml`**, beside the team file, where the config-cascade contract puts the personal layer. `setup apply local` writes it there and offers the `docs/conventions/**/*.local.*` ignore line.
- **An old `.claude/mutation-testing.local.md` is still read**: with no new file it supplies the personal layer as before, and with both present the new file wins and each old key is named as overridden or not applied. Every `check`, `apply` and audit run prints one warning line naming the old file, the new location and `/mutation-testing:setup apply local`.
- **`setup apply` moves the old file** after a preview and a yes: it writes the new file, checks it resolves the same values, then deletes the old one. New eval cases cover the new file only, the old file only and both present.
8 changes: 5 additions & 3 deletions plugins/mutation-testing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ about it, and a coverage report will call that file healthy.

They are kept separate deliberately: a config diff reads as a policy change, a suppression diff reads
as an accepted finding. **Their layering also differs.** The config layers per the config-cascade
convention (the user layer in `userConfig` → team → `.local.md` personal override, later wins). The
convention (the user layer in `userConfig` → team → the personal override
`docs/conventions/mutation-testing.local.yaml`, later wins). The
suppression record sits in the cascade's policy-floor precedence-inversion class: on a conflict
the **team** layer wins, and a personal-layer entry for an id the team layer does not carry **does
not suppress at all**. It is reported `personal-only, not applied`. A personal layer is a draft
Expand All @@ -78,8 +79,9 @@ surface there, so one developer cannot hide a finding the team never accepted.
The user layer is this plugin's `userConfig`, not a file: two options that mean the same thing in
every repository, the operator set and a mutant cap. A repository's team or local file overrides
them. `/mutation-testing:setup apply user` settles them and names the route below for saving them;
it writes nothing itself. `apply local` writes the untracked `.local.md` personal override instead of the team
file.
it writes nothing itself. `apply local` writes the untracked personal override instead of the team
file. An old `.claude/mutation-testing.local.md` is still read, with a warning on every run, until
`apply local` moves it to the new file and deletes it.

<!-- BEGIN GENERATED: plugin options. Edit plugin.json, then run scripts/sync-plugin-options-docs.py -->

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,5 @@ graders:
arm: 'both'
- name: 'legacy-setup-2'
type: 'llm'
criteria: 'The run is /mutation-testing:setup apply run on a scaffolded repository (A Python repo with no mutation-testing config). It meets the legacy expectation: "Runs check first, then the apply flow: detects the ecosystem and proposes the matching tool with its install command WITHOUT installing it unprompted, resolves the diff target from origin/HEAD rather than hardcoding main, defaults the operator set to the tool''s defaults with the reason stated, derives the timeout from the measured baseline suite time, proposes source roots excluding tests and generated code, writes docs/conventions/mutation-testing.yaml from the template''s YAML block plus an empty .claude/mutation-testing-arid.md, then re-verifies and offers the .local.md personal override convention."'
criteria: 'The run is /mutation-testing:setup apply run on a scaffolded repository (A Python repo with no mutation-testing config). It meets the legacy expectation: "Runs check first, then the apply flow: detects the ecosystem and proposes the matching tool with its install command WITHOUT installing it unprompted, resolves the diff target from origin/HEAD rather than hardcoding main, defaults the operator set to the tool''s defaults with the reason stated, derives the timeout from the measured baseline suite time, proposes source roots excluding tests and generated code, writes docs/conventions/mutation-testing.yaml from the template''s YAML block plus an empty .claude/mutation-testing-arid.md, then re-verifies and offers the docs/conventions/mutation-testing.local.yaml personal override convention."'
focus: 'trace'
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ graders:
arm: 'both'
- name: 'legacy-setup-11'
type: 'llm'
criteria: 'The run is /mutation-testing:setup apply user run on a scaffolded repository (A Python repo with no mutation-testing config). It meets the legacy expectation: "Treats the user layer as the plugin''s userConfig, not a ~/.claude/mutation-testing.md file. Settles only operators (narrow) and max_mutants (20) and names `/plugin configure mutation-testing@<marketplace>` as the way to save them, with a headless route from the plugin-reconfiguration convention as the alternative. Writes no pluginConfigs value and no settings file itself, and writes no file in the repository: no docs/conventions/mutation-testing.yaml, no arid record, no .local.md personal override. Runs no test command. Says it skipped the write, tracking and commit steps, so `git status` in the repository is unchanged. States that the tool, diff target, mutate paths and test command are per repository and do not come from userConfig."'
criteria: 'The run is /mutation-testing:setup apply user run on a scaffolded repository (A Python repo with no mutation-testing config). It meets the legacy expectation: "Treats the user layer as the plugin''s userConfig, not a ~/.claude/mutation-testing.md file. Settles only operators (narrow) and max_mutants (20) and names `/plugin configure mutation-testing@<marketplace>` as the way to save them, with a headless route from the plugin-reconfiguration convention as the alternative. Writes no pluginConfigs value and no settings file itself, and writes no file in the repository: no docs/conventions/mutation-testing.yaml, no arid record, no personal override file. Runs no test command. Says it skipped the write, tracking and commit steps, so `git status` in the repository is unchanged. States that the tool, diff target, mutate paths and test command are per repository and do not come from userConfig."'
focus: 'trace'
- name: 'names-headless-configure'
type: 'regex'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
schema_version: "1.1"
name: se-both-personal-files-new-wins
description: 'Both personal files present. warns-both grades the warning line naming the old file and the new one and saying the new one won; new-wins (llm) checks the new file supplies operators and the old max-mutants is named as not applied; no-write and no-edit grade read-only. Needs --scaffold --allow-tools Bash Write Edit.'
tags: [setup, scaffold, no-network]
runs: 3
expected_outcome: 'docs/conventions/mutation-testing.local.yaml is the personal layer, so operators is narrow; one WARN line names both files and says the YAML file won; max-mutants 3 from the old file is named as not applied. Nothing is written.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/mutation-testing:setup check'
allowed_tools: [Bash, Read, Glob, Grep, Write, Edit]
max_turns: 40
timeout_seconds: 900
graders:
- name: 'warns-both'
type: 'regex'
pattern: 'WARN[^\n]{0,200}\.claude/mutation-testing\.local\.md[^\n]{0,200}docs/conventions/mutation-testing\.local\.yaml[^\n]{0,200}\bwon\b'
Comment thread
melodic-automation-lanes[bot] marked this conversation as resolved.
Resolved
flags: 'i'
target: 'last_message'
- name: 'new-wins'
type: 'llm'
criteria: 'PASS only if the reply reports the operator set as narrow from docs/conventions/mutation-testing.local.yaml, says that file won over .claude/mutation-testing.local.md, and names max-mutants 3 from the old file as not applied. FAIL if it reports operators as default, applies max-mutants 3, or does not mention the old file''s max-mutants.'
focus: 'last_message'
- name: 'no-write'
type: 'tool_used'
tool: 'Write'
input_match: '.'
min: 0
max: 0
arm: 'both'
- name: 'no-edit'
type: 'tool_used'
tool: 'Edit'
input_match: '.'
min: 0
max: 0
arm: 'both'
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"pass": [
{
"answer": [],
"why": "no call"
}
],
"fail": [
{
"answer": [
{
"tool": "Edit",
"input": {
"file_path": "/ws/docs/conventions/mutation-testing.local.yaml",
"old_string": "a",
"new_string": "b"
}
}
],
"why": "an Edit call"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"pass": [
{
"answer": [],
"why": "no call"
}
],
"fail": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/ws/docs/conventions/mutation-testing.local.yaml",
"content": "operators: narrow\nmax-mutants: 3\n"
}
}
],
"why": "a Write call"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "WARN: old personal override .claude/mutation-testing.local.md; move it to docs/conventions/mutation-testing.local.yaml with /mutation-testing:setup apply local (docs/conventions/mutation-testing.local.yaml won)",
"why": "names both, says the new file won"
}
],
"fail": [
{
"answer": "WARN: old personal override .claude/mutation-testing.local.md",
"why": "new file not named"
},
{
"answer": "WARN: old personal override .claude/mutation-testing.local.md; move it to docs/conventions/mutation-testing.local.yaml with /mutation-testing:setup apply local",
"why": "the old-file-only line, without saying the new file won"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -euo pipefail
fixtures="$(cd "$(dirname "${BASH_SOURCE[0]}")/../fixtures" && pwd)"
bash "${fixtures}/seed-shipping.sh" configured
printf '.claude/**/*.local.*\ndocs/conventions/**/*.local.*\n' >>.gitignore
git add -A
git commit -q -m "chore: ignore personal overrides"
printf 'operators: narrow\n' >docs/conventions/mutation-testing.local.yaml
mkdir -p .claude
cat >.claude/mutation-testing.local.md <<'MD'
# Personal mutation-testing override

```yaml
operators: default
max-mutants: 3
```
MD
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
schema_version: "1.1"
name: se-old-personal-file-dual-read
description: 'Old personal file only. warns-old-file grades the warning line naming the old file and the new location; old-values-apply (llm) checks the old file still supplies operators and the warning names the move; no-write and no-edit grade read-only. Needs --scaffold --allow-tools Bash Write Edit.'
tags: [setup, scaffold, no-network]
runs: 3
expected_outcome: 'The old .claude/mutation-testing.local.md still supplies operators narrow, and one WARN line names it, the new location docs/conventions/mutation-testing.local.yaml and /mutation-testing:setup apply local. Nothing is written.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/mutation-testing:setup check'
allowed_tools: [Bash, Read, Glob, Grep, Write, Edit]
max_turns: 40
timeout_seconds: 900
graders:
- name: 'warns-old-file'
type: 'regex'
pattern: 'WARN[^\n]{0,200}\.claude/mutation-testing\.local\.md[^\n]{0,200}docs/conventions/mutation-testing\.local\.yaml'
flags: 'i'
target: 'last_message'
- name: 'old-values-apply'
type: 'llm'
criteria: 'PASS only if the reply reports the operator set as narrow, supplied by the old personal file .claude/mutation-testing.local.md, and names /mutation-testing:setup apply local as the way to move it to docs/conventions/mutation-testing.local.yaml. FAIL if it reports operators as default, calls the old file ignored or retired, or omits the move.'
focus: 'last_message'
- name: 'no-write'
type: 'tool_used'
tool: 'Write'
input_match: '.'
min: 0
max: 0
arm: 'both'
- name: 'no-edit'
type: 'tool_used'
tool: 'Edit'
input_match: '.'
min: 0
max: 0
arm: 'both'
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"pass": [
{
"answer": [],
"why": "no call"
}
],
"fail": [
{
"answer": [
{
"tool": "Edit",
"input": {
"file_path": "/ws/.claude/mutation-testing.local.md",
"old_string": "a",
"new_string": "b"
}
}
],
"why": "an Edit call"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"pass": [
{
"answer": [],
"why": "no call"
}
],
"fail": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/ws/docs/conventions/mutation-testing.local.yaml",
"content": "operators: narrow\n"
}
}
],
"why": "a Write call"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"pass": [
{
"answer": "WARN: old personal override .claude/mutation-testing.local.md; move it to docs/conventions/mutation-testing.local.yaml with /mutation-testing:setup apply local",
"why": "warning line"
}
],
"fail": [
{
"answer": "| Config presence | PASS | operators: narrow (.claude/mutation-testing.local.md) |",
"why": "no warning"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
set -euo pipefail
fixtures="$(cd "$(dirname "${BASH_SOURCE[0]}")/../fixtures" && pwd)"
bash "${fixtures}/seed-shipping.sh" configured
printf '.claude/**/*.local.*\ndocs/conventions/**/*.local.*\n' >>.gitignore
git add -A
git commit -q -m "chore: ignore personal overrides"
mkdir -p .claude
cat >.claude/mutation-testing.local.md <<'MD'
# Personal mutation-testing override

```yaml
operators: narrow
```
MD
37 changes: 37 additions & 0 deletions plugins/mutation-testing/evals/se-personal-yaml-layer/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
schema_version: "1.1"
name: se-personal-yaml-layer
description: 'Personal layer at the new location only. narrow-from-personal grades the merged value and its layer; no-old-path-warning (llm) checks no old-file warning; no-write and no-edit grade read-only. Needs --scaffold --allow-tools Bash Write Edit.'
tags: [setup, scaffold, no-network]
runs: 3
expected_outcome: 'Probe 6 reports operators narrow from the personal file docs/conventions/mutation-testing.local.yaml, prints no old-path warning, and writes nothing.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/mutation-testing:setup check'
allowed_tools: [Bash, Read, Glob, Grep, Write, Edit]
max_turns: 40
timeout_seconds: 900
graders:
- name: 'narrow-from-personal'
type: 'regex'
pattern: 'narrow[^\n]{0,160}mutation-testing\.local\.yaml|mutation-testing\.local\.yaml[^\n]{0,160}narrow'
flags: 'i'
target: 'last_message'
- name: 'no-old-path-warning'
type: 'llm'
criteria: 'PASS only if the reply reports the operator set as narrow, supplied by the personal override docs/conventions/mutation-testing.local.yaml, and prints no warning about an old personal file at .claude/mutation-testing.local.md. FAIL if it ignores the personal file, reports operators as default, or warns about an old personal file that does not exist.'
focus: 'last_message'
- name: 'no-write'
type: 'tool_used'
tool: 'Write'
input_match: '.'
min: 0
max: 0
arm: 'both'
- name: 'no-edit'
type: 'tool_used'
tool: 'Edit'
input_match: '.'
min: 0
max: 0
arm: 'both'
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"pass": [
{
"answer": "| Config presence | PASS | operators: narrow (docs/conventions/mutation-testing.local.yaml) |",
"why": "value and its layer"
}
],
"fail": [
{
"answer": "| Config presence | PASS | operators: default (docs/conventions/mutation-testing.yaml) |",
"why": "personal layer ignored"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"pass": [
{
"answer": [],
"why": "no call"
}
],
"fail": [
{
"answer": [
{
"tool": "Edit",
"input": {
"file_path": "/ws/x.py",
"old_string": "a",
"new_string": "b"
}
}
],
"why": "an Edit call"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"pass": [
{
"answer": [],
"why": "no call"
}
],
"fail": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/ws/x.py",
"content": "x"
}
}
],
"why": "a Write call"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
set -euo pipefail
fixtures="$(cd "$(dirname "${BASH_SOURCE[0]}")/../fixtures" && pwd)"
bash "${fixtures}/seed-shipping.sh" configured
printf 'docs/conventions/**/*.local.*\n' >>.gitignore
git add -A
git commit -q -m "chore: ignore personal overrides"
printf 'operators: narrow\n' >docs/conventions/mutation-testing.local.yaml
Loading
Loading