Skip to content

Say in the property guidance that it covers every target and that fuzzing follows it #15825

Closed

Description

@kyle-sexton

Parent

Refs #15655 and #15649 (the map decisions on the talk's fuzzing and release pass, where the owner decided this).

Depends on

Outcome

/testing:write's property guidance says plainly that it covers every target and that fuzzing is the same mechanism, so nobody builds or looks for a separate fuzzing or UI-property skill.

Demo: ask /testing:write for a property test of a running UI, or for a fuzz test of a parser; it follows the one property guidance, finds the consumer's own tool, and names none itself.

Source: the talk "All Code Is Slop (Except Mine)" (Lauren Tan), slide 28 and the transcript at 1:12:39: "property-based testing as well for UIs". The tool she shows describes itself as "property-based testing, or fuzzing if you will".

Decisions

  • Owner, 2026-11-12: property testing is one thing with one home and is not split by target ("I think we just consolidate property testing and then split it based on the target"); flood-style fuzzing belongs with it.
  • A fuzz test is generated input with the weak rule "does not crash"; a property test is generated input with a stated rule. Same mechanism, same guidance.
  • Generated user actions against a running UI are a generated call sequence on a stateful thing, which the guidance already covers.
  • No tool is named or shipped; the tool comes from the consumer's repository.
  • This ships separately from the pilot cut so that cut's before-and-after result is not confounded by added text.

Key interfaces

None.

Done when

  • The property guidance states, in one or two sentences, that it applies to every target (a running UI included) and that a fuzz test follows it; it names no tool.
  • One new eval case, frozen first, covers the sentences; the skill's existing write cases are non-inferior on a two-version compare (opus and sonnet aliases, three runs per side, --case filter, a --max-cost-usd ceiling, no new batch after a usage-limit error), run in the one local eval-runner session; the pull request carries do-not-merge until the record lands.
  • The pull request body names this repository's dogfooding step.

Out of scope

  • Any per-language or per-tool page.
  • The release pass.

Activity

  1. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 13, 2026
  2. added theissue type on Nov 13, 2026
  3. removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 13, 2026
  4. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: add one or two sentences to the /testing:write property guidance saying it applies to every target (a running UI included) and that a fuzz test follows it. The sentences name no tool. The change also needs one new frozen eval case and a non-inferiority compare on the existing write cases, and the pull request body names the dogfooding step.

    Automated triage. A maintainer reviews it.

  5. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: merged pull request #16464 names it with Refs: after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is **Decision recorded.** saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"a522e6a34f2e287f8ac73c60603a525bca8dfaa52aa4c35d400932f270c43b36","body_sha256":"2e81b8949b5679bf96df16984207b862439f7aad32d4f2058e3bb8b2fc874ea9"}

  7. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor
    No description provided.
  8. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Pull request #16464 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

  9. github-actions commented on Nov 16, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).

  10. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 18, 2026
  11. kyle-sexton commented on Nov 18, 2026

    @kyle-sexton
    MemberAuthor

    Targeted eval result: PASS

    One targeted case per changed skill, run on the old text (before) and the new text (after), on sonnet at 3 runs a side, Claude Code 2.1.327 (Claude Code).

    What this gate checks: that the skill's existing write cases and the new case do no worse on the new text than on the old (the issue's non-inferiority compare). The issue states no margin, so the rule registered before any run is the per-case no-loss rule below (PROVISIONAL until the owner confirms it). There is no wrong-text side, so a PASS says these cases did not get worse, not that they would have noticed a bad edit.

    Scope: not a cut. PR 16464 adds two sentences to context/property.md (the guidance covers every target, a running UI included, and a fuzz test follows it) and widens the SKILL.md Step 0 route row to fuzz tests and any target. Measured: all 22 cases whose names start wr at the merge commit (21 existing write cases plus the new wr-fuzz-follows-property), non-inferiority; the SKILL.md row is loaded with the skill, and the property.md row counts as measured only if opened in at least 2 of 3 runs a side. Unmeasured: the running-UI clause (no case has a running-UI fixture, as the PR body says).

    • Cases: every plugin-eval case whose name starts wr at the merge commit (wr-* and write-*, 22), copied byte-identical under their own names; wr-parallel-worker-isolation, added to main after the merge, is not in the set.

    • No target is registered for the new case: the issue says it 'covers the sentences', not that it must show a change; its per-grader before and after counts are in the table.

    • Opus not run. The issue names the opus and sonnet aliases; the evals plugin's reading guide says a skill-text change is compared on Sonnet and Opus runs only to confirm a loss. The issue's two-alias clause is not met by this record.

    • Result rule PROVISIONAL: the issue states no margin. Registered: the per-case no-loss rule (a case's mean score falls by 0.15 or more, or any grader passes in 2 or more fewer after runs). Not registered: the reading guide's few-cases rule (noise-report margin 0.05; at about 20 cases or more a case-drop line or a not-shown-non-inferior verdict is a LOSS reading).

    • 'Frozen first' is not met: the new case and the text change landed in one commit, as the PR body says.

    • Unmeasured rows: the running-UI clause of property.md.

    • Before: 8a544e1bd53f57302bd306803a68ea4ef3777cad (the merge commit's first parent)

    • After: 109b87a8f2b44bc108ab26e2796a13615e365100 (the merge commit on main; the pull request merged before this eval, merge commit read when this was written: 109b87a8f2b44bc108ab26e2796a13615e365100)

    • Loss: the mean case score falls by 0.15 or more, or any one grader passes in 2 or more fewer after runs than before runs.

    Passing runs per grader, per side:

    Case Grader Before After
    wr-fuzz-follows-property skill-fired 3/3 3/3
    wr-fuzz-follows-property read-property 3/3 3/3
    wr-fuzz-follows-property uses-project-library 3/3 3/3
    wr-fuzz-follows-property no-separate-fuzzer 3/3 3/3
    wr-fuzz-follows-property states-round-trip 3/3 2/3
    wr-fuzz-follows-property mean case score 1.000 0.933
    wr-glossary-absent skill-fired 3/3 3/3
    wr-glossary-absent canonical-term 3/3 2/3
    wr-glossary-absent no-rejected-term 0/3 0/3
    wr-glossary-absent mean case score 0.667 0.556
    wr-glossary-docs-path skill-fired 3/3 3/3
    wr-glossary-docs-path canonical-term 3/3 3/3
    wr-glossary-docs-path no-rejected-term 3/3 3/3
    wr-glossary-docs-path glossary-read 3/3 3/3
    wr-glossary-docs-path mean case score 1.000 1.000
    wr-glossary-root skill-fired 3/3 3/3
    wr-glossary-root canonical-term 3/3 3/3
    wr-glossary-root no-rejected-term 3/3 3/3
    wr-glossary-root glossary-read 3/3 3/3
    wr-glossary-root mean case score 1.000 1.000
    wr-go-uses-project-library skill-fired 3/3 3/3
    wr-go-uses-project-library read-property 3/3 3/3
    wr-go-uses-project-library uses-rapid 3/3 3/3
    wr-go-uses-project-library meets-expectations 1/3 2/3
    wr-go-uses-project-library mean case score 0.833 0.917
    wr-handler-unit-tests skill-fired 3/3 3/3
    wr-handler-unit-tests read-write-md 3/3 3/3
    wr-handler-unit-tests colocated-file 3/3 3/3
    wr-handler-unit-tests documented-names 3/3 3/3
    wr-handler-unit-tests plain-asserts 3/3 3/3
    wr-handler-unit-tests no-tests-dir 3/3 3/3
    wr-handler-unit-tests meets-expected-output 0/3 0/3
    wr-handler-unit-tests mean case score 0.857 0.857
    wr-hollow-out-receipt skill-fired 3/3 3/3
    wr-hollow-out-receipt asserts-on-receipt 3/3 3/3
    wr-hollow-out-receipt meets-expectations 3/3 3/3
    wr-hollow-out-receipt mean case score 1.000 1.000
    wr-integration-shared-library skill-fired 3/3 3/3
    wr-integration-shared-library read-organize 3/3 3/3
    wr-integration-shared-library read-convention 3/3 3/3
    wr-integration-shared-library names-documented-location 3/3 3/3
    wr-integration-shared-library meets-expected-output 0/3 0/3
    wr-integration-shared-library mean case score 0.800 0.800
    wr-light-skips-advisor-checkpoint reads-level 3/3 3/3
    wr-light-skips-advisor-checkpoint writes-skip-token 3/3 3/3
    wr-light-skips-advisor-checkpoint mean case score 1.000 1.000
    wr-no-skip-red skill-fired 3/3 3/3
    wr-no-skip-red test-before-fix 0/3 0/3
    wr-no-skip-red separator-test 3/3 3/3
    wr-no-skip-red meets-expectations 0/3 0/3
    wr-no-skip-red mean case score 0.500 0.500
    wr-placement-from-conventions documented-path 3/3 3/3
    wr-placement-from-conventions read-conventions 3/3 3/3
    wr-placement-from-conventions mean case score 1.000 1.000
    wr-property-dependent-input skill-fired 3/3 3/3
    wr-property-dependent-input uses-given 3/3 3/3
    wr-property-dependent-input builds-dependent-input 3/3 3/3
    wr-property-dependent-input no-discarded-draws 3/3 3/3
    wr-property-dependent-input mean case score 1.000 1.000
    wr-property-past-bug skill-fired 3/3 3/3
    wr-property-past-bug uses-given 3/3 3/3
    wr-property-past-bug pins-past-bug 3/3 3/3
    wr-property-past-bug mean case score 1.000 1.000
    wr-property-reimplements-split skill-fired 3/3 3/3
    wr-property-reimplements-split points-to-test-value 0/3 1/3
    wr-property-reimplements-split sum-invariant 2/3 3/3
    wr-property-reimplements-split meets-expectations 0/3 1/3
    wr-property-reimplements-split mean case score 0.417 0.667
    wr-python-idempotence-property skill-fired 3/3 3/3
    wr-python-idempotence-property uses-given 3/3 3/3
    wr-python-idempotence-property idempotence 3/3 3/3
    wr-python-idempotence-property meets-expectations 1/3 1/3
    wr-python-idempotence-property mean case score 0.833 0.833
    wr-round-trip-property skill-fired 3/3 3/3
    wr-round-trip-property read-property 3/3 3/3
    wr-round-trip-property round-trip 3/3 3/3
    wr-round-trip-property literal-example 3/3 3/3
    wr-round-trip-property no-recomputed-expected 3/3 3/3
    wr-round-trip-property meets-expectations 0/3 0/3
    wr-round-trip-property mean case score 0.833 0.833
    wr-ts-async-property-marked skill-fired 3/3 3/3
    wr-ts-async-property-marked property-marker 3/3 3/3
    wr-ts-async-property-marked awaits-async-property 3/3 3/3
    wr-ts-async-property-marked mean case score 1.000 1.000
    write-first-unit flow-test-written 3/3 3/3
    write-first-unit resolver-ran 3/3 3/3
    write-first-unit settings-line 3/3 3/3
    write-first-unit skill-fired 3/3 3/3
    write-first-unit unit-test-first 3/3 2/3
    write-first-unit mean case score 1.000 0.933
    write-first-user-flow flow-test-first 3/3 3/3
    write-first-user-flow resolver-ran 3/3 3/3
    write-first-user-flow settings-line 3/3 3/3
    write-first-user-flow skill-fired 3/3 3/3
    write-first-user-flow mean case score 1.000 1.000
    write-refactor-checkpoint checkpoint-reported 3/3 3/3
    write-refactor-checkpoint resolver-ran 3/3 3/3
    write-refactor-checkpoint settings-line 3/3 3/3
    write-refactor-checkpoint skill-fired 3/3 3/3
    write-refactor-checkpoint mean case score 1.000 1.000
    write-refactor-off no-checkpoint 3/3 3/3
    write-refactor-off resolver-ran 3/3 3/3
    write-refactor-off settings-line 3/3 3/3
    write-refactor-off skill-fired 3/3 3/3
    write-refactor-off mean case score 1.000 1.000
    write-refactor-on checkpoint-reported 3/3 3/3
    write-refactor-on resolver-ran 3/3 3/3
    write-refactor-on settings-line 3/3 3/3
    write-refactor-on skill-fired 3/3 3/3
    write-refactor-on mean case score 1.000 1.000

    Graders that cannot show a loss, because they failed on the old text in every run:

    • wr-glossary-absent no-rejected-term: cannot show a loss: failed on the old text in every run (0/3)
    • wr-handler-unit-tests meets-expected-output: cannot show a loss: failed on the old text in every run (0/3)
    • wr-integration-shared-library meets-expected-output: cannot show a loss: failed on the old text in every run (0/3)
    • wr-no-skip-red test-before-fix: cannot show a loss: failed on the old text in every run (0/3)
    • wr-no-skip-red meets-expectations: cannot show a loss: failed on the old text in every run (0/3)
    • wr-property-reimplements-split points-to-test-value: cannot show a loss: failed on the old text in every run (0/3)
    • wr-property-reimplements-split meets-expectations: cannot show a loss: failed on the old text in every run (0/3)
    • wr-round-trip-property meets-expectations: cannot show a loss: failed on the old text in every run (0/3)

    Changed files the compare runs opened, per case and side:

    • wr-fuzz-follows-property before: opened skills/write/context/property.md in 3 of 3 runs
    • wr-fuzz-follows-property after: opened skills/write/context/property.md in 3 of 3 runs
    • wr-glossary-absent before: opened no changed file in 3 runs
    • wr-glossary-absent after: opened no changed file in 3 runs
    • wr-glossary-docs-path before: opened no changed file in 3 runs
    • wr-glossary-docs-path after: opened no changed file in 3 runs
    • wr-glossary-root before: opened no changed file in 3 runs
    • wr-glossary-root after: opened no changed file in 3 runs
    • wr-go-uses-project-library before: opened skills/write/context/property.md in 3 of 3 runs
    • wr-go-uses-project-library after: opened skills/write/context/property.md in 3 of 3 runs
    • wr-handler-unit-tests before: opened no changed file in 3 runs
    • wr-handler-unit-tests after: opened no changed file in 3 runs
    • wr-hollow-out-receipt before: opened no changed file in 3 runs
    • wr-hollow-out-receipt after: opened no changed file in 3 runs
    • wr-integration-shared-library before: opened no changed file in 3 runs
    • wr-integration-shared-library after: opened no changed file in 3 runs
    • wr-light-skips-advisor-checkpoint before: opened no changed file in 3 runs
    • wr-light-skips-advisor-checkpoint after: opened no changed file in 3 runs
    • wr-no-skip-red before: opened no changed file in 3 runs
    • wr-no-skip-red after: opened no changed file in 3 runs
    • wr-placement-from-conventions before: opened no changed file in 3 runs
    • wr-placement-from-conventions after: opened no changed file in 3 runs
    • wr-property-dependent-input before: opened skills/write/context/property.md in 3 of 3 runs
    • wr-property-dependent-input after: opened skills/write/context/property.md in 3 of 3 runs
    • wr-property-past-bug before: opened skills/write/context/property.md in 3 of 3 runs
    • wr-property-past-bug after: opened skills/write/context/property.md in 3 of 3 runs
    • wr-property-reimplements-split before: opened skills/write/context/property.md in 2 of 3 runs
    • wr-property-reimplements-split after: opened skills/write/context/property.md in 2 of 3 runs
    • wr-python-idempotence-property before: opened skills/write/context/property.md in 3 of 3 runs
    • wr-python-idempotence-property after: opened skills/write/context/property.md in 3 of 3 runs
    • wr-round-trip-property before: opened skills/write/context/property.md in 3 of 3 runs
    • wr-round-trip-property after: opened skills/write/context/property.md in 3 of 3 runs
    • wr-ts-async-property-marked before: opened skills/write/context/property.md in 3 of 3 runs
    • wr-ts-async-property-marked after: opened skills/write/context/property.md in 3 of 3 runs
    • write-first-unit before: opened no changed file in 3 runs
    • write-first-unit after: opened no changed file in 3 runs
    • write-first-user-flow before: opened no changed file in 3 runs
    • write-first-user-flow after: opened no changed file in 3 runs
    • write-refactor-checkpoint before: opened no changed file in 3 runs
    • write-refactor-checkpoint after: opened no changed file in 3 runs
    • write-refactor-off before: opened no changed file in 3 runs
    • write-refactor-off after: opened no changed file in 3 runs
    • write-refactor-on before: opened no changed file in 3 runs
    • write-refactor-on after: opened no changed file in 3 runs

    Rows these runs measured (the row's file opened in at least 2 of 3 runs on each side, or loaded with the skill or agent):

    • SKILL-route-row-fuzz (skills/write/SKILL.md, case wr-fuzz-follows-property): measured; the file is loaded with the skill or agent, never opened, so no open count applies
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-fuzz-follows-property): opened in 3 of 3 before, 3 of 3 after runs
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-go-uses-project-library): opened in 3 of 3 before, 3 of 3 after runs
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-property-dependent-input): opened in 3 of 3 before, 3 of 3 after runs
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-property-past-bug): opened in 3 of 3 before, 3 of 3 after runs
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-property-reimplements-split): opened in 2 of 3 before, 2 of 3 after runs
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-python-idempotence-property): opened in 3 of 3 before, 3 of 3 after runs
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-round-trip-property): opened in 3 of 3 before, 3 of 3 after runs
    • PROP-every-target-and-fuzz (skills/write/context/property.md, case wr-ts-async-property-marked): opened in 3 of 3 before, 3 of 3 after runs

    Rows these runs did not measure:

    • none

    Verdicts:

    • verdict wr-fuzz-follows-property (wr-fuzz-follows-property): PASS (no loss: after 0.933, before 1.000)
    • verdict wr-glossary-absent (wr-glossary-absent): PASS (no loss: after 0.556, before 0.667)
    • verdict wr-glossary-docs-path (wr-glossary-docs-path): PASS (no loss: after 1.000, before 1.000)
    • verdict wr-glossary-root (wr-glossary-root): PASS (no loss: after 1.000, before 1.000)
    • verdict wr-go-uses-project-library (wr-go-uses-project-library): PASS (no loss: after 0.917, before 0.833)
    • verdict wr-handler-unit-tests (wr-handler-unit-tests): PASS (no loss: after 0.857, before 0.857)
    • verdict wr-hollow-out-receipt (wr-hollow-out-receipt): PASS (no loss: after 1.000, before 1.000)
    • verdict wr-integration-shared-library (wr-integration-shared-library): PASS (no loss: after 0.800, before 0.800)
    • verdict wr-light-skips-advisor-checkpoint (wr-light-skips-advisor-checkpoint): PASS (no loss: after 1.000, before 1.000)
    • verdict wr-no-skip-red (wr-no-skip-red): PASS (no loss: after 0.500, before 0.500)
    • verdict wr-placement-from-conventions (wr-placement-from-conventions): PASS (no loss: after 1.000, before 1.000)
    • verdict wr-property-dependent-input (wr-property-dependent-input): PASS (no loss: after 1.000, before 1.000)
    • verdict wr-property-past-bug (wr-property-past-bug): PASS (no loss: after 1.000, before 1.000)
    • verdict wr-property-reimplements-split (wr-property-reimplements-split): PASS (no loss: after 0.667, before 0.417)
    • verdict wr-python-idempotence-property (wr-python-idempotence-property): PASS (no loss: after 0.833, before 0.833)
    • verdict wr-round-trip-property (wr-round-trip-property): PASS (no loss: after 0.833, before 0.833)
    • verdict wr-ts-async-property-marked (wr-ts-async-property-marked): PASS (no loss: after 1.000, before 1.000)
    • verdict write-first-unit (write-first-unit): PASS (no loss: after 0.933, before 1.000)
    • verdict write-first-user-flow (write-first-user-flow): PASS (no loss: after 1.000, before 1.000)
    • verdict write-refactor-checkpoint (write-refactor-checkpoint): PASS (no loss: after 1.000, before 1.000)
    • verdict write-refactor-off (write-refactor-off): PASS (no loss: after 1.000, before 1.000)
    • verdict write-refactor-on (write-refactor-on): PASS (no loss: after 1.000, before 1.000)

    Cost: 31.45 USD (smoke and compare, the runs' costUsd).

    The cases live outside the repository; their full text:

    The 22 cases: prompt.md and case.yaml

    wr-fuzz-follows-property

    case.yaml:

    schema_version: "1.1"
    name: wr-fuzz-follows-property
    description: 'Fuzz follows the property guidance. A fuzz test asked for a Python duration parser routes to context/property.md, uses the property library the project already lists (Hypothesis in the test extras) rather than a separate fuzzing tool, and states a rule stronger than "does not crash" (the round trip through format_duration). skill-fired and read-property read the tool calls; uses-project-library, no-separate-fuzzer and states-round-trip read tests/test_duration_fuzz.py. Needs --scaffold --allow-tools Write Edit.'
    tags: [testing, write, property, fuzz, project-library, scaffold, no-network]
    expected_outcome: 'Reads context/property.md; tests/test_duration_fuzz.py imports hypothesis, not atheris, and checks that parse_duration reads format_duration''s output back to the same seconds.'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Write a fuzz test for parse_duration in src/duration.py and put it in tests/test_duration_fuzz.py. Dependencies are not installed in this checkout, so write the file and do not run it. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: read-property
        type: tool_used
        tool: Read
        input_match: '"file_path"\s*:\s*"[^"]*skills/write/context/property\.md"'
      - name: uses-project-library
        type: regex
        pattern: '^\s*(?:from\s+hypothesis\b|import\s+hypothesis\b)'
        flags: m
        target: { source: file, path: tests/test_duration_fuzz.py }
      - name: no-separate-fuzzer
        type: regex
        pattern: '\batheris\b|\bpythonfuzz\b'
        match: not_contains
        target: { source: file, path: tests/test_duration_fuzz.py }
      - name: states-round-trip
        type: regex
        pattern: '\bparse_duration\(\s*format_duration\(|=\s*format_duration\([\s\S]*?\bparse_duration\('
        target: { source: file, path: tests/test_duration_fuzz.py }

    wr-glossary-absent

    prompt.md:

    This is the catalogue code for our tool library: `Library` in `tool_library.py`, with its tests in
    `tests/test_tool_library.py`. Run the tests with `python3 -m unittest discover -s tests -t .`. Use
    the `testing:write` skill if you have it.
    
    Members want to check tools out and bring them back. Add that to `Library`, tests first, with the
    tests in `tests/test_tool_library.py`:
    
    1. Checking out an available tool for a member records who has it and a due date 14 days after the
       day it went out.
    2. A tool that is already out cannot go out again: raise `ValueError`.
    3. Bringing a tool back makes it available again.
    4. An unknown card number cannot take a tool out: raise `KeyError`.
    
    The work is done when each point has a test and the whole suite passes.

    case.yaml:

    schema_version: "1.1"
    name: wr-glossary-absent
    context:
      scaffold_script: scaffold.sh
    description: "Glossary reader, control. The same workspace and prompt as wr-glossary-root with no glossary anywhere, so canonical-term and no-rejected-term record what the run names the feature on the user's words alone. It has no glossary-read grader: there is no file to read. Read its pass rates beside wr-glossary-root's: the cross-case gap is the measurement, since the CLI's delta ablates the plugin, not the glossary. Expected to fail canonical-term and no-rejected-term in most runs, because the request says 'check out' throughout and nothing in the workspace says Loan; a high pass rate here means the suite has no headroom to show a glossary's effect. Source: the maintainers' own writing, an owner-approved measurement before the glossary capability is restructured. Routine: a control, not a difficulty probe. Needs --scaffold --allow-tools Bash Edit Write."
    tags: [testing, write, glossary, control, routine]
    runs: 3
    execution:
      max_turns: 60
      timeout_seconds: 900
      allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Adds the feature to Library with unittest tests in tests/test_tool_library.py and runs the suite until it passes. With no glossary, naming it after the request (checkout) is the likely result, and the two term graders record how often it happens"
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: canonical-term
        type: regex
        target:
          source: file
          path: tests/test_tool_library.py
        flags: i
        pattern: 'loan'
      - name: no-rejected-term
        type: regex
        target:
          source: file
          path: tests/test_tool_library.py
        flags: i
        match: not_contains
        pattern: 'check(?:ed|s|ing)?[ _-]?outs?|rental|booking'

    wr-glossary-docs-path

    prompt.md:

    This is the catalogue code for our tool library: `Library` in `tool_library.py`, with its tests in
    `tests/test_tool_library.py`. Run the tests with `python3 -m unittest discover -s tests -t .`. Use
    the `testing:write` skill if you have it.
    
    Members want to check tools out and bring them back. Add that to `Library`, tests first, with the
    tests in `tests/test_tool_library.py`:
    
    1. Checking out an available tool for a member records who has it and a due date 14 days after the
       day it went out.
    2. A tool that is already out cannot go out again: raise `ValueError`.
    3. Bringing a tool back makes it available again.
    4. An unknown card number cannot take a tool out: raise `KeyError`.
    
    The work is done when each point has a test and the whole suite passes.

    case.yaml:

    schema_version: "1.1"
    name: wr-glossary-docs-path
    context:
      scaffold_script: scaffold.sh
    description: "Glossary reader, docs placement. The same workspace and prompt as wr-glossary-root with the glossary at docs/glossary.md instead of a root GLOSSARY.md. Graded on whether the run finds and reads the glossary at docs/glossary.md, a sibling of tool_library.py rather than an ancestor on its walk-up path, and the tests use Loan and none of its rejected synonyms. Source: the maintainers' own writing, an owner-approved measurement before the glossary capability is restructured. Hard: the glossary is off the documented read path and the request says 'check out' throughout. Needs --scaffold --allow-tools Bash Edit Write."
    tags: [testing, write, glossary, hard]
    runs: 3
    execution:
      max_turns: 60
      timeout_seconds: 900
      allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Reads docs/glossary.md, then adds the feature to Library under the glossary's term, such as a loan with a due date, with unittest tests in tests/test_tool_library.py that use Loan and none of checkout, rental or booking, and runs the suite until it passes"
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: canonical-term
        type: regex
        target:
          source: file
          path: tests/test_tool_library.py
        flags: i
        pattern: 'loan'
      - name: no-rejected-term
        type: regex
        target:
          source: file
          path: tests/test_tool_library.py
        flags: i
        match: not_contains
        pattern: 'check(?:ed|s|ing)?[ _-]?outs?|rental|booking'
      - name: glossary-read
        type: regex
        target: trace
        pattern: '"name":"Read","input":\{"file_path":"[^"]*docs/glossary\.md"|"name":"Bash","input":\{"command":"(?=(?:[^"\\]|\\.)*\b(?:cat|sed|head|tail|less|grep)\b)(?:[^"\\]|\\.)*docs/(?:glossary(?:\.md|\*)|\*+(?:\.md)?)'

    wr-glossary-root

    prompt.md:

    This is the catalogue code for our tool library: `Library` in `tool_library.py`, with its tests in
    `tests/test_tool_library.py`. Run the tests with `python3 -m unittest discover -s tests -t .`. Use
    the `testing:write` skill if you have it.
    
    Members want to check tools out and bring them back. Add that to `Library`, tests first, with the
    tests in `tests/test_tool_library.py`:
    
    1. Checking out an available tool for a member records who has it and a due date 14 days after the
       day it went out.
    2. A tool that is already out cannot go out again: raise `ValueError`.
    3. Bringing a tool back makes it available again.
    4. An unknown card number cannot take a tool out: raise `KeyError`.
    
    The work is done when each point has a test and the whole suite passes.

    case.yaml:

    schema_version: "1.1"
    name: wr-glossary-root
    context:
      scaffold_script: scaffold.sh
    description: "Glossary reader, root placement. The workspace holds a project glossary at the root, GLOSSARY.md, the place testing:write's walk-up rule finds (skills/write/context/write.md, Sequence step 1). Graded on whether the new tests name the feature with the glossary's canonical term (Loan) instead of its rejected synonyms (checkout, rental, booking), and whether the run read the glossary at all. Compare its pass rates with wr-glossary-absent, the same prompt with no glossary: that cross-case gap, not the CLI's with/without-plugin delta, answers whether a glossary helps a skill that reads it, because the CLI ablates the plugin and never the glossary. Source: the maintainers' own writing, an owner-approved measurement before the glossary capability is restructured. Hard: the request itself says 'check out' throughout, so the canonical name has to come from the glossary over the user's own words. Needs --scaffold --allow-tools Bash Edit Write."
    tags: [testing, write, glossary, hard]
    runs: 3
    execution:
      max_turns: 60
      timeout_seconds: 900
      allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Reads GLOSSARY.md, then adds the feature to Library under the glossary's term, such as a loan with a due date, with unittest tests in tests/test_tool_library.py that use Loan and none of checkout, rental or booking, and runs the suite until it passes"
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: canonical-term
        type: regex
        target:
          source: file
          path: tests/test_tool_library.py
        flags: i
        pattern: 'loan'
      - name: no-rejected-term
        type: regex
        target:
          source: file
          path: tests/test_tool_library.py
        flags: i
        match: not_contains
        pattern: 'check(?:ed|s|ing)?[ _-]?outs?|rental|booking'
      - name: glossary-read
        type: regex
        target: trace
        pattern: '"name":"Read","input":\{"file_path":"[^"]*GLOSSARY\.md"|"name":"Bash","input":\{"command":"(?=(?:[^"\\]|\\.)*\b(?:cat|sed|head|tail|less|grep)\b)(?:[^"\\]|\\.)*(?:GLOSSARY\.md|\s(?:\./)?\*(?:\.md)?(?=[\s"\\;|&)]))'

    wr-go-uses-project-library

    case.yaml:

    schema_version: "1.1"
    name: wr-go-uses-project-library
    description: 'Legacy write 7. A property test for a Go Merge function uses the guidance in context/property.md (commutativity, generators that build valid interval lists) and the property library the module already requires, rapid, rather than one the skill picks. skill-fired and read-property read the tool calls, uses-rapid the go test file; meets-expectations (llm) quotes the legacy expectations. Needs --scaffold --allow-tools Bash Write Edit.'
    tags: [testing, write, property, project-library, scaffold, no-network]
    expected_outcome: 'Reads context/property.md; intervals/merge_test.go imports pgregory.net/rapid and checks Merge(a, b) against Merge(b, a).'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Our Go module has a Merge function for interval lists (intervals/merge.go) that should be commutative. Write a property-based test for it in intervals/merge_test.go. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: read-property
        type: tool_used
        tool: Read
        input_match: '"file_path"\s*:\s*"[^"]*skills/write/context/property\.md"'
      - name: uses-rapid
        type: regex
        pattern: '"pgregory\.net/rapid"'
        target: { source: file, path: intervals/merge_test.go }
      - name: meets-expectations
        type: llm
        criteria: 'The user asked for a property-based test of a Go Merge function for interval lists that should be commutative; go.mod already requires pgregory.net/rapid. PASS only if the run meets every legacy expectation: "Uses the guidance in context/property.md" "Writes the property with the library go.mod already requires" "States commutativity (merging a with b equals merging b with a) as the property". FAIL if any point is missing or contradicted.'

    wr-handler-unit-tests

    case.yaml:

    schema_version: "1.1"
    name: wr-handler-unit-tests
    description: 'Legacy write 1. A new create-template-item command handler in a Python repository whose docs/testing.md co-locates unit tests beside the code and sets a test naming pattern. The skill routes to context/write.md, treats the handler as application logic needing unit tests, lists happy, validation and edge cases, and writes the tests at the documented path with the documented names and plain pytest asserts. skill-fired and read-write-md read the tool calls; colocated-file, documented-names, plain-asserts and no-tests-dir read the files; meets-expected-output (llm) quotes the legacy expected output. Needs --scaffold --allow-tools Bash Write Edit.'
    tags: [testing, write, placement, naming, scaffold, no-network]
    expected_outcome: 'src/templates/create_template_item_test.py holds at least three tests named test_<action>_<outcome>_when_<condition>, using plain assert; no tests/ folder is created.'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'I just wrote a new command handler for creating template items, src/templates/create_template_item.py. Write tests for it.'
      allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: read-write-md
        type: tool_used
        tool: Read
        input_match: '"file_path"\s*:\s*"[^"]*skills/write/context/write\.md"'
      - name: colocated-file
        type: file_exists
        path: src/templates/create_template_item_test.py
      - name: documented-names
        type: regex
        pattern: '(?:[\s\S]*?\bdef test_[a-z0-9]+(?:_[a-z0-9]+)*_when_[a-z0-9_]+\s*\(){3}'
        target: { source: file, path: src/templates/create_template_item_test.py }
      - name: plain-asserts
        type: regex
        pattern: '\bimport unittest\b|\bunittest\.TestCase\b|\bself\.assert[A-Z]'
        match: not_contains
        target: { source: file, path: src/templates/create_template_item_test.py }
      - name: no-tests-dir
        type: file_exists
        path: tests/**/*.py
        exists: false
      - name: meets-expected-output
        type: llm
        criteria: 'The user asked for tests of a new create-template-item command handler in a Python repository whose docs/testing.md sets pytest with plain assert, co-located unit tests and a naming pattern. PASS only if the run matches the legacy expected output: "Routes to context/write.md. Identifies the handler as application-layer logic needing unit tests. Follows TDD: generates a test list (happy path, validation failures, edge cases), names tests using the consuming project''s documented naming pattern (mirroring the ecosystem''s own idiom when undocumented), and places them in the project''s documented co-located test location. Defers to the consuming project''s documented framework and assertion conventions rather than assuming any specific stack." FAIL if any part is missing or contradicted.'

    wr-hollow-out-receipt

    case.yaml:

    schema_version: "1.1"
    name: wr-hollow-out-receipt
    description: 'Legacy write 8. The per-cycle checklist run on a buildReceipt test whose only assertion reads the order the test built: the hollow-out item fails, since a buildReceipt that returns nothing still passes, and the fix captures the receipt buildReceipt returns and asserts on its lines. skill-fired reads the Skill calls, asserts-on-receipt the reply; meets-expectations (llm) quotes the legacy expectations. Read-only tools, graded on the reply.'
    tags: [testing, write, per-cycle-checklist, hollow-out, no-network]
    expected_outcome: 'Says the test passes when buildReceipt returns nothing because it asserts on the order it built, and rewrites it to assert on the receipt buildReceipt returns.'
    execution:
      prompt: "I just went green on this cycle for buildReceipt. Run the per-cycle checklist on the test before I refactor:\n\nimport { buildReceipt } from './receipt';\n\ntest('a receipt lists every order line', () => {\n  const order = { id: 'A17', lines: [{ sku: 'TEA', qty: 2 }, { sku: 'MUG', qty: 1 }] };\n  buildReceipt(order);\n  expect(order.lines).toHaveLength(2);\n});"
      allowed_tools: [Read, Glob, Grep, Skill]
      max_turns: 15
      timeout_seconds: 420
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: asserts-on-receipt
        type: regex
        pattern: '\b(?:const|let)\s+(\w+)\s*=\s*buildReceipt\(\s*order\s*\)[\s\S]{0,400}?\bexpect\(\s*\1\b|\bexpect\(\s*buildReceipt\(\s*order\s*\)'
        target: last_message
      - name: meets-expectations
        type: llm
        criteria: 'The user asked for the per-cycle checklist on a buildReceipt test that calls buildReceipt(order) and then asserts expect(order.lines).toHaveLength(2). PASS only if the reply meets every legacy expectation: "Says the test keeps passing when buildReceipt returns nothing" "Notes that the assertion checks the order the test built rather than any receipt" "Suggests asserting on the receipt buildReceipt returns". FAIL if any point is missing or contradicted.'

    wr-integration-shared-library

    case.yaml:

    schema_version: "1.1"
    name: wr-integration-shared-library
    description: 'Legacy write 2. Where integration tests for a new shared pricing library go, in a monorepo whose docs/testing.md separates single-project from cross-project integration tests. The skill routes to context/organize.md, finds the library has testable behavior (branching discount rules), and lets the documented convention decide the location instead of pre-committing. skill-fired, read-organize and read-convention read the tool calls, names-documented-location the reply; meets-expected-output (llm) quotes the legacy expected output. Read-only tools; needs --scaffold.'
    tags: [testing, write, organize, placement, scaffold, no-network]
    expected_outcome: 'Reads context/organize.md and docs/testing.md, says pricing has logic worth its own tests, and names libs/pricing/tests/integration/ for library-only tests or tests/integration/pricing/ for tests that span the services, per the document.'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Where should I put integration tests for the new shared library in libs/pricing? Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill]
      max_turns: 15
      timeout_seconds: 420
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: read-organize
        type: tool_used
        tool: Read
        input_match: '"file_path"\s*:\s*"[^"]*skills/write/context/organize\.md"'
      - name: read-convention
        type: tool_used
        tool: Read
        input_match: '"file_path"\s*:\s*"[^"]*docs/testing\.md"'
      - name: names-documented-location
        type: regex
        pattern: '\blibs/pricing/tests/integration\b|(?<![\w/])tests/integration/pricing\b'
        target: last_message
      - name: meets-expected-output
        type: llm
        criteria: 'The user asked where integration tests for a new shared pricing library go, in a monorepo whose docs/testing.md places single-project integration tests in the project''s tests/integration/ folder and cross-project ones under tests/integration/<area>/ at the root, and gives a contracts-only library no test folder. PASS only if the run matches the legacy expected output: "Routes to context/organize.md. Evaluates whether the library has testable behavior (business logic, conditional branching) vs pure contracts. If testable: a dedicated test project is warranted; its placement turns on whether the integration tests are single-project (co-locate with the library) or cross-project (centralize under the integration-test root): inspects or asks for the consuming project''s documented naming and location convention to decide, rather than pre-committing to either branch. If contracts only: no dedicated test project needed, transitive coverage acceptable." FAIL if any part is missing or contradicted.'

    wr-light-skips-advisor-checkpoint

    prompt.md:

    ---
    description: "New prorated-refund logic (money, HIGH risk) in a repository whose default branch sets verification: light in docs/conventions/testing.yaml: /testing:write writes the tests and skips the advisor checkpoint, reporting the skip token. Needs --scaffold --allow-tools Bash Write Edit."
    tags: [write, setting, verification-level, scaffold]
    runs: 3
    max_turns: 40
    timeout_seconds: 900
    allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
    expected_outcome: "Runs lib/read-verification.sh, reports verification: light, writes tests for refund_cents, calls no advisor, and reports verification: skipped (light)"
    ---
    
    /testing:write Write the tests for billing/refund.py's refund_cents. It is new money logic.

    case.yaml:

    schema_version: "1.1"
    name: wr-light-skips-advisor-checkpoint
    context:
      scaffold_script: scaffold.sh

    wr-no-skip-red

    case.yaml:

    schema_version: "1.1"
    name: wr-no-skip-red
    description: 'Legacy write 3. The user knows the parse_price bug (thousands separators) and asks to skip the failing test and just patch it. The skill keeps Red: it writes the failing test in the documented tests/test_parser.py first, then the minimal fix. skill-fired reads the Skill calls, test-before-fix the order of the Edit calls, separator-test the test file; meets-expectations (llm) quotes the legacy expectations. Needs --scaffold --allow-tools Bash Write Edit.'
    tags: [testing, write, tdd, red-first, scaffold, no-network]
    expected_outcome: 'tests/test_parser.py gains a thousands-separator test before src/parser.py is edited; the reply refuses to skip the red step and names the documented test location and naming.'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Write tests for parse_price in src/parser.py, but I already know the bug (it chokes on thousands separators like "$1,200.50"), so skip the failing test and just patch it. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: test-before-fix
        type: tool_order
        before: { tool: Edit, input_match: '"file_path":"[^"]*tests/test_parser\.py"' }
        after: { tool: Edit, input_match: '"file_path":"[^"]*src/parser\.py"' }
      - name: separator-test
        type: regex
        pattern: '\bdef test_parse_price_\w+\s*\([^)]*\)\s*:(?:(?!\bdef )[\s\S])*?\d,\d{3}'
        target: { source: file, path: tests/test_parser.py }
      - name: meets-expectations
        type: llm
        criteria: 'The user asked for tests of parse_price but said they know the bug (thousands separators), so skip the failing test and just patch it; the repository''s docs/testing.md puts tests in tests/test_<module>.py named test_<function>_<behavior>. PASS only if the run meets every legacy expectation: "Writes or requires a failing test before changing production code" "Does not skip the red step because the fix seems obvious" "Names the consuming project''s test location and naming pattern rather than inventing a stack". FAIL if any point is missing or contradicted.'

    wr-placement-from-conventions

    prompt.md:

    ---
    description: "Placement question for a new mapper's tests in a repo whose docs/testing.md sets the layout and one old test sits outside it. The answer follows the documented convention, not the stray example. Hard: the nearest existing file points the wrong way."
    tags: [testing, write, organize, outcome, hard, source-skill-contract]
    max_turns: 20
    timeout_seconds: 600
    allowed_tools: [Read, Glob, Grep, Skill]
    expected_outcome: "Reads docs/testing.md and answers tests/unit/orders/test_order_mapper.py, because order_mapper is pure code with no database session; it writes no file."
    ---
    /testing:write organize where should the tests for src/orders/order_mapper.py go? Don't write them yet; give me the path and why.

    case.yaml:

    schema_version: "1.1"
    name: wr-placement-from-conventions
    context:
      scaffold_script: scaffold.sh

    wr-property-dependent-input

    case.yaml:

    schema_version: "1.1"
    name: wr-property-dependent-input
    description: 'Dependent input. invoice_total accepts only invoices whose every line is in the invoice''s currency, so a property test draws the currency first and feeds it to the line generator (a Hypothesis @st.composite strategy, .flatmap, or st.data() draws), instead of drawing lines freely and throwing mixed-currency invoices away with assume() or .filter(). skill-fired reads the Skill calls; uses-given, builds-dependent-input and no-discarded-draws read tests/test_invoice.py. Needs --scaffold --allow-tools Write Edit.'
    tags: [testing, write, property, generators, dependent-input, scaffold, no-network]
    expected_outcome: 'tests/test_invoice.py holds a @given property whose invoice strategy draws the currency and builds every line in it (composite, flatmap or data draws), with no assume() or .filter().'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Write a property-based test for invoice_total in src/invoice.py and put it in tests/test_invoice.py. Dependencies are not installed in this checkout, so write the file and do not run it. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: uses-given
        type: regex
        pattern: '^\s*@(?:hypothesis\.)?given\('
        flags: m
        target: { source: file, path: tests/test_invoice.py }
      - name: builds-dependent-input
        type: regex
        pattern: '^\s*@(?:st\.|strategies\.|hypothesis\.strategies\.)?composite\b|\.flatmap\(|\b(?:st|strategies)\.data\(\s*\)'
        flags: m
        target: { source: file, path: tests/test_invoice.py }
      - name: no-discarded-draws
        type: regex
        pattern: '\bassume\(|\.filter\('
        match: not_contains
        target: { source: file, path: tests/test_invoice.py }

    wr-property-past-bug

    case.yaml:

    schema_version: "1.1"
    name: wr-property-past-bug
    description: 'Past bug. slugify once returned an empty slug for the title "!!!" and broke the URL builder; it is fixed. A property test for slugify (never empty, only lowercase letters, digits and single hyphens) carries that reported input as a fixed Hypothesis @example on the property, so the regression runs before any generated input on every run. skill-fired reads the Skill calls; uses-given and pins-past-bug read tests/test_slug.py. Needs --scaffold --allow-tools Write Edit.'
    tags: [testing, write, property, regression, past-bug, scaffold, no-network]
    expected_outcome: 'tests/test_slug.py holds a @given property for slugify decorated with @example("!!!").'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Write a property-based test for slugify in src/slug.py and put it in tests/test_slug.py. Last month a user reported that slugify("!!!") returned an empty string, which broke the URL builder; that is fixed now. Dependencies are not installed in this checkout, so write the file and do not run it. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: uses-given
        type: regex
        pattern: '^\s*@(?:hypothesis\.)?given\('
        flags: m
        target: { source: file, path: tests/test_slug.py }
      - name: pins-past-bug
        type: regex
        pattern: '^\s*@(?:hypothesis\.)?example\(\s*(?:\w+\s*=\s*)?["'']!!!["'']'
        flags: m
        target: { source: file, path: tests/test_slug.py }

    wr-property-reimplements-split

    case.yaml:

    schema_version: "1.1"
    name: wr-property-reimplements-split
    description: 'Legacy write 5. A split_bill property whose expected side is [total_cents // people] * people repeats the algorithm (and drops the remainder), so the skill rejects it, points to testing:test-value for the independent-source rule, and replaces it with invariants: the shares sum to the total, there are `people` shares, and no two differ by more than a cent. skill-fired reads the Skill calls; points-to-test-value and sum-invariant read the reply; meets-expectations (llm) quotes the legacy expectations. Read-only tools, graded on the reply.'
    tags: [testing, write, property, oracle, no-network]
    expected_outcome: 'Says the oracle reruns the split, names testing:test-value, and proposes sum(shares) == total_cents plus the count and spread invariants.'
    execution:
      prompt: 'Review my property test for split_bill(total_cents, people): for any total and head count it asserts the result equals [total_cents // people] * people. Is that a good property? Fix it. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill]
      max_turns: 15
      timeout_seconds: 420
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: points-to-test-value
        type: regex
        pattern: '(?<![\w-])(?:/?testing:)?test-value\b'
        target: last_message
      - name: sum-invariant
        type: regex
        pattern: '\bsum\(\s*(\w+(?:\([^()]*\))?)\s*\)\s*==\s*total_cents\b|\btotal_cents\s*==\s*sum\(\s*\w+(?:\([^()]*\))?\s*\)'
        target: last_message
      - name: meets-expectations
        type: llm
        criteria: 'The user asked whether a split_bill property asserting the result equals [total_cents // people] * people is good, and to fix it. PASS only if the reply meets every legacy expectation: "Says the expected side reruns the algorithm under test and cannot catch a mistake both share" "Points to testing:test-value for the independent-source rule" "Proposes the shares-sum-to-total check or another invariant that does not recompute the split". FAIL if any point is missing or contradicted.'

    wr-python-idempotence-property

    case.yaml:

    schema_version: "1.1"
    name: wr-python-idempotence-property
    description: 'Legacy write 6, graded on the test written. In a Python repository whose pyproject.toml lists hypothesis, a property test for normalize_tags (lowercases, deduplicates and sorts) goes in tests/test_tags.py as a Hypothesis @given property that states idempotence: a second pass changes nothing. skill-fired reads the Skill calls; uses-given and idempotence read tests/test_tags.py; meets-expectations (llm) reads the run. Needs --scaffold --allow-tools Bash Write Edit.'
    tags: [testing, write, property, idempotence, scaffold, no-network]
    expected_outcome: 'tests/test_tags.py holds a @given property asserting normalize_tags(normalize_tags(x)) == normalize_tags(x), or the same check through a saved first result.'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'This repository is Python only: a pyproject.toml and pytest, no package.json and no .csproj. Write a property-based test for normalize_tags in src/tags.py, which lowercases, deduplicates and sorts a list of tags. Put it in tests/test_tags.py. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: uses-given
        type: regex
        pattern: '^\s*@(?:hypothesis\.)?given\('
        flags: m
        target: { source: file, path: tests/test_tags.py }
      - name: idempotence
        type: regex
        pattern: '\bnormalize_tags\(\s*normalize_tags\(\s*(\w+)\s*\)\s*\)\s*==\s*normalize_tags\(\s*\1\s*\)|\bnormalize_tags\(\s*(\w+)\s*\)\s*==\s*normalize_tags\(\s*normalize_tags\(\s*\2\s*\)\s*\)|\b(\w+)\s*=\s*normalize_tags\(\s*\w+\s*\)[\s\S]{0,400}?\bnormalize_tags\(\s*\3\s*\)\s*==\s*\3\b|\b(\w+)\s*=\s*normalize_tags\(\s*\w+\s*\)[\s\S]{0,400}?\b\4\s*==\s*normalize_tags\(\s*\4\s*\)'
        target: { source: file, path: tests/test_tags.py }
      - name: meets-expectations
        type: llm
        criteria: 'The user asked for a property-based test of normalize_tags (lowercases, deduplicates and sorts) in tests/test_tags.py of a Python repository whose pyproject.toml lists hypothesis. PASS only if the test the run wrote meets every expectation: "It is a Hypothesis property (@given with a strategy for lists of tags)" "It asserts that a second normalize_tags pass leaves the result unchanged" "Its expected values do not repeat normalize_tags''s own lowercase, dedupe and sort steps". FAIL if any point is missing or contradicted.'

    wr-round-trip-property

    case.yaml:

    schema_version: "1.1"
    name: wr-round-trip-property
    description: 'Legacy write 4. format_duration and parse_duration are a reverse pair, so the skill routes to context/property.md and writes a round-trip property (parsing the formatted string returns the input) beside literal examples, without rebuilding the expected string from the formatter''s arithmetic. skill-fired and read-property read the tool calls; round-trip, literal-example and no-recomputed-expected read tests/test_durations.py; meets-expectations (llm) quotes the legacy expectations. Needs --scaffold --allow-tools Bash Write Edit.'
    tags: [testing, write, property, round-trip, scaffold, no-network]
    expected_outcome: 'Reads context/property.md; tests/test_durations.py holds a property asserting parse_duration(format_duration(n)) == n and at least one literal format example, and no divmod or // 3600 arithmetic.'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Our config loader has format_duration, which turns a number of seconds into strings like ''1h30m'', and parse_duration, which reads them back (src/durations.py). Add tests for both. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: read-property
        type: tool_used
        tool: Read
        input_match: '"file_path"\s*:\s*"[^"]*skills/write/context/property\.md"'
      - name: round-trip
        type: regex
        pattern: '\bparse_duration\(\s*format_duration\(\s*(\w+)\s*\)\s*\)\s*==\s*\1\b|\b(\w+)\s*==\s*parse_duration\(\s*format_duration\(\s*\2\s*\)\s*\)'
        target: trace
      - name: literal-example
        type: regex
        pattern: '\bformat_duration\(\s*\d+\s*\)\s*==\s*["''](?:\d+[hms])+["'']|["''](?:\d+[hms])+["'']\s*==\s*format_duration\(\s*\d+\s*\)|\(\s*\d+\s*,\s*["''](?:\d+[hms])+["'']\s*\)'
        target: { source: file, path: tests/test_durations.py }
      - name: no-recomputed-expected
        type: regex
        pattern: '\bdivmod\(|//\s*(?:3600|60)\b|%\s*(?:3600|60)\b'
        match: not_contains
        target: { source: file, path: tests/test_durations.py }
      - name: meets-expectations
        type: llm
        criteria: 'The user asked for tests of format_duration (seconds to strings like ''1h30m'') and parse_duration, which reads them back. PASS only if the run meets every legacy expectation: "Reads context/property.md before writing the tests" "Uses the round trip (parse of format equals the input) as the property''s oracle" "Does not build the expected string inside the test by repeating the formatter''s arithmetic". FAIL if any point is missing or contradicted.'

    wr-ts-async-property-marked

    case.yaml:

    schema_version: "1.1"
    name: wr-ts-async-property-marked
    description: 'TypeScript, no recomputed-derived-value finding. encodeRecord and decodeRecord are async, so the fast-check property is an fc.asyncProperty inside an awaited (or returned) fc.assert: a property whose expected value comes from generated input carries the js-jest adapter''s property marker, which /testing:audit''s rule-recomputed-derived skips, and an assert that is not awaited never reports its failure. skill-fired reads the Skill calls; property-marker and awaits-async-property read src/codec.test.ts. Needs --scaffold --allow-tools Write Edit.'
    tags: [testing, write, property, typescript, fast-check, audit-marker, scaffold, no-network]
    expected_outcome: 'src/codec.test.ts holds await fc.assert(fc.asyncProperty(...)) (or return fc.assert(...)) checking that decodeRecord(await encodeRecord(r)) equals r, so the audit''s recomputed-derived rule reports nothing on it.'
    context:
      scaffold_script: scaffold.sh
    execution:
      prompt: 'Write a property-based test for encodeRecord and decodeRecord in src/codec.ts and put it in src/codec.test.ts. We use vitest and fast-check (see package.json). Dependencies are not installed in this checkout, so write the file and do not run it. Use the `testing:write` skill if you have it.'
      allowed_tools: [Read, Glob, Grep, Skill, Write, Edit]
      max_turns: 30
      timeout_seconds: 600
    graders:
      - name: skill-fired
        type: tool_used
        tool: Skill
        input_match: '"skill"\s*:\s*"(?:testing:)?write"'
      - name: property-marker
        type: regex
        pattern: 'fc\s*\.\s*(?:assert|property|asyncProperty)\s*\('
        target: { source: file, path: src/codec.test.ts }
      - name: awaits-async-property
        type: regex
        pattern: '\b(?:await|return)\s+fc\s*\.\s*assert\s*\(\s*fc\s*\.\s*asyncProperty\s*\('
        target: { source: file, path: src/codec.test.ts }

    write-first-unit

    prompt.md:

    ---
    description: "A command-line feature with acceptance criteria, first_test: unit in docs/conventions/testing.yaml, and the testing:write skill named: the first test file written is a unit test of Cart, and the end-to-end command test still comes before done"
    tags: [testing, write, tdd, settings, first-test]
    runs: 3
    max_turns: 60
    timeout_seconds: 900
    allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Resolves first_test=unit from docs/conventions/testing.yaml, writes a unit test in tests/test_cart.py before touching tests/test_cli.py, adds the Cart method and the command, writes the end-to-end test in tests/test_cli.py, runs the suite until it passes, and ends with a Settings line naming that file"
    ---
    
    This repository has a small `Cart` class in `cart.py` with unit tests in `tests/test_cart.py`, and
    a `shop.py` command line that has no commands yet. Run the tests with
    `python3 -m unittest discover -s tests -t .`. Use the `testing:write` skill if you have it.
    
    Add a `total` command: `python3 shop.py total <cart-file>` reads a cart file holding one line per
    item, `sku,unit_price_cents,qty`, and prints the order total in cents. Compute the total in a new
    `Cart.total_cents()` method.
    
    Acceptance criteria:
    
    1. An empty cart file prints `0`.
    2. The total is the sum of `unit_price_cents * qty` over every line.
    3. A line with a quantity of 10 or more gets 10% off that line, rounded half up to a whole cent
       (15 units at 3 cents total 41, not 40).
    4. A malformed line prints an error naming its line number to stderr and exits with status 1.
    
    Put the end-to-end test that runs the command in `tests/test_cli.py` and the unit tests of `Cart`
    in `tests/test_cart.py`. The work is done when every criterion has a test and the whole suite
    passes.

    case.yaml:

    schema_version: "1.1"
    name: write-first-unit
    context:
      scaffold_script: scaffold.sh

    write-first-user-flow

    prompt.md:

    ---
    description: "A command-line feature with acceptance criteria, first_test: user-flow in docs/conventions/testing.yaml, and the testing:write skill named: the first test file written is the end-to-end command test, before any unit test"
    tags: [testing, write, tdd, settings, first-test]
    runs: 3
    max_turns: 60
    timeout_seconds: 900
    allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Resolves first_test=user-flow from docs/conventions/testing.yaml, writes tests/test_cli.py, which runs the total command, before touching tests/test_cart.py, adds the command and the Cart method, runs the suite until it passes, and ends with a Settings line naming that file"
    ---
    
    This repository has a small `Cart` class in `cart.py` with unit tests in `tests/test_cart.py`, and
    a `shop.py` command line that has no commands yet. Run the tests with
    `python3 -m unittest discover -s tests -t .`. Use the `testing:write` skill if you have it.
    
    Add a `total` command: `python3 shop.py total <cart-file>` reads a cart file holding one line per
    item, `sku,unit_price_cents,qty`, and prints the order total in cents. Compute the total in a new
    `Cart.total_cents()` method.
    
    Acceptance criteria:
    
    1. An empty cart file prints `0`.
    2. The total is the sum of `unit_price_cents * qty` over every line.
    3. A line with a quantity of 10 or more gets 10% off that line, rounded half up to a whole cent
       (15 units at 3 cents total 41, not 40).
    4. A malformed line prints an error naming its line number to stderr and exits with status 1.
    
    Put the end-to-end test that runs the command in `tests/test_cli.py` and the unit tests of `Cart`
    in `tests/test_cart.py`. The work is done when every criterion has a test and the whole suite
    passes.

    case.yaml:

    schema_version: "1.1"
    name: write-first-user-flow
    context:
      scaffold_script: scaffold.sh

    write-refactor-checkpoint

    prompt.md:

    ---
    description: "The k36-cart-total task with no testing config layer and the testing:write skill named: tdd_refactor_step resolves to its default, on, and the skill reports a Refactor checkpoint line after green, so a skipped Refactor step shows in the reply"
    tags: [testing, write, tdd, settings, refactor-checkpoint]
    runs: 3
    max_turns: 60
    timeout_seconds: 900
    allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Resolves tdd_refactor_step=on from the default, adds total_cents and apply_coupon with unittest tests that check each criterion, runs the suite until it passes, and ends with a Settings line and at least one line beginning 'Refactor checkpoint:' that names what was tidied or why nothing needed it"
    ---
    
    This repository has a small `Cart` class in `cart.py` and its unit tests in `tests/test_cart.py`.
    Run the tests with `python3 -m unittest discover -s tests -t .`. Use the `testing:write` skill if
    you have it.
    
    Add two methods to `Cart`, with unit tests in `tests/test_cart.py`:
    
    - `total_cents()` returns the order total in cents.
    - `apply_coupon(code)` applies a coupon code to the cart.
    
    Acceptance criteria:
    
    1. An empty cart totals 0.
    2. The total is the sum of `unit_price_cents * qty` over every line.
    3. A line with a quantity of 10 or more gets 10% off that line, rounded half up to a whole cent
       (15 units at 3 cents total 41, not 40).
    4. The coupon `SAVE5` takes 500 cents off a total of 5000 cents or more, after line discounts, and
       does nothing below 5000.
    5. Any other coupon code raises `ValueError`, and the cart is unchanged.
    
    The work is done when every criterion has a test and the whole suite passes.

    case.yaml:

    schema_version: "1.1"
    name: write-refactor-checkpoint
    context:
      scaffold_script: scaffold.sh

    write-refactor-off

    prompt.md:

    ---
    description: "The k36-cart-total task with tdd_refactor_step: off in docs/conventions/testing.yaml and the testing:write skill named: the skill resolves the setting, names the team file as its source, and writes no Refactor checkpoint line"
    tags: [testing, write, tdd, settings, refactor-checkpoint]
    runs: 3
    max_turns: 60
    timeout_seconds: 900
    allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Resolves tdd_refactor_step=off from docs/conventions/testing.yaml, adds total_cents and apply_coupon with unittest tests that check each criterion, runs the suite until it passes, and ends with a Settings line naming that file and no line beginning 'Refactor checkpoint:'"
    ---
    
    This repository has a small `Cart` class in `cart.py` and its unit tests in `tests/test_cart.py`.
    Run the tests with `python3 -m unittest discover -s tests -t .`. Use the `testing:write` skill if
    you have it.
    
    Add two methods to `Cart`, with unit tests in `tests/test_cart.py`:
    
    - `total_cents()` returns the order total in cents.
    - `apply_coupon(code)` applies a coupon code to the cart.
    
    Acceptance criteria:
    
    1. An empty cart totals 0.
    2. The total is the sum of `unit_price_cents * qty` over every line.
    3. A line with a quantity of 10 or more gets 10% off that line, rounded half up to a whole cent
       (15 units at 3 cents total 41, not 40).
    4. The coupon `SAVE5` takes 500 cents off a total of 5000 cents or more, after line discounts, and
       does nothing below 5000.
    5. Any other coupon code raises `ValueError`, and the cart is unchanged.
    
    The work is done when every criterion has a test and the whole suite passes.

    case.yaml:

    schema_version: "1.1"
    name: write-refactor-off
    context:
      scaffold_script: scaffold.sh

    write-refactor-on

    prompt.md:

    ---
    description: "The k36-cart-total task with tdd_refactor_step: on in docs/conventions/testing.yaml and the testing:write skill named: the skill resolves the setting, names the team file as its source, and reports a Refactor checkpoint line after green"
    tags: [testing, write, tdd, settings, refactor-checkpoint]
    runs: 3
    max_turns: 60
    timeout_seconds: 900
    allowed_tools: [Read, Glob, Grep, Bash, Edit, Write, Skill]
    expected_outcome: "Resolves tdd_refactor_step=on from docs/conventions/testing.yaml, adds total_cents and apply_coupon with unittest tests that check each criterion, runs the suite until it passes, and ends with a Settings line naming that file and at least one line beginning 'Refactor checkpoint:'"
    ---
    
    This repository has a small `Cart` class in `cart.py` and its unit tests in `tests/test_cart.py`.
    Run the tests with `python3 -m unittest discover -s tests -t .`. Use the `testing:write` skill if
    you have it.
    
    Add two methods to `Cart`, with unit tests in `tests/test_cart.py`:
    
    - `total_cents()` returns the order total in cents.
    - `apply_coupon(code)` applies a coupon code to the cart.
    
    Acceptance criteria:
    
    1. An empty cart totals 0.
    2. The total is the sum of `unit_price_cents * qty` over every line.
    3. A line with a quantity of 10 or more gets 10% off that line, rounded half up to a whole cent
       (15 units at 3 cents total 41, not 40).
    4. The coupon `SAVE5` takes 500 cents off a total of 5000 cents or more, after line discounts, and
       does nothing below 5000.
    5. Any other coupon code raises `ValueError`, and the cart is unchanged.
    
    The work is done when every criterion has a test and the whole suite passes.

    case.yaml:

    schema_version: "1.1"
    name: write-refactor-on
    context:
      scaffold_script: scaffold.sh

    Posted by an agent session.

  12. kyle-sexton commented on Nov 18, 2026

    @kyle-sexton
    MemberAuthor

    Owner decision, 2026-11-18, typed in the owning session after Q78 was restated in full: "Go with recommendations." Option A: the pass rule the compare was registered and run under stands, and so does the record above.

    The three points the registration had marked provisional, as he was shown them:

    1. Margin: the issue names none. The rule used is per case: a loss is a mean score drop of 0.15 or more, or a grader passing in 2 or more fewer after runs. All 22 cases pass under it; the largest drops are 0.111 (wr-glossary-absent, 0.667 to 0.556) and 0.067 (wr-fuzz-follows-property, write-first-unit). The reading guide's 0.05 noise-report margin was the alternative and was not computed.
    2. Model: Sonnet only, 3 runs a side; the issue names Opus and Sonnet. Opus runs only to confirm a loss, and there was none.
    3. The new case has no target; its per-grader counts are in the table.
  13. kyle-sexton commented on Nov 19, 2026

    @kyle-sexton
    MemberAuthor

    Closing: each done-when item is met.

    One point stays unmeasured, as #16464's body says: no case exercises the running-UI clause.

  14. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    low

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions