Skip to content

docs(testing): master test plan — 17 chapters, 1,186 rows, a three-tier release gate - #60

Open
wshallwshall wants to merge 1 commit into
mainfrom
master-test-plan
Open

docs(testing): master test plan — 17 chapters, 1,186 rows, a three-tier release gate#60
wshallwshall wants to merge 1 commit into
mainfrom
master-test-plan

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

The umbrella test plan the two existing artifacts plug into.
FEATURE-COVERAGE-PLAN.md owns the per-subsystem
coverage-gap audit; WIN2025-TEST-PLAN.md owns what only a real
Server 2025 host under the NSSM service identity can prove. Neither covers the product as a whole,
and neither covers the IDE, the tray, the Steps editor or the publish path at all.

Ships as docs/testing/MASTER-TEST-PLAN.md (index) plus 20
chapter files under docs/testing/master-test-plan/ — the assembled document is 1.8 MB, past
GitHub's markdown render limit. Every chapter carries scope, what is already covered and must not
be re-tested
, a risk analysis, the test matrix, detailed scenarios, an automation disposition,
prerequisites, exit criteria and open questions.

Foreign IDs are prefixed FCP: / W25: throughout, because both predecessors use IDs that collide
with this plan's own — HA-20 and API-13 exist in both spaces.

What is load-bearing

Row classes. Only class-T rows gate a release. A criterion of "Pass = report received" or
"Pass = a published number" cannot fail, so it cannot be a gate — those are class C
(characterisation, tracked) and class A (external assurance, gates production exposure only).
The draft had a penetration-test procurement event sitting at P0. The gate is now 196 blocking
T-rows + 9 campaign-gated + 3 C + 3 A = 211 P0-marked, reconciled row-for-row against an
independent parse of all 17 matrices.

The publishing boundary. docs/BACKLOG.md is a published baseline ending at ## 231. and
docs/security/ is gitignored post-cutover, so a citation above the baseline or into a withheld tree
is valid evidence, not a dangling reference. An earlier pass had this backwards and was filing
withheld material as coverage defects — including one that had reached a release-gate criterion.

Two owner-named items were structurally unprovable. The tray application held zero P0 rows
(all eight sat on NSSM/install) even though ADR 0113 names the untested Win32 layer as the design's
main risk; the state→icon map, action routing, message-pump dispatch and the "lying tray" are now
blocking. HA had no P0 proving a failover works — six now do, all on container-CI with a
service container, none needing a second host.

Surfaces nothing previously owned

last_resort.py (the process-level PHI-egress backstop), netaddr.py (the shared allow-list
decision its co-location exists to keep consistent), pki.py, the shipped PySide6 harness GUI —
which is both a distribution and the plan's only route to hostile-input injection — plus disk-full /
ENOSPC, licence and copyleft compliance, destructive operator errors, and non-ASCII patient data
end to end.

Review notes

  • 225 open questions are addressed to the owner; several are blocking decisions (the WCAG target,
    whether load_settings should refuse unknown keys, which transports are deliberately not
    authorable from connections.toml).
  • The plan's own headline finding: messagefoundry check is invoked by no workflow, and one
    meta-test plus one ci.yml regex fix turns 36 already-written tests from dormant into
    executing. That is items 1–2 of §24's prioritised ten.
  • All traffic in the plan is synthetic and PHI-free; reports are metrics-and-metadata only.
  • Docs only — no code, no test, no CI change in this PR.

🤖 Generated with Claude Code

…three-tier gate

The umbrella test plan the two existing artifacts plug into. FEATURE-COVERAGE-PLAN.md
owns the per-subsystem coverage-gap audit and WIN2025-TEST-PLAN.md owns what only a real
Server 2025 host can prove; neither covers the product as a whole, and neither covers the
IDE, the tray, the Steps editor or the publish path at all. This states what the product
is, what has to be true before it ships, and who proves it — delegating rather than
restating, with foreign IDs prefixed FCP:/W25: because both predecessors use IDs that
collide with this plan's own (HA-20 and API-13 exist in both spaces).

Three things are load-bearing:

Row classes. Only class-T rows gate a release. A criterion of "Pass = report received" or
"Pass = a published number" cannot fail, so it cannot be a gate — those are class C
(characterisation, tracked) and class A (external assurance, gates production exposure
only). The draft had a penetration-test procurement event sitting at P0. The gate is now
196 blocking T-rows + 9 campaign-gated + 3 C + 3 A = 211 P0-marked, reconciled row-for-row
against an independent parse.

The publishing boundary. docs/BACKLOG.md is a published baseline ending at #231 and
docs/security/ is gitignored post-cutover, so a citation above the baseline or into a
withheld tree is valid evidence, not a dangling reference. An earlier pass had this
backwards and was filing withheld material as coverage defects, including one that reached
a release-gate criterion.

Two owner-named items were structurally unprovable. The tray had zero P0 rows — all eight
sat on NSSM/service/install — so the icon-state mapping, action routing, Win32 message
pump and the lying-tray case are now blocking. HA had no P0 that proved a failover works;
six now do.

Also folds in surfaces nothing owned: last_resort.py (the PHI-egress backstop),
netaddr.py (the shared allow-list decision the co-location exists to keep consistent),
pki.py, the shipped PySide6 harness GUI, disk-full/ENOSPC, licence/copyleft compliance,
destructive operator errors and non-ASCII patient data end to end.

Ships as an index plus 20 chapter files: the assembled document is 1.8 MB, past GitHub's
markdown render limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wshallwshall added a commit that referenced this pull request Jul 30, 2026
PR #80 appended eight Steps-view items to docs/BACKLOG.md without the
status banner every numbered item must carry, so
tests/test_backlog_status_check.py::test_the_real_backlog_satisfies_the_invariant
began failing on main itself (8 errors, first at line 6905). Because
GitHub tests each PR merged into main, every open PR inherited the
failure: #81, #74, #71, #66 and #60 were all blocked, three of them with
auto-merge armed and unable to fire.

Adds exactly one leading banner per item. Seven use the open/prioritized
form; #239 uses the partial form, because its measurement ran and is
recorded on PR #81 while the re-runnable scripts/quality/lens_coverage.py
is still unmerged -- the number is not yet reproducible from main.

The banners are deliberately minimal and do not re-litigate any item's
verdict. #234 in particular is left explicitly unsettled rather than
entrenched: it was filed as "revisit, not a bug" after the owner asked
for a fix, and that framing is still open.

Verified: scripts/docs/backlog_status_check.py exits 0 (237 items) and
tests/test_backlog_status_check.py is 15 passed, was 14 passed 1 failed.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
wshallwshall added a commit that referenced this pull request Aug 2, 2026
That is the exact defect the backlog-hygiene gate exists for -- #60 shipped while its banner still
read "PRE-RESERVED", and the stale banner was then repeated as fact in a merged PR. The gate did not
fire here because it only inspects engine/IDE paths and this item is script-and-test work, which is
precisely why the banner had to be corrected by hand rather than trusted to CI.

Records what actually shipped (Half A) against what did not (Half B), and why Half B was backed out:
claim.ps1 had three sessions in it at once, so the deferral is a coordination decision, not an
estimate of difficulty. Someone reading this item needs that distinction to know it is pickup-able.

The first draft of this named the contending BRANCH and the leak gate blocked it -- a worktree slug
is internal naming and this ledger is public. Described by the code path it touches instead.
wshallwshall added a commit that referenced this pull request Aug 2, 2026
…OG #345) (#141)

* fix(coord): a pruned worktree stranded its work claims forever (BACKLOG #345)

A claim (scripts/coord/claim.ps1) is a JSON file under <git-common-dir>/mefor-coord/claims/. It
lives beside the SHARED object store, so it outlives the worktree that took it. prune-merged.ps1
removed worktrees and, across 1,108 lines, never touched claims -- and `claim.ps1 -Take` hard-blocks
on any claim file that exists. So every prune could leave a key unclaimable by every future session
until a human happened to run `-Release <key> -Force`. Nothing surfaced the condition and nothing
could: the registry has no way to observe that a holder ceased to exist.

Released on EVIDENCE, never on a timer. claim.ps1 is right that an auto-expiring claim silently
re-opens the race it exists to prevent; this hooks only into the branch that has already PROVEN the
directory is gone and deregistered ("OUTCOME, NOT EXIT CODE"), so there is no session left to
collide with. A merely-quiet holder is never touched. Dry-run safety is inherited, not re-asserted:
the whole apply loop is gated on -Apply.

The false positive is worse than the bug. Releasing a claim held by a different, LIVING worktree
hands its key away and invites the duplicate build the registry exists to stop, so the match is full
normalised equality via ConvertTo-Norm -- no leaf name, no prefix, no StartsWith. Two tests pin it:
a living sibling's claim survives, and the PRIMARY checkout's claim survives a sibling prune even
though `<primary>` is a strict prefix of `<primary>-clean`. Both carry a positive control in the
same invocation, so neither can pass on a run that released nothing.

Two defects found by the tests rather than by review, both of the shape this file already guards
against elsewhere:

  * an unreadable claim was counted once per REMOVED worktree, so one blocked key reported as 2.
    It belongs to the registry, not to any worktree -- not being able to read it is precisely not
    knowing whose it is -- so it is now surveyed once per run and reported at run level.
  * that survey ran only inside the removal branch, making it invisible to a dry run, and its
    Set-Exit sat AFTER the -Json block that emits the receipt and exits. The receipt would have
    carried exitCode 0 over a key nothing can claim -- the surface CI reads.

`claims.scanned` distinguishes "read the registry, found it clean" from "never looked" (no
git-common-dir, or no claims directory): an empty `unreadable` list is not a green light.

Verified: the 8 new tests FAIL against the unpatched script before they pass against the patched
one, so they can see the class. Full file green, 71 tests.

* docs(backlog): #345's banner said "not started" on the PR that builds it

That is the exact defect the backlog-hygiene gate exists for -- #60 shipped while its banner still
read "PRE-RESERVED", and the stale banner was then repeated as fact in a merged PR. The gate did not
fire here because it only inspects engine/IDE paths and this item is script-and-test work, which is
precisely why the banner had to be corrected by hand rather than trusted to CI.

Records what actually shipped (Half A) against what did not (Half B), and why Half B was backed out:
claim.ps1 had three sessions in it at once, so the deferral is a coordination decision, not an
estimate of difficulty. Someone reading this item needs that distinction to know it is pickup-able.

The first draft of this named the contending BRANCH and the leak gate blocked it -- a worktree slug
is internal naming and this ledger is public. Described by the code path it touches instead.

* docs(worktrees): the prune receipt grew a claims surface and nothing described it

WORKTREES.md is where the pruner's outcome contract is written down -- exit codes, which counts are
subsets of which, why `git worktree prune` is never run. A new field in that receipt that only the
source explains is the same drift this section already guards against elsewhere.

States the property that makes the release safe rather than just the behaviour: it fires on PROVEN
deletion, so a quiet holder is untouched and a dry run releases nothing, and the match is full path
equality because freeing a LIVING worktree's key is worse than the orphan being cleaned up.

* style(worktree): my #345 comments put 13 non-ASCII em-dashes in a BOM-less script

prune-merged.ps1 had exactly ZERO non-ASCII bytes before this branch and no BOM -- every comment in
it uses the ASCII `--`. A BOM-less file whose encoding is only inferred is a bad place to start
introducing multi-byte characters: pwsh 7 assumes UTF-8 and would be fine, Windows PowerShell 5.1
assumes the ANSI codepage and would not, and nothing in the repo pins which one an operator runs it
with. The file is back to 0 non-ASCII bytes.

Comments only; CRLF line endings verified byte-identical in count before and after, and the claim
tests re-run green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant