Strengthen verification tests and add the ASD test-audit workflow

.agents/skills/test-audit/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: test-audit
+description: Audit test value when requested, and apply a behavior-first authoring gate to test changes. Preserve meaningful contracts, coverage, and the task's editing limits.
+metadata:
+  version: "1.0"
+  status: released
+---
+
+# Test audit
+
+Use the installed delivery roles and owner-approved task scope. This supporting
+skill adapts [OpenClaw's test audit](https://github.com/openclaw/openclaw/blob/e09f3ecd491db23f4fdc6e79b4349808d010f394/.agents/skills/test-audit/SKILL.md)
+to dphtools. It adds neither permission nor another role. Use `implement-task`
+for routine authors and `review-work` for independent reviewers. Specialist A/B
+keep their blind inputs: they apply the authoring/value criteria without reading
+implementation, source history, or other roles' conversations.
+
+## Authoring gate
+
+Before changing a test, identify its observable contract, a credible regression,
+the gap in existing proof, and the real public or process boundary it exercises.
+Extend an existing case table when it covers the same contract. Another test
+layer needs a distinct risk, such as transport, lifecycle, or platform behavior.
+Expected values must come from approved behavior or independent references and
+invariants. Never add a production interface solely to make a test possible.
+
+A claimed bug regression needs evidence that valid input failed at the intended
+behavior before repair. Existing-behavior and supplemental tests may pass on the
+baseline; label them honestly rather than manufacturing red evidence.
+
+## Read-only audit
+
+For an authorized routine audit, inspect each complete candidate test, its real
+owner/entry point, callers/callees, overlapping tests, routing, and relevant
+history. Inspect applicable dependency source/types when the claim depends on
+them. Parallelize independent discovery where useful, then select one coherent
+owner-boundary batch. Broad campaigns need their own approved scope.
+
+Look for repeated proof, fixture self-comparisons, implementation-shaped greps,
+private-helper replays, coverage-only execution, expectations calculated by the
+subject, mocks/fixtures producing the promised result, negative cases rejected
+by the wrong guard, and names overstating the exercised behavior.
+
+These patterns are evidence to investigate, not automatic deletion rules.
+Retain independent public, platform, structural, packaging, release, and security
+contracts; observable ordering; credible regressions; and hard-to-reach numerical
+regimes. An externally used library API is not dead because only tests call it
+in this repository. An approved exception or successful completion can be the
+observable result. Static checks and slow tests can protect meaningful contracts.
+
+Before any deletion or consolidation, record:
+- exact test location and the failure it can detect;
+- non-test callers and the reason the test/support seam exists in history;
+- stronger retained proof, or the approved reason no proof is needed;
+- test/support edits enabled, risk, and focused validation command.
+
+Missing evidence leaves the candidate unchanged. A valid failing test remains
+visible: classify it under specification section 4. A tests-only task reports a
+product defect; it cannot repair production or delete the test to obtain green.
+
+## Repair, review, and completion
+
+Keep repairs within the owner's scope. Remove only demonstrated redundancy,
+move retained regressions to their owning boundary, and simplify repeated test
+setup without weakening checks or freezing implementation details. Production,
+tooling, workflows, test discovery, and coverage policy remain untouched in a
+tests-only audit. Instruction edits require their own owner authorization.
+
+Do not edit files while checks of that candidate are running. Run the focused
+owner/sibling tests, targeted formatting, and `git diff --check`. Then classify
+and verify the exact committed candidate with
+`python tools/verification.py pr --base BASE`; retain the selected complete-domain
+coverage requirements. A focused run is diagnostic, not full coverage evidence.
+
+Use a fresh independent reviewer after repairs. When the owner authorizes a
+review/repair loop, return concrete findings to the author and renew review after
+each changed candidate until acceptance or a real blocker. Respect inherited
+budgets and specialist allowances; a new session does not reset them. Never let
+the repair author approve their own changes.
+
+Report removed categories, retained false positives, focused/required evidence,
+test versus production/tooling line changes, unresolved findings, and PR state.
+Commit, push, PR creation, merge, and publication still need their applicable
+authorization. Use dphtools' commands; OpenClaw's Node runners, linked campaign
+and landing skills, and production-deletion preference are not installed here.

AGENTS.md

@@ -17,6 +17,8 @@ Use one task per worktree. Routine work uses an author and one fresh independent
 - D uses `review-work` in candidate mode, forms findings before consulting current-task implementation lessons, and reviews actual behavior, evidence, and simplicity. Do not fix and approve your own repair.
 - Setup may create infrastructure when explicitly authorized. Doctor uses `review-work` in doctor mode and may edit documentation under `DOCTOR-PROMPT.md`; neither exception is permission for C to weaken checks.
 
+Use the supporting [test-audit skill](.agents/skills/test-audit/SKILL.md) for requested test audits and its authoring criteria when changing tests. Keep its implementation-reading discovery on the routine route; blind A/B retain their restrictions. Audit findings do not authorize production edits.
+
 Prefer real end-to-end/public-entry-point tests. Smaller tests fill genuine gaps through interfaces production code or real callers also need. Existing-code tests may pass initially; do not manufacture red evidence or require mutation testing. Require exact 100% measured statements and branches across every owned domain required by the owner-approved verification hierarchy whenever that domain runs, including never-imported files and relevant subprocesses. Canonical `full` remains fresh global and per-package 100% measurement. Narrow runs establish only their declared domains; do not represent an omitted domain or reused evidence as fresh coverage. Report unsupported measurement. Do not hide failures, exclusions, skips, or missing reports.
 
 Before repair, the observing role classifies failures using specification section 4: environment/tooling, test defect, product defect, or unresolved requirement. The coordinator routes work; classification grants no new editing scope. Diagnose uncertain causes before product repair. Only a valid test failing at the intended approved behavior supplies product-red evidence. Routine test/fixture changes need independent review; specialist test/fixture changes need fresh A/B and a revised checkpoint. Unchanged tests keep their checkpoint after environment repair.

LESSONS/20261011T225144Z-TEST-AUDIT-002-coordinator-isolate-negative-guards.md

@@ -0,0 +1,9 @@
+# Isolate the guard a negative test claims to exercise
+
+- ID: 20261011T225144Z-TEST-AUDIT-002-coordinator-isolate-negative-guards
+- Date: 2026-10-11T22:51:44Z
+- Task/role: TEST-AUDIT-002 / coordinator
+- Status: confirmed
+- Observation: Invalid test inputs can trigger an earlier integrity or conservative-fallback guard, leaving the named semantic guard untested.
+- Evidence: Base b995f584; test_verification_shards.py foreign run/platform cases retained stale seals; its index-duplicate case retained foreign executions; test_verification_scope.py invalid inventory also changed the inventory path, and unchanged-prefix proof added an unsupported call. Read-only owner tracing is retained in /private/tmp/dphtools-test-audit-002-boundaries.md. These are test-value findings, not mutation or product-red evidence.
+- Lesson: Keep unrelated guards valid, isolate the candidate delta, and observe the intended rejection reason. Preserve separate byte-integrity/fallback controls.

docs/agentic-software-delivery-v1.0/DELIVERY-SYSTEM-SPEC.md

@@ -105,7 +105,7 @@ For each changed requirement, start with the smallest realistic success, failure
 
 Use integration or unit tests for a genuine gap: difficult error injection, combinatorial logic, or a branch that would make a whole-system test slow or brittle. No unit-test quota or fixed test-type percentage. Do not forbid valid negative assertions or property tests just because a heuristic dislikes them.
 
-A behavioral test is implementation-coupled when a refactor preserving its approved observable contract would break the test. Test interfaces that production code or real callers also need; do not require production exports, wrappers, or injection hooks solely for tests. D investigates code referenced only by tests, while recognizing that a public library API may serve external callers. An explicit public, platform, or architectural contract can justify a structural check; suspicion alone is not authority to delete an existing test. These judgment criteria adapt [OpenClaw's test audit](REFERENCES.md#r10--openclaw-test-audit), not its implementation-reading workflow.
+A behavioral test is implementation-coupled when a refactor preserving its approved observable contract would break the test. Test interfaces that production code or real callers also need; do not require production exports, wrappers, or injection hooks solely for tests. D investigates code referenced only by tests, while recognizing that a public library API may serve external callers. An explicit public, platform, or architectural contract can justify a structural check; suspicion alone is not authority to delete an existing test. These judgment criteria adapt [OpenClaw's test audit](REFERENCES.md#r10--openclaw-test-audit). The supporting [test-audit skill](../../.agents/skills/test-audit/SKILL.md) applies its authoring gate and evidence-first routine audit workflow under the installed roles. Routine audits may inspect implementation to establish test value; blind specialist A/B retain their source restrictions. An owner-authorized review/repair loop uses separate author and independent reviewer sessions, preserves inherited allowances, and ends at acceptance or a real blocker. Audit findings grant no product-editing scope.
 
 The independent reviewer (B on the specialist route) rejects execution solely to raise coverage without a meaningful correctness check; negative tests passing because an unrelated validation rejected the input; fixtures that supply the result whose production behavior is supposedly tested; and test names claiming more than their inputs exercise. A check need not use a literal assertion statement: an expected exception or successful completion may itself be approved observable behavior. Require the intended path and outcome, not a particular assertion syntax.
 

docs/agentic-software-delivery-v1.0/REFERENCES.md

@@ -72,6 +72,8 @@ Apply these as judgment criteria. A meaningful correctness check need not use a
 
 Do not import OpenClaw's implementation-reading audit, campaign, or landing workflows into blind A/B roles. Linked skills and commands were not read or installed. This source supplies no numerical-tolerance authority. The [inert archive](../references/openclaw/test-audit.source.txt) preserves the complete source with its [MIT notice](../references/licenses/openclaw-MIT.txt).
 
+The owner subsequently authorized the supporting [test-audit skill](../../.agents/skills/test-audit/SKILL.md) in [TEST-AUDIT-002](../tasks/TEST-AUDIT-002.md). It adapts routine implementation-reading discovery and separate repair/review to dphtools, while preserving blind specialist inputs, editing scope, and verification. Upstream text was reread at `e09f3ecd491db23f4fdc6e79b4349808d010f394`; the original archive and setup provenance above remain historical. OpenClaw campaign, runner, review, and landing commands remain uninstalled.
+
 ## Original articles and earlier supporting references
 
 The original four articles were revisited during setup. Their complete substantive prose was read, except that Symphony's embedded 1,363-line controller specification was deliberately excluded. The full narrative before and after that embed was read. OpenAI article bytes returned HTTP 403; the web tool supplied readable narrative text. No original-byte OpenAI article snapshot or article hash is claimed. Anthropic article bytes were retrieved and hashed, but full text is not republished without an identified redistribution license. [Reading notes](../references/READING-NOTES.md) distinguish the adopted ideas from excluded machinery.

docs/tasks/TEST-AUDIT-002.md

@@ -0,0 +1,91 @@
+# Test audit: verification
+
+## Authorization and candidate pointer
+
+The owner authorized adding the linked OpenClaw test-audit workflow to ASD,
+running separate audit and repair agents until complete, and opening a PR.
+Non-test code edits are forbidden. Instruction changes are the supporting skill
+and ASD pointers; runtime, tooling, workflows, discovery, and coverage policy
+remain outside repair scope. No merge or publication is authorized.
+
+Route: routine maintenance under the established local verification contract.
+The two discovery lanes and separate author produced the test repairs below.
+A fresh independent reviewer must inspect the candidate and applicable evidence.
+No new scenario/behavior contract or specialist checkpoint is introduced.
+
+Base: `b995f5843edbb9247cdb3d9c0b1b08c97010e02a` on `codex-main`.
+Branch: `codex/test-audit-verification`. This is the prepared task pointer:
+the authoritative Current state, final candidate identity, commands/environment,
+focused and required results, independent review, platform status, blockers,
+next action, and launch/budget metrics are recorded in the conversation or
+linked PR, outside this tracked candidate. This document preserves approved
+scope and pre-edit evidence rather than a competing mutable status copy.
+
+The owner supplied no spending cap; the routine review/repair loop runs until
+acceptance or a real blocker. Specialist allowances are not reset or used.
+Skill frontmatter validation passed using a disposable YAML dependency without
+changing the locked verification environment. A confirmed metrics product defect
+is explicitly deferred below because runtime code repair is forbidden.
+
+## Scope and method
+
+Apply [.agents/skills/test-audit](../../.agents/skills/test-audit/SKILL.md).
+Upstream text was read at `e09f3ecd491db23f4fdc6e79b4349808d010f394`
+(SHA-256 `fc38d2a34ade41960355d951b018dea1a0dfc89faca28914ba2c9a45d007fa3f`).
+The original inert archive and R10 setup provenance remain unchanged. This is
+a local adaptation; OpenClaw campaign/Node/landing skills are not installed.
+
+The coherent batch is `tests/test_verification*.py` and directly used test
+support, including [TEST-AUDIT-001's named follow-ups](TEST-AUDIT-001.md#named-follow-ups-not-edits-in-this-batch).
+Auditors inspected complete tests, runtime owners/entry points/callers/callees,
+overlaps, CI routing, applicable installed dependency sources/types, and history.
+Source reading was confined to routine discovery; blind A/B restrictions remain.
+
+Full pre-edit candidate fields and retained false positives are recorded outside
+Git in [core evidence](/private/tmp/dphtools-test-audit-002-core.md) and
+[boundary evidence](/private/tmp/dphtools-test-audit-002-boundaries.md).
+Locations below refer to the base, not shifted candidate lines.
+
+| Finding | Evidence and authorized test repair | Retained proof |
+|---|---|---|
+| Fixture coverage rereads | `test_verification.py:515` checks facts authored by its external-tool fixture. Remove redundant rereads. | Real coverage instrumentation, never-imported files and all invalid-report cases. |
+| Duplicate lint launches | `test_verification.py:908` directly reruns the same dependency already reached by the real fast command. Remove direct probes. | Public fast success and precise F821/file rejection. |
+| Incidental total order | `test_verification.py:503,702,781,1200` fixes independent operations to one sequence. Reuse inventory/dependency observations. | No missing/duplicate operations; quality/build/install/test/report prerequisites; receipts matching actual calls; live streaming. |
+| Vacuous identity assertion | `test_verification.py:1392` OR is already forced true by changed tags. Remove it and its unused local. | Actual Git/tag/history/dirty/version/artifact observations and changed-input reuse proof. |
+| Foreign identity masked by seal | `test_verification_shards.py:283` changes run/platform but retains a stale digest. Reseal to reach semantic identity rejection. | Separate byte-tamper cases and explicit intended-guard diagnostics. |
+| Duplicate-index case masks its claim | `test_verification_shards.py:530` reaches foreign execution first. Remove the redundant parameter/branch. | Real CLI duplicate-directory rejection and remaining semantic/execution cases. |
+| Invalid inventory masked by path fallback | `test_verification_scope.py:200` changes the inventory file itself. Establish invalid inventory in fixture base, then change only prose and require disclosed inventory failure. | Valid-inventory prose control and all closure/fallback cases. |
+| Dynamic prefix masked by new call | `test_verification_scope.py:293` introduces an unsupported call. Change only a literal while retaining unchanged helper/hook prefixes. | Existing ordinary literal control and unsupported-call tests. |
+| Dead support | Unused hook full observer branch, shard resealer and scope imports. Reuse support where meaningful; remove the remainder. | Every hook process/platform/backup test and complete receipt/measurement cases. |
+
+Real installation, receipt reuse, Versioneer, lock/startup bytes, hook platform
+behavior, sharded parent/child coverage, lifecycle, report integrity, and metrics
+parser tests stay. A suspicious pattern alone does not justify deletion.
+No runtime seam is removed; no threshold, exclusion, skip, or discovery rule changes.
+
+## Deferred product issue
+
+At the base, `tools/verification_metrics.py:88` accepts old receipt modes but
+rejects `library`, `doctor`, and `release`. Minimal temporary orchestration
+fixtures invoked the unchanged public verifier for each mode. Each returned 0
+with a complete passed version 2.0 receipt and the requested actual mode.
+The unchanged metrics CLI rejected each with exit 1 and
+`metrics: Unknown receipt mode`.
+
+[Probe results](/private/tmp/dphtools-test-audit-002-metrics-probe/results.json)
+retain exact receipts and logs. These controlled external-tool fixtures confirm
+CLI/receipt compatibility only; they are not real product coverage or a successful
+domain/full gate. No branch test is removed or introduced to conceal this issue.
+Repairing that runtime code requires a separately authorized task.
+
+## Validation and release impact
+
+Run focused modified-owner/sibling tests, targeted Black and `git diff --check`,
+then `python tools/verification.py pr --base origin/codex-main` on the exact
+committed candidate. Added instruction/task files and verification changes
+require Tier 5: fresh full verification, exact 100% statements/branches globally
+and per package. Hosted Linux/macOS/Windows evidence stays pending until observed.
+
+No package version, public behavior, or publication change. Draft release note:
+verification tests retain stronger process proof, isolate intended rejection
+guards, and ASD gains a supporting test-audit workflow.