feat: add a family conformance gate that executes documented exit contracts

[chaoz23]

Closes the gap found by the 2026-08-17 family-wide conformance audit. Implements the gate proposed in #81.

The problem

Every member's SKILL.md passed the clause-7 acceptance test — "a fresh-context agent, given only this file, can produce a well-formed invocation" — when the front doors landed in #71. All four still misstate their own exit-code contract.

That test checked invocability. Nothing checked truth. --schema, tool.json, and the MCP surface are all generated or tested; SKILL.md is hand-maintained prose that nothing executes.

What this adds

scripts/family_conformance.py — stdlib only, no new dependencies. It runs a member's CLI with read-only probes and diffs observed behaviour against its SKILL.md:

rule	catches
UNDOCUMENTED_EXIT_CODE	the CLI returns a code SKILL.md never documents
HONEST_LANE_OVERLOAD	a code reserved for cannot-adjudicate is also returned by plain usage errors
STREAM_MISMATCH	SKILL.md names one stream for failures, the tool uses the other
MISSING_PIPE	clause 7 codifies --pipe by name; the CLI rejects it
SCHEMA_NOT_FLAG	clause 7 codifies --schema as a flag, not a subcommand

It obeys the contract it enforces — exit 0 conformant / 1 findings / 2 cannot-adjudicate, plus --pipe, --schema, and an action field on the honest lane. Pointed at a repo with no SKILL.md it exits 2 rather than guessing.

python3 scripts/family_conformance.py ../charactercheck ../dmcheck ../table-kit

Current results

repo	verdict	findings
srdcheck	PASS	—
charactercheck	FAIL	honest-lane overload
dmcheck	FAIL	missing --pipe, honest-lane overload, stream mismatch
table-kit	FAIL	missing --pipe, --schema not a flag, honest-lane overload

Reproduces 4 of the 8 independently-confirmed audit findings plus part of a fifth, with zero per-repo configuration. The highest-impact one is dmcheck's STREAM_MISMATCH: its SKILL.md promises failures print to stderr, and every envelope goes to stdout (stderr is 0 bytes) — so an agent following the docs sees no payload at all. Filed as chaoz23/dmcheck#13.

Scope and safety

All probes are read-only: --help, --schema, a nonexistent flag, and one verb pointed at a path that does not exist. Nothing is written, installed, or mutated.

Python 3.10-safe: tomllib is 3.11+, so [project.scripts] parsing falls back to a regex, verified to produce identical output to tomllib on all four members' manifests.

Known blind spots, stated up front

• no MCP-surface probe — misses table-kit having no MCP server ([TK-008] No MCP server; FAMILY.md clause 7 breach table-kit#22)
• no worked-example runner — misses SKILL.md examples that don't reproduce
• srdcheck's own exit-2 semantic gap (SKILL.md states the wrong meaning and handling for exit 2, omitting refusal contract 1.1 #77) is unreachable by the generic bad-flag probe, since that probe returns 3, which srdcheck correctly documents as a usage error

One caveat worth recording

The first version of this gate passed dmcheck clean. The stream claim wraps across two source lines — "Failures print JSON on" / "stderr, never a traceback." — and a line-based matcher saw neither half. A checker that reports PASS because it structurally cannot perceive the defect is the exact failure class this gate exists to catch.

Fixed by matching at paragraph level, and it is why the gate was validated against known-bad repos rather than trusted on a green result. A gate's first test is whether it can fail.

Adoption

It currently fails 3 of 4 repos, so land it non-blocking first, fix the members, then flip to blocking. HONEST_LANE_OVERLOAD is deliberately medium, not high — it fires on 3 of 4 members, and a rule that flags most of the family at high severity trains readers to ignore it.

Refs #81. Related contract decision: #80.

🤖 Generated with Claude Code