fix: conformance gate rejected correct documentation (stream-claim detection) - #84
Merged
Merged
Conversation
The gate decided which stream a SKILL.md claims by asking whether the word "stderr" appeared anywhere in the paragraph. That works only while the documentation is wrong. A correct doc names both streams -- "every envelope prints JSON on stdout; stderr carries only argparse usage errors" -- and the old test saw "stderr", concluded stderr was the claim, and reported STREAM_MISMATCH against a file that was now accurate. It rejected exactly the fix it exists to prompt. A gate that blocks the correct change is worse than one that misses a defect: it teaches you to bypass it. The claim is the stream governed by the printing verb, not any stream mentioned. Detection now matches (prints|emits|writes|outputs) ... (stdout| stderr) within a clause, falling back to the old heuristic when no verb is found. Verified both directions against real files: dmcheck's SKILL.md at origin/main still resolves to "stderr" and still reports STREAM_MISMATCH, while the corrected text resolves to "stdout" and clears -- with MISSING_PIPE and HONEST_LANE_OVERLOAD still reported, so the fix narrows nothing else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
chaoz23
added a commit
that referenced
this pull request
Aug 20, 2026
v2 introduced member classes but stated them only in FAMILY.md's members table. The gate had no way to read them, so the moment v2 merged it began reporting MISSING_PIPE against table-kit -- a transport member that v2 explicitly exempts. The gate contradicted the contract it enforces. That is the same failure as #84 arriving from the other side: a gate that flags a correct state teaches you to bypass it. Members now declare their own class in tool.json: "family_class": "verdict" | "transport" | "adjacent" The declaration lives in the artifact rather than in FAMILY.md's prose. The alternative -- parsing the members table out of the contract -- would couple the gate to markdown formatting, and this parser has already been the defect twice for exactly that kind of brittleness. Gate changes: - CLASS_SURFACES gates --pipe (and MCP, still unprobed) to the verdict class - CLAUSE_1_CLASSES skips honest-lane and stream checks for the adjacent class, which clause 1 does not bind - an undeclared class emits UNDECLARED_CLASS and names the surfaces it did not check, rather than guessing: assuming "verdict" invents breaches for a transport, assuming "transport" hides real ones for a verdict tool - family_class is reported in output tool.json is generated here, so the field is added in gen_tool_json.py and the card regenerated; tests/test_metadata_fresh.py passes. Verified in all three states: srdcheck (verdict) PASS; table-kit undeclared -> UNDECLARED_CLASS + SCHEMA_NOT_FLAG with --pipe correctly unevaluated; table-kit declared transport -> MISSING_PIPE gone, SCHEMA_NOT_FLAG and HONEST_LANE_OVERLOAD correctly retained. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 23, 2026
chaoz23
added a commit
that referenced
this pull request
Aug 25, 2026
* feat: FAMILY.md v2.2 harmonises exit 3; gate can see codes above 3 Two halves of one problem, surfaced by chaoz23/charactercheck#18. CONTRACT. v2.1 said "srdcheck's exit 3 is the family precedent", which was too weak to settle what a member should do when 3 is already spent. v2.2 states the pattern outright: 0/1/2 are the universal verdict contract, >= 3 is the no-verdict taxonomy, and 3 is usage error in EVERY member. A malformed call is the one non-verdict outcome every tool has, so harmonising it means an agent that mis-invokes any member gets the same answer. Further outcomes take 4+. charactercheck therefore moves could-not-retrieve from 3 to 4 rather than taking a different usage code. Decided deliberately over the cheaper option of letting it keep 3 and putting usage on 4: that would have left the family with two spellings for the concept an agent hits most often. GATE. parse_skill capped recorded codes at 0..3, so a documented exit 4 was silently dropped -- and the CLI returning 4 then surfaced as a bogus UNDOCUMENTED_EXIT_CODE against a tool that had documented itself correctly. The gate would have rejected the very refactor the contract now requires. That is the third time this parser has been the defect rather than the tools it checks (line-vs-paragraph in #82, stream-word matching in #84). The shape repeats: the checker silently cannot perceive the thing it is checking. Verified both directions: a synthetic 0/1/2/3/4 SKILL.md now parses all five with honest-lane=2 and usage=4, and all four real SKILL.md files parse unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: drop charactercheck's HONEST_LANE_OVERLOAD waiver charactercheck is harmonising onto exit 3 for usage errors (chaoz23/charactercheck#18). The waiver comes out before the fix lands, as the ratchet requires; the SHA pin keeps charactercheck's CI green in the meantime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #82, found immediately by trying to use it.
The bug
The gate decided which stream a
SKILL.mdclaims for its failure payload by asking whether the wordstderrappeared in the paragraph:That works only while the documentation is wrong.
Fixing chaoz23/dmcheck#13 means writing a paragraph that names both streams, because that is what is actually true:
The old test saw
stderr, concluded stderr was the claim, and reportedSTREAM_MISMATCHagainst a file that had just become accurate. The gate rejected exactly the fix it exists to prompt.A gate that blocks the correct change is worse than one that misses a defect — it teaches you to bypass it.
The fix
The claim is the stream governed by the printing verb, not any stream mentioned:
Falls back to the old heuristic when no verb is found, so nothing that parsed before stops parsing.
Verified both directions against real files
STREAM_MISMATCHSKILL.mdatorigin/mainstderrSKILL.mdwith chaoz23/dmcheck#13 appliedstdoutMISSING_PIPEandHONEST_LANE_OVERLOADstill report on both, so this narrows nothing else.This is the second time this gate's parser was the defect rather than the tools it checks — the first was matching line-by-line against a claim that wrapped across two lines (#82). Both had the same shape: the checker silently could not perceive the thing it was checking. Worth remembering when adding rules to it.
🤖 Generated with Claude Code