Skip to content

ci: run the conformance gate on every PR, ratcheted by a baseline - #86

Merged
chaoz23 merged 1 commit into
mainfrom
family/conformance-ci
Aug 21, 2026
Merged

chaoz23 merged 1 commit into
mainfrom
family/conformance-ci

Conversation

@chaoz23

@chaoz23 chaoz23 commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Closes the remaining half of #81. The gate landed in #82, but nothing ran it — it could only catch defects for whoever remembered to invoke it by hand, which is the same failure mode as SKILL.md being prose nobody executes.

Blocking, not advisory

The obvious move is a non-blocking advisory job. That produces a permanently-red check people learn to scroll past — the gate currently reports real findings in three of four members, so it would be red from day one and stay red.

Instead: known findings are waived in family-conformance-baseline.json, and the job fails only on a new divergence. That makes it blocking and useful immediately.

Two rules keep the waiver list a ratchet rather than a place defects go to die:

  1. Every waiver names the issue tracking it. A waiver without a ticket is a hidden defect with extra steps.
  2. A waiver that no longer fires FAILS the gate. Fixing a defect and leaving its waiver behind would silently pre-accept the next regression, so fixes have to shrink the file.
"dmcheck": {
  "MISSING_PIPE": "chaoz23/dmcheck#14",
  "HONEST_LANE_OVERLOAD": "chaoz23/dmcheck#15"
}

Verified in all three directions

A gate that cannot fail is worthless, so I tested it failing:

scenario expected result
known findings waived exit 0 ✅ 0 not in the baseline
a waived finding un-waived (simulated regression) exit 1 ✅ 1 not in the baseline
a waiver that no longer fires exit 1 ✅ STALE BASELINE: … Remove it from the baseline

srdcheck itself passes with no waivers of its own.

Scope

This job audits this repo. The baseline's sibling entries are consulted only when those repos are actually audited, so this job reports nothing for them and cannot go stale on their behalf.

Sibling CI jobs come next — three small PRs, each checking out srdcheck and running the same canonical gate and baseline against itself rather than copying either, per FAMILY.md's pin-by-link rule.

Note

tests/test_event_apply_validation.py::test_deep_json_parse_boundaries_do_not_crash_cli_or_mcp_stdio fails on local Python 3.14 (-32600 == -32700). That is #72, pre-existing and verified identical on untouched main; CI's 3.10–3.13 matrix is unaffected. Full suite otherwise: 711 passed.

🤖 Generated with Claude Code

Closes the remaining half of #81. The gate landed in #82 but nothing ran it,
so it could only catch defects for whoever remembered to invoke it by hand.

Blocking rather than advisory. An advisory job that is permanently red is
noise people learn to scroll past, and the gate currently reports real
findings in three of four members. So known findings are waived in
family-conformance-baseline.json and the job fails only on a NEW divergence.

Two rules make the waiver list a ratchet instead of a place defects go to die:

- every waiver must name the issue tracking it. A waiver without a ticket is
  just a hidden defect with extra steps.
- a waiver that no longer fires FAILS the gate. Fixing a defect and leaving
  its waiver behind would silently pre-accept the next regression, so fixes
  have to shrink the file.

Verified in all three directions, because a gate that cannot fail is
worthless:

  known findings waived            -> exit 0
  a waived finding un-waived       -> exit 1, reported as not in the baseline
  a waiver that no longer fires    -> exit 1, reported as STALE BASELINE

srdcheck itself passes with no waivers. The baseline's sibling entries are
consulted only when those repos are audited, so this job reports nothing for
them; the sibling CI jobs come next and will read the same canonical file
rather than copying it, per FAMILY.md's pin-by-link rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chaoz23
chaoz23 merged commit 27dee98 into main Aug 21, 2026
15 checks passed
@chaoz23
chaoz23 deleted the family/conformance-ci branch August 21, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant