82% of the test suite (933 of 1136 files) runs in no CI job

[vivekchand]

Measured while shipping this: 82% of the test suite runs in no CI job

Wiring tests/test_ingest_doc_drift.py into ci.yml was the fourth time in one session that a guard turned out to be named in no workflow file list, so I measured the class instead of just fixing the instance:

test files in tests/            1136
named in NO workflow file list   933   (82%)

This is not a measurement artifact. There is no catch-all pytest tests/ anywhere in .github/workflows/ — every invocation is an explicit file list, and the workflow comments say so themselves ("a guard added without a line here runs in no job at all"). Spot-checked both directions:

test_api.py                   listed=True
test_workflow_yaml_valid.py   listed=True
test_ac_coverage_guard.py     listed=False   (9 test funcs, 135 lines)
test_adapters.py              listed=False   (15 test funcs, 256 lines)
test_agent_principals.py      listed=False   (17 test funcs, 194 lines)

Those are real suites with real assertions, not helpers or fixtures.

Why it matters more than it sounds. The explicit-file-list convention is deliberate and documented, and there are good reasons for it (job scoping, dependency isolation, runtime). The problem is that the convention has no enforcement, so the failure mode is silent in exactly the direction that hurts: a test you just wrote passes locally, the PR goes green, and the guard never runs again. Every instance I hit this session was a newly added guard — the case where the author is most confident it is protecting something.

I have not added a gate for this, deliberately. A check asserting every tests/test_*.py appears in some workflow list would fail 933 times on day one, which makes it unshippable as a hard gate and useless as a ratchet without a baseline. The shape that would work is the one this repo already uses for known holes elsewhere: record the current 933 as an accepted baseline, fail only when the count grows, and let it ratchet down. That is a real piece of work and a policy decision about CI cost, not something to slip into a docs PR — so I am reporting it rather than doing it.

The four instances I hit and fixed along the way: tests/test_ci_workflow_invocations_are_real.py (#5744), tests/test_field_failure_autoclose.py (#5790), tests/test_obs_gap_openclaw_update_pipeline_5749.py (#5763), and this one.

[vivekchand]

Independently re-measured against origin/main today (2026-09-11) — the claim holds:

tests/test_*.py on origin/main:              1135
distinct test files named across ALL
  .github/workflows/*.yml (not just ci.yml):   201
unlisted:                                      934  (82.3%)

That's consistent with the 1136/933 (82%) this issue reported yesterday — the 1-file drift is just a day of commits, not a discrepancy in the method.

Agree with the issue's own conclusion: this isn't a quick fix. A hard "every test file must appear in some workflow" gate would fail on day one against an 82% baseline, and picking the ratchet mechanism (baseline file, --check mode, which workflows count) is a policy call, not a mechanical patch. Not opening a PR for it this run — no fix is proposed to apply, and the backlog rules cap this session at one PR in flight across all three repos regardless (clawmetry-cloud#2382, still open).

Leaving open, unclaimed, for whoever makes the baseline-ratchet call.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Fj6oPuFbgLLVNLkdX95bxH

---

Generated by Claude Code

[vivekchand]

Closed by #5822, merged and verified live on main. CI-only (scripts/, docs/, .github/, Makefile) — nothing ships in the wheel, so no release is involved.

Verified against origin/main after merge

.github/workflows/ci.yml:146   python3 scripts/check_ci_test_coverage.py --check

$ python3 scripts/check_ci_test_coverage.py --check
CI test-file coverage ratchet OK: 206 listed, 932 unlisted (max allowed: 932)

Proven red, on main, with a single new unlisted test file:

$ printf 'def test_a(): assert True\n' > tests/test_zz_probe_on_main.py
$ python3 scripts/check_ci_test_coverage.py --check
CI test-file coverage ratchet exceeded: 933 unlisted (max 932, over by 1).
exit 1

That one-file sensitivity is the point, and it is why the baseline was tightened from 933 to 932 before merge: merging main brought in three tests wired into jobs since the baseline was taken (206 listed, up from 203), leaving exactly one unit of slack that would have let the next unlisted test through unnoticed.

The shape it took

As proposed, it records the debt and fails only on growth rather than demanding a 933-file cleanup first — a check that fails for pre-existing reasons is one people learn to satisfy without reading. Tightening is explicit via --update-baseline, so the number only moves down and the move is visible in review.

One limit, stated rather than implied

Listed is not executed. A file named in a job that cannot run it — no duckdb installed, say — collects nothing and pytest exits 5: listed, and still never run. This ratchet checks naming only. That distinction cost a CI cycle earlier this week when a guard was wired into Syntax & Lint, which has no duckdb, so it is recorded in the script's docstring and in the blueprint rather than left as a trap.

Recorded as "A test named in no workflow runs in no job" on the Release Verification and Merge Gating blueprint, with four contracts and an ADR for the record-the-debt trade-off.

The 932 that remain are unchanged by this, and nothing forces them down — that is the accepted half of the trade, written down rather than implied.