ci: expansion to whole test suite - #44
Draft
suraj-subrahmanyan wants to merge 10 commits into
Draft
suraj-subrahmanyan wants to merge 10 commits into
suraj-subrahmanyan wants to merge 10 commits into
Conversation
313 module names exist in both directories, and for the refactored ones the tools file is a CLI wrapper around the lib implementation. The path loop inserted each entry at position 0, reversing its own tuple and putting tools first, so the suite imported wrappers. That alone caused 101 setup errors in tests/harness/scenarios. Asserted by tests/repository/governance/test_import_precedence.py, including the case where PYTHONPATH already orders tools first.
Runs all 883 runnable test files across nine shards and blocks a pull request that turns a passing test red. Several hundred tests are already failing, so the gate compares against a recorded set of known failures rather than a pass rate. That set may only shrink; CI extracts the same file from the base commit and rejects any entry the branch added. - scripts/check-test-census.py every test file collects or declares a lane - scripts/run-test-shard.py runs one shard, emits JUnit - scripts/check-test-baseline.py eight blocking verdicts 18 adversarial tests cover the gate. Nine mutations reintroducing each bypass are caught by a named assertion. ENFORCING is false pending a baseline recorded on CI rather than locally.
The August move relocated tests one directory deeper. Ten shell tests reached harness/ with `cd "$(dirname "$0")/.."`, which now lands in tests/. Four Python tests read fixtures through the stale harness/tests/ copy. Two carried a literal ~/Solar/harness that Path never expands. test_smoke_run.py appended to a tracked fixture on every run, leaving the checkout dirty. test_pm_dispatch.py read task_graph.json directly, which no longer holds node status; it now goes through load_graph. Adds test_neutralized_record_never_consumable: deleting the applied=False check from is_gate_consumable survived all 126 existing gate-ledger tests.
Both were excused on pull requests, so a new secret or shellcheck warning reported and merged. The five findings behind the secret-scan excuse are fixtures for the scrubber itself, and are now pinned in .secret-scan-allowlist by rule, SHA-256 and path. The shellcheck excuse covered one SC2034 on an unused assignment in lib/installer/common.sh, which is removed.
The previous revision was generated locally from a working tree with 65 files staged for deletion, so tests that inspect repository state saw a tree that exists nowhere. That produced 10 missing entries and 8 entries never red on CI. Net 324 -> 326, matching the run's reported count. All 10 additions also fail at the base commit with a clean worktree, so none is a regression from this branch. ENFORCING stays false for one more run: flipping it in the same change that rewrote the baseline would make a clean verdict unfalsifiable.
Red tests made each shard exit 1, putting nine failure annotations on every pull request even when the gate verdict was clean. The step already carried continue-on-error, so the jobs were green and only the annotations were not. A shard reports and the gate judges, so the shard now fails only when it produced no usable JUnit. For the pytest lane, exit 0, 1 and 5 mean the run worked; 2, 3 and 4 stay fatal. Four tests cover the exit paths, each confirmed by mutation.
scripts/check-repo-hygiene.sh has been failing on this branch, on openJiuwen-Solar, and on every pull request against it. It names 21 tracked paths: 19 under a machine source archive, one harness runtime stderr dump, and one raw test workbook. Eleven are zero bytes and three are the same spreadsheet committed three times. Nothing references them, and all 21 remain in git history.
Python smoke, HF and AI Influence smoke, Harness shell smoke and Verification release gate ran 32 files between them, all of which the nine shards already run. Their one unique step, a py_compile of three runtime files, moves to repository-hygiene. autosci-premerge-gate stays: test_autosci_phase_c_premerge_readiness.py asserts the job exists by name. It was failing because 44 generated artifacts under harness/artifacts/autosci/runs/ were tracked, which its own guard forbids. Removing them fixes the job and the test, so that test leaves the baseline. ENFORCING is now true. This change removes four blocking jobs, and leaving the gate advisory while deleting them would drop protection rather than move it. Run 87 named every discrepancy in the local baseline; run 88 reported clean against the corrected one.
The census inspected only Python and shell, then reported 887 files with none unclassified. Extending it raises the count to 916 and surfaces 34 test files that nothing executes: 17 TypeScript files no workflow runs, 9 Electron tests reachable only from a workflow that does not trigger on this branch, 6 Node files referenced nowhere, and 2 PowerShell files of which CI runs one. All 34 now carry an excluded lane and a written reason. They are recorded, not covered. Also removes seven stale duplicates at the root of tests/, each with a live twin deeper in the tree. CI ran both copies, and they had drifted: test-release-checklist.sh was red while test_release_checklist.sh passes. Four were in the baseline; those entries go with them, 325 -> 321.
suraj-subrahmanyan
changed the title
The first enforcing run blocked on one unrecorded fix. That test was red on run 87, is green now, and is green at the base commit. Nothing in this branch touches it, so it is nondeterministic. The baseline cannot hold a green-side flaky test: recorded, it blocks as an unrecorded fix; absent, it blocks as a new failure. Removed because it passes, which the ratchet requires. No fix attempted. The test spawns a subprocess with a 5 second timeout and asserts on the resulting error type, but run 87's failure reason was not captured. GAP-005 records the instance and the quarantine state still owed.
suraj-subrahmanyan
force-pushed
the
ci/test-regression-gate
branch
from
August 17, 2026 21:11
3a54dba to
cc95c83
Compare
Collaborator
|
The PR direction looks correct overall and is likely mergeable. The expanded suite ( Before merge, please add a short operational note on baseline governance:
This helps prevent future baseline drift from becoming an untracked process risk. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
CI already runs. It covers about 30 named files in narrow smoke jobs, out of
887 test files and roughly 7,200 checks. This expands it to all of them, across
nine parallel shards: pytest 0-5, script 0, shell 0-1.
Because several hundred tests are already red, the expanded CI gates on
regressions, not on a pass rate. It records which tests are already failing
by identity, and blocks a pull request that turns a green test red.
The existing smoke jobs are untouched.
The three pieces
scripts/check-test-census.pyEvery test file under
tests/either collects under pytest or declares a laneand a written reason in
tests/ci_lanes.json. Anything that is neither failsthe build. Currently 887 files, 0 unclassified — 714 pytest, 155 shell,
14 script, 4 excluded. This is what stops a test from silently falling out of
CI, which is how 39 tests stopped running in August without anyone noticing.
scripts/run-test-shard.pyRuns one shard and emits JUnit. Suite name carries shard identity, testcase name
carries the lane, so a test keeps its identity if the shards are ever
renumbered. Per-test and per-file timeouts with a process-group kill, so a hung
grandchild cannot hold the output pipe open.
scripts/check-test-baseline.pyThe gate. Eight blocking verdicts: new failure, baseline addition, stale
baseline entry, unrecorded fix, missing shard, unexpected shard, duplicate
identity, unreadable JUnit.
The known-failure list may only shrink. CI extracts the same file from the base
commit and blocks any entry this branch added, so a test cannot be silenced by
adding it to the list. Deleting or renaming a failing test looks like a stale
entry and is blocked too.
Shard counts live in
tests/ci_lanes.json, and both the workflow matrix and thegate read that one file. A matrix that gains or loses a shard fails the gate
rather than quietly changing what CI covers.
Fixes needed to get the suite running
tests/conftest.py— 313 module names exist in bothharness/libandharness/tools. The path loop inserted each entry at position 0, reversing itsown tuple and putting
toolsfirst, so the suite imported thin CLI wrappersinstead of implementations. That alone produced 101 setup errors in
tests/harness/scenarios.20 path and state repairs left behind by the August
tests/move: ten shelltests running
cd "$(dirname "$0")/.."that now land intests/instead ofharness/, four Python tests reading fixtures through the staleharness/tests/copy, two carrying a literal~/Solar/harnessthatPathnever expands, and one appending to a tracked fixture on every run.
GAP-006 — the secret scan and shellcheck block pull requests again. Five
key-shaped literals in secret-scrubbing tests are pinned by SHA-256 rather than
excusing the whole gate.
No file is moved, renamed, or deleted. 37 files, +3,915 / −74.
ENFORCING is "false" on this PR
Two things need a real CI run before enforcement is honest:
one laptop, and nine shards on one machine is not nine shards on nine VMs.
run. Re-running 255 of its 324 entries on a quiet machine showed five that
pass — never product failures, only machine contention. They are not
hand-removed, because a partial correction would imply the other 319 were
checked. The file carries
"provisional": true.So this PR's run reports into the job summary and blocks nothing. The baseline
is regenerated from that run and enforcement flipped in a follow-up. It is one
word.
Verification
18 mutations applied to the gate and the highest-risk control paths, all caught
by a named test through its own assertion rather than an import error. Zero
survived. 24 tests guard the gate itself. Negative controls were replayed
against real generated artifacts rather than hand-authored fixture XML.
Known gaps are tracked with IDs in
docs/testing/COVERAGE-GAPS.md, includingone weakness left unfixed on purpose:
test_pm_dispatch.pydetects removal ofthe graph status write but not corruption of it, and 2 of 3 mutations survive
it.