ci: make the gate real — the tests never ran, and now they must - #9
Closed
Plantucha wants to merge 2 commits into
Closed
ci: make the gate real — the tests never ran, and now they must#9Plantucha wants to merge 2 commits into
Plantucha wants to merge 2 commits into
Conversation
35 source files reformatted by `ruff format`, which replaces black here. Split from the CI-enforcement commit so that change is reviewable without formatting churn in the diff. Also carried, because it lives in one of the same files and cannot be split from it cleanly: tests/test_integration_e2e.py imported `varidex.io.loaders.vcf_loader`, which does not exist. The module is `vcfloader.py` (no underscore) and it exports exactly the two names the tests ask for, `load_vcf` and `load_vcf_chunked`. Seven imports corrected; the test imports were wrong, not the shipped module, so no production code moved. VERIFIED BEHAVIOUR-NEUTRAL BY MEASUREMENT rather than assumption: the failing set is an EXACT match against the recorded baseline before and after the reformat — the ratchet reports equality, not a subset. A formatter that changed behaviour would have moved at least one of 867 tests. Fixing those seven imports raised the visible failure count from 70 to 68 while revealing five downstream failures the ImportError had been masking. That is honest arithmetic, not a regression: the tests were always broken, the import error just stopped anyone seeing how. A Markdown file was excluded — ruff reformats Python snippets inside prose, and rewriting a documentation example is not what this commit is for. varidex/pipeline/phase1_enhancement.py is also excluded: it is not valid Python (its function body was commented out, leaving a def with no block), so no formatter can parse it. Recorded in scripts/known-broken-syntax.txt.
…ther than lies
THE PROBLEM WAS NOT continue-on-error. THE TESTS NEVER RAN.
`test.yml` installed with:
pip install -e . # pyproject.toml had NO [project] table
pip install -r requirements-test.txt # that file was never committed
Both lines fail. The install step had no continue-on-error, so the job died
there and every pytest step after it was UNREACHABLE — the
`continue-on-error: true` on those steps was irrelevant, because they never
executed. Every test.yml run in this repository's history is a failure; the most
recent was 2026-02-09, six months ago. A repo can look maintained while nothing
has been examined for half a year.
MEASURED FIRST (2026-08-28), before any gate was designed:
867 tests: 789 passing, 68 failing, 10 skipped
coverage 27% (floor recorded at 26)
mypy 41 errors in 16 files, 110 files checked
ruff 1316 findings
Test dependencies were DERIVED, not guessed: tqdm, psutil and click were each
discovered by a collection failure and declared by nothing; pysam and pyyaml
accounted for four more failures that were dependency-driven rather than logic.
PACKAGING REPAIR — nothing could be gated until this worked. Added a minimal
[project] table and a setuptools backend so `pip install -e .` succeeds, and
committed requirements-test.txt with its provenance in the header.
THE GATE IS NOT green-only, because the suite is not green: 68 failures are
recorded in tests/known-failures.txt under a TWO-DIRECTION ratchet.
* a failure not in the baseline -> RED. New breakage cannot enter.
* a baseline entry that starts PASSING -> RED, until it is delisted.
The second direction is the point: without it a baseline rots into a permanent
excuse list and a fixed test silently keeps its licence to fail. Both directions
were verified against the real suite, not only against selftest fixtures.
EVERY CHECK DISTINGUISHES EXAMINED-AND-CLEAN FROM NEVER-EXAMINED, with a
distinct exit code (2) for refusal:
* pytest — a collection error, zero tests, or output with no pass/fail
counts REFUSES instead of passing.
* mypy — prints neither "Found N errors" nor "Success" when it dies. That
silence REFUSES. It was dying: mypy.ini pinned python_version to
3.9, which current mypy rejects outright, and a syntax error
stopped it after one file. Nothing reported this for months.
* ruff — no count line REFUSES.
* coverage — no TOTAL line REFUSES rather than reading as 0.
* syntax — every tracked .py must parse; a file that cannot be parsed is
invisible to ruff, mypy and pytest alike.
* install — fails loudly as INSTALL FAILED, never as a test result.
check.sh IS the CI. Run it locally and you have run the job.
WORKFLOWS 9 -> 3: ci (this), security, release. Removed ci.yml (0 bytes),
ci-enhanced.yml, test.yml, badges.yml, cd.yml, dependabot.yml, and
dependency-updates.yml — the last had failed weekly for six weeks unattended
until GitHub auto-disabled it for inactivity. Security Scanning was disabled the
same way and has been re-enabled.
ONE FILE IS LEFT BROKEN DELIBERATELY. varidex/pipeline/phase1_enhancement.py is
not valid Python: the body of add_phase1_codes_to_pipeline was commented out,
leaving a def with no block. It is orphaned — orchestrator_v2 defines its own
apply_phase1_enhancements and does not import it. The original implementation is
gone, and inventing one would be fabrication rather than repair, so it is
recorded in scripts/known-broken-syntax.txt where the syntax gate blocks any NEW
unparseable file while it stands.
The 1316 ruff findings and 41 mypy errors are ratcheted, not auto-fixed: mass
-fixing would bury this change under a whole-repo rewrite. They can only go down.
| EXIT_OK, EXIT_FAIL, EXIT_REFUSE = 0, 1, 2 | ||
|
|
||
| _ANSI = re.compile(r"\x1b\[[0-9;]*m") | ||
| _SUMMARY = re.compile(r"^=+ .*?(\d+) failed.*?(\d+) passed", re.M) |
Owner
Author
|
Withdrawn at owner's direction. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The finding that changes the framing
The problem was never
continue-on-error. The tests have not run since 2026-02-09.test.ymlinstalled with:Both lines fail. The install step had no
continue-on-error, so the job died there and every pytest step after it was unreachable — thecontinue-on-error: trueon those steps was irrelevant, because they never executed. Everytest.ymlrun in this repository's history is a failure. The repo has looked maintained for six months while nothing was examined.Measured first, before any gate was designed
Test dependencies were derived, not guessed:
tqdm,psutilandclickwere each discovered by a collection failure and declared by nothing;pysamandpyyamlaccounted for four more failures that were dependency-driven rather than logic.Packaging repair — nothing could be gated until this worked
Minimal
[project]table plus a setuptools backend sopip install -e .succeeds, andrequirements-test.txtcommitted with its provenance in the header.The gate is not green-only, because the suite is not green
68 failures are recorded in
tests/known-failures.txtunder a two-direction ratchet:The second direction is the point: without it a baseline rots into a permanent excuse list, and a fixed test silently keeps its licence to fail. Both directions were verified against the real suite, not only against selftest fixtures.
Every check distinguishes examined-and-clean from never-examined
Each returns a distinct exit code (2) for refusal:
Found N errorsnorSuccess— it was dying:mypy.inipinnedpython_versionto 3.9, which current mypy rejects, and a syntax error stopped it after one fileTOTALline (rather than reading as 0).pyfails to parseINSTALL FAILED, never as a test resultcheck.shIS the CI. Run it locally and you have run the job.Workflows 9 → 3
ci(new),security,release. Removedci.yml(0 bytes),ci-enhanced.yml,test.yml,badges.yml,cd.yml,dependabot.yml, anddependency-updates.yml— the last had failed weekly for six weeks unattended until GitHub auto-disabled it. Security Scanning was disabled the same way and I have re-enabled it.One file left broken deliberately
varidex/pipeline/phase1_enhancement.pyis not valid Python — the body ofadd_phase1_codes_to_pipelinewas commented out, leaving adefwith no block. It is orphaned:orchestrator_v2defines its ownapply_phase1_enhancementsand does not import it. The original implementation is gone, and inventing one would be fabrication rather than repair, so it is recorded inscripts/known-broken-syntax.txtwhere the syntax gate blocks any new unparseable file while it stands.Open question for you
SPDX headers. Tepna's convention is Apache-2.0 headers on every source file; VariDex is AGPL-3.0, so that convention does not port as-is. I have added no headers. If you want them, they should read
AGPL-3.0-or-later— your call, not one to make by analogy.Review notes