Skip to content

ci: make the gate real — the tests never ran, and now they must - #9

Closed
Plantucha wants to merge 2 commits into
mainfrom
claude/ci-enforcement
Closed

ci: make the gate real — the tests never ran, and now they must#9
Plantucha wants to merge 2 commits into
mainfrom
claude/ci-enforcement

Conversation

@Plantucha

Copy link
Copy Markdown
Owner

The finding that changes the framing

The problem was never continue-on-error. The tests have not run since 2026-02-09.

test.yml installed with:

pip install -e .                      # pyproject.toml had NO [project] table
pip install -r requirements-test.txt  # that file was never committed

Both lines fail. The install step had no continue-on-error, so the job died there and every pytest step after it was unreachable — the continue-on-error: true on those steps was irrelevant, because they never executed. Every test.yml run in this repository's history is a failure. The repo has looked maintained for six months while nothing was examined.

⚠️ I could not confirm the mechanism from CI logs — GitHub's retention has expired them (HTTP 410) — so that is inference from the file state, strongly supported by the uniform failure record.

Measured first, before any gate was designed

tests 867: 789 passing, 68 failing, 10 skipped
coverage 27% (floor recorded at 26)
mypy 41 errors in 16 files, 110 files checked
ruff 1316 findings

Test dependencies were derived, not guessed: tqdm, psutil and click were each discovered by a collection failure and declared by nothing; pysam and pyyaml accounted for four more failures that were dependency-driven rather than logic.

Packaging repair — nothing could be gated until this worked

Minimal [project] table plus a setuptools backend so pip install -e . succeeds, and requirements-test.txt committed with its provenance in the header.

The gate is not green-only, because the suite is not green

68 failures are recorded in tests/known-failures.txt under a two-direction ratchet:

  • a failure not in the baseline → RED. New breakage cannot enter.
  • a baseline entry that starts passingRED, until it is delisted.

The second direction is the point: without it a baseline rots into a permanent excuse list, and a fixed test silently keeps its licence to fail. Both directions were verified against the real suite, not only against selftest fixtures.

Every check distinguishes examined-and-clean from never-examined

Each returns a distinct exit code (2) for refusal:

gate refuses when
pytest collection error, zero tests, or no pass/fail counts
mypy it prints neither Found N errors nor Successit was dying: mypy.ini pinned python_version to 3.9, which current mypy rejects, and a syntax error stopped it after one file
ruff no count line
coverage no TOTAL line (rather than reading as 0)
syntax any tracked .py fails to parse
install fails loudly as INSTALL FAILED, never as a test result

check.sh IS the CI. Run it locally and you have run the job.

Workflows 9 → 3

ci (new), security, release. Removed ci.yml (0 bytes), ci-enhanced.yml, test.yml, badges.yml, cd.yml, dependabot.yml, and dependency-updates.yml — the last had failed weekly for six weeks unattended until GitHub auto-disabled it. Security Scanning was disabled the same way and I have re-enabled it.

One file left broken deliberately

varidex/pipeline/phase1_enhancement.py is not valid Python — the body of add_phase1_codes_to_pipeline was commented out, leaving a def with no block. It is orphaned: orchestrator_v2 defines its own apply_phase1_enhancements and does not import it. The original implementation is gone, and inventing one would be fabrication rather than repair, so it is recorded in scripts/known-broken-syntax.txt where the syntax gate blocks any new unparseable file while it stands.

Open question for you

SPDX headers. Tepna's convention is Apache-2.0 headers on every source file; VariDex is AGPL-3.0, so that convention does not port as-is. I have added no headers. If you want them, they should read AGPL-3.0-or-later — your call, not one to make by analogy.

Review notes

  • Two commits: formatting (mechanical, verified behaviour-neutral by measurement — the failing set is an exact match before and after) and the CI work.
  • The 1316 ruff findings and 41 mypy errors are ratcheted, not auto-fixed — mass-fixing would bury this change under a whole-repo rewrite. They can only go down.
  • No Kodiak on this repo, so this waits for you.

35 source files reformatted by `ruff format`, which replaces black here. Split
from the CI-enforcement commit so that change is reviewable without formatting
churn in the diff.

Also carried, because it lives in one of the same files and cannot be split from
it cleanly: tests/test_integration_e2e.py imported
`varidex.io.loaders.vcf_loader`, which does not exist. The module is
`vcfloader.py` (no underscore) and it exports exactly the two names the tests
ask for, `load_vcf` and `load_vcf_chunked`. Seven imports corrected; the test
imports were wrong, not the shipped module, so no production code moved.

VERIFIED BEHAVIOUR-NEUTRAL BY MEASUREMENT rather than assumption: the failing
set is an EXACT match against the recorded baseline before and after the
reformat — the ratchet reports equality, not a subset. A formatter that changed
behaviour would have moved at least one of 867 tests.

Fixing those seven imports raised the visible failure count from 70 to 68 while
revealing five downstream failures the ImportError had been masking. That is
honest arithmetic, not a regression: the tests were always broken, the import
error just stopped anyone seeing how.

A Markdown file was excluded — ruff reformats Python snippets inside prose, and
rewriting a documentation example is not what this commit is for.

varidex/pipeline/phase1_enhancement.py is also excluded: it is not valid Python
(its function body was commented out, leaving a def with no block), so no
formatter can parse it. Recorded in scripts/known-broken-syntax.txt.
…ther than lies

THE PROBLEM WAS NOT continue-on-error. THE TESTS NEVER RAN.

`test.yml` installed with:
    pip install -e .                      # pyproject.toml had NO [project] table
    pip install -r requirements-test.txt  # that file was never committed
Both lines fail. The install step had no continue-on-error, so the job died
there and every pytest step after it was UNREACHABLE — the
`continue-on-error: true` on those steps was irrelevant, because they never
executed. Every test.yml run in this repository's history is a failure; the most
recent was 2026-02-09, six months ago. A repo can look maintained while nothing
has been examined for half a year.

MEASURED FIRST (2026-08-28), before any gate was designed:
    867 tests: 789 passing, 68 failing, 10 skipped
    coverage 27% (floor recorded at 26)
    mypy 41 errors in 16 files, 110 files checked
    ruff 1316 findings
Test dependencies were DERIVED, not guessed: tqdm, psutil and click were each
discovered by a collection failure and declared by nothing; pysam and pyyaml
accounted for four more failures that were dependency-driven rather than logic.

PACKAGING REPAIR — nothing could be gated until this worked. Added a minimal
[project] table and a setuptools backend so `pip install -e .` succeeds, and
committed requirements-test.txt with its provenance in the header.

THE GATE IS NOT green-only, because the suite is not green: 68 failures are
recorded in tests/known-failures.txt under a TWO-DIRECTION ratchet.
  * a failure not in the baseline         -> RED. New breakage cannot enter.
  * a baseline entry that starts PASSING  -> RED, until it is delisted.
The second direction is the point: without it a baseline rots into a permanent
excuse list and a fixed test silently keeps its licence to fail. Both directions
were verified against the real suite, not only against selftest fixtures.

EVERY CHECK DISTINGUISHES EXAMINED-AND-CLEAN FROM NEVER-EXAMINED, with a
distinct exit code (2) for refusal:
  * pytest    — a collection error, zero tests, or output with no pass/fail
                counts REFUSES instead of passing.
  * mypy      — prints neither "Found N errors" nor "Success" when it dies. That
                silence REFUSES. It was dying: mypy.ini pinned python_version to
                3.9, which current mypy rejects outright, and a syntax error
                stopped it after one file. Nothing reported this for months.
  * ruff      — no count line REFUSES.
  * coverage  — no TOTAL line REFUSES rather than reading as 0.
  * syntax    — every tracked .py must parse; a file that cannot be parsed is
                invisible to ruff, mypy and pytest alike.
  * install   — fails loudly as INSTALL FAILED, never as a test result.

check.sh IS the CI. Run it locally and you have run the job.

WORKFLOWS 9 -> 3: ci (this), security, release. Removed ci.yml (0 bytes),
ci-enhanced.yml, test.yml, badges.yml, cd.yml, dependabot.yml, and
dependency-updates.yml — the last had failed weekly for six weeks unattended
until GitHub auto-disabled it for inactivity. Security Scanning was disabled the
same way and has been re-enabled.

ONE FILE IS LEFT BROKEN DELIBERATELY. varidex/pipeline/phase1_enhancement.py is
not valid Python: the body of add_phase1_codes_to_pipeline was commented out,
leaving a def with no block. It is orphaned — orchestrator_v2 defines its own
apply_phase1_enhancements and does not import it. The original implementation is
gone, and inventing one would be fabrication rather than repair, so it is
recorded in scripts/known-broken-syntax.txt where the syntax gate blocks any NEW
unparseable file while it stands.

The 1316 ruff findings and 41 mypy errors are ratcheted, not auto-fixed: mass
-fixing would bury this change under a whole-repo rewrite. They can only go down.
EXIT_OK, EXIT_FAIL, EXIT_REFUSE = 0, 1, 2

_ANSI = re.compile(r"\x1b\[[0-9;]*m")
_SUMMARY = re.compile(r"^=+ .*?(\d+) failed.*?(\d+) passed", re.M)
@Plantucha

Copy link
Copy Markdown
Owner Author

Withdrawn at owner's direction.

@Plantucha Plantucha closed this Aug 28, 2026
@Plantucha
Plantucha deleted the claude/ci-enforcement branch August 28, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants