checkwash

Check whether a code change weakens your tests.

A test can pass after it stops checking something useful. checkwash reads your Git diff and flags known patterns such as deleted assertions, skipped tests, relaxed expectations, and CI checks that stop failing.

- assert total == 105.3
+ assert total > 0

The new assertion accepts many incorrect totals. checkwash can flag changes like this for review, including changes written by coding agents.

Runs locally. No LLM. No network during analysis. Never executes your code.

v0.3.0 replaces five file-wide exemption rules with content-bound ones. In v0.2.13 those rules could reuse a recorded approval for a later change to the same path; v0.3.0 retires their path-only keys. The upgrade notes describe the content-bound replacement, retired keys, and installation fixes.

v0.4.0 extends detection of runtime suppression, collection changes, expectation rewrites and substituted subjects, and recognizes more table refactors. The proofs retain input identity, execution context and conservative handling of unknown code. Release evidence and remaining limits

v0.4.1 repairs Node assertion recognition (#164) and covers Node test-file names, bounded import aliases and lexical shadows. Coverage warnings expose known assertion candidates the scanner cannot represent; they do not prove complete JS/TS coverage. The same assertion contract checks the source, wheel and zipapp. v0.4.1 evidence and limits

v0.4.2 strengthens JS/TS assertion evidence. Test callbacks own their assertions; changing a scalar expected value or loosening positive toBeCloseTo precision reaches the existing detectors. Equivalent scalar spellings remain equivalent across subject checks. This is bounded syntax support, not general JS/TS analysis. v0.4.2 evidence and limits

Try it

You need Python 3.11+ and Git. Download and try the offline examples first.

Windows PowerShell (including 5.1):

curl.exe -LO https://github.com/taipei49314/checkwash/releases/download/v0.4.2/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

PowerShell 5.1 aliases curl to Invoke-WebRequest; use curl.exe as written. macOS/Linux or Git Bash:

curl -LO https://github.com/taipei49314/checkwash/releases/download/v0.4.2/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

Then, from a repository with at least two commits:

python checkwash.pyz check HEAD~1..HEAD

This checks your last commit. Start with a change you already understand. You can also download the file in your browser. For uncommitted changes use python checkwash.pyz check. For a branch review, use python checkwash.pyz check BASE...HEAD to compare from the merge base; BASE..HEAD compares the two named snapshots directly.

Result	What to do
Pass · exit 0	No finding requires blocking under your configuration. Keep running your normal tests and review.
Block · exit 1	Read the finding and the diff. It may be weakened verification or a false positive.
Error · exit 2	Resolve the input or analysis error before relying on the result.

The default threshold is high: a visible warn can still pass. REPAIR_EVIDENCE describes related changes in the same diff; it does not prove a repair is correct.

For JSON/SARIF output and more examples, see the usage guide.

Know the limits

v0.4.2 is alpha. A pass does not prove that a change is correct or honest. Python is the main language supported; JS/TS support covers a limited set of test patterns. Known gaps remain.

Legitimate refactors can be flagged. The frozen historical corpus records 22 blocks out of 60 (36.7%). A separate pre-release source replay records 4/60 blocks, including 2/57 strictly qualified cases; three existing fixture errors remain. These are selected examples, not a general false-positive rate. Try it on your own changes before making it required. Coverage and limitations · Known gaps

Use it in CI

To stop a merge, make the checkwash status check required in your repository's branch rules. Installing the tool or adding a workflow alone does not enforce its verdict.

The recommended Action is pinned to v0.4.1; the CLI above is v0.4.2. The prior-release Action includes the Node assertion repair and coverage diagnostics, but lacks this release's callback and scalar-evidence fixes. Record which version you use. Full setup and exemptions

Copy the GitHub Actions workflow and require the check

Save this as .github/workflows/checkwash.yml:

# .github/workflows/checkwash.yml
on: [pull_request]

permissions:
  contents: read

jobs:
  checkwash:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: taipei49314/checkwash/action@ea726c9cbd172b838bcf42b2c9969759e4ca358e # v0.4.1

After the workflow runs, open Settings → Rules → Rulesets and require the checkwash status context. Keep the job unconditional.

If you administer the repository and have this project's ruleset file, you can instead create the rule with:

gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json

This adds a ruleset; it does not replace existing ones. checkwash doctor can inspect the local workflow, but cannot verify live branch protection.

Why the older Action pin? A release cannot embed its own commit SHA, so the documented Action adopts a verified pin from the prior release. The recommended v0.4.1 Action includes the Node assertion repair and coverage diagnostics, but not the v0.4.2 assertion foundation. A CLI upgrade does not update an existing Action. To verify another trusted release, use git rev-parse 'vX.Y.Z^{commit}'. Action reference

More options and evidence

Install as a CLI or try the included examples

If you already use pipx, install the fixed version:

pipx install checkwash==0.4.2
# or from the release tag:
pipx install git+https://github.com/taipei49314/checkwash@v0.4.2

checkwash check HEAD~1..HEAD
checkwash demo                  # 8 real tampering cases, blocked, offline

checkwash demo replays eight real tampering cases and one honest fix. It illustrates known patterns; it is not a coverage guarantee. You can run the same examples with python checkwash.pyz demo.

Read the measurements and their limits

The historical six-repo sweep recorded 46 / 1800 = 2.56% blocks: 31 false positives (1.72%), 15 legitimate policy blocks (0.83%). The tracked artifacts record engine v0.3.0 (the release commit, swept 2026-09-07) on a corpus used to tune the detectors, not a held-out result.

1.33% of the corpus (24/1800) records opaque production changes. That flag does not establish that each verdict changed or each diff was unanalyzed.

Review methods vary: the three-rater agreement study covers an older 35-diff cohort, not all 46 blocks. The dedicated 22/60 refactor result (benchmarks/refactors/results-latest.json, a dated v0.3.1 snapshot) is a separate population; the general-commit rate does not predict it.

Measurements and source data · Generated results · Failure ledger

Looking for…	Start here
Installation checks, versions and first use	v0.4.2 guide
JSON/SARIF contracts and upgrades	Stability
Assertion support, coverage warnings and artifact checks	Assertion coverage
Required checks and reviewed exemptions	Enterprise setup
Contributing or reporting a problem	Contributing · Issues · Security reports
Readiness for 1.0	Criteria — not met

Related projects: checkwash-corpus and smallestlie hold evaluation work.

The package also includes a limited quality preview for coverage, Ruff and mypy configuration review. Setup and CI adoption guide. Its frozen legacy-byte comparison remains red; publication does not establish natural-case acceptance or effective enforcement.

Alpha pre-release. 22 detectors, 9040 tests in the current source tree. Zero runtime dependencies. Apache-2.0.