Skip to content
hamarshehmhmdPublic

About

Find out why your pytest tests are flaky: test order (with the culprit test), timezone, hash seed or real nondeterminism, each confirmed with a one-line repro.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

flakewhy

CI License: AGPL-3.0-or-later Python 3.9+ pytest 7+ zero dependencies

Find out why your pytest tests are flaky. flakewhy runs your suite under controlled variations (same order again, shuffled order, different timezones, different hash seeds), sees which tests flip, then re-runs just those tests to confirm the cause and hands you a one-line repro:

flakewhy demo

curl -fsSL https://raw.githubusercontent.com/hamarshehmhmd/flakewhy/main/install.sh | sh
FLAKY   tests/test_pricing.py::test_price_in_usd
        order-dependent: fails when another test runs before it (leaked state)  (2/21 runs failed)
        - fails when run after tests/test_settings.py::test_switch_to_eur
        - passes when run alone (checked twice)
        - confirmed: [culprit, then this test] fails twice
        - failure: AssertionError: assert '2.50 EUR' == '2.50 USD'
        repro: python3 -m pytest -p no:randomly tests/test_settings.py::test_switch_to_eur tests/test_pricing.py::test_price_in_usd

Why

Knowing a test is flaky is the easy part. Retry plugins and CI dashboards will tell you that it flaked; you still lose an afternoon working out why, because it never fails when you run it on its own.

In Python the answer is usually boring. A study of 22,352 PyPI projects (Gruber et al., ICST 2021) found 59% of flaky tests were caused by test order dependency: some earlier test leaks state (a module global, an env var, a cache, a DB row) and a later one trips over it. Most of the rest came from randomness, network, time and unordered collections. Those causes leave fingerprints you can test for, so flakewhy does that instead of guessing.

What it finds

verdict how flakewhy decides what you get
order-dependent fails in some orders, passes alone (twice), then bisects the tests that ran before it the culprit test (or smallest group it found) and pytest culprit victim repro
needs-other-test fails alone, passes after some other test which test it secretly relies on
timezone-dependent flips between TZ values (+14, −3:30, +5:45, −11, UTC), confirmed alone failing / passing zones, TZ=... pytest repro
hashseed-dependent flips with PYTHONHASHSEED, confirmed alone a seed that fails, PYTHONHASHSEED=... pytest repro
leaks-between-runs passed in the first run(s), failed in every run after, and now fails alone too which files under your project and ~ changed during the hunt, e.g. a test writing your real ~/.app/settings.json
nondeterministic flips with nothing changed (same order, TZ and seed) failure rate; it's randomness, timing, threads or something outside
failing fails in every run listed separately: broken, not flaky

A verdict is only given after the cause is reproduced on demand. If a test failed once and never again, it's reported as nondeterministic with confirmed: false rather than blamed on whatever happened to vary in that run.

Usage

cd your-project
flakewhy                               # whole suite: baseline + 5 rounds of repeat, shuffle, TZ, hash seed
flakewhy -- tests/test_api.py -k cart  # anything after -- goes to pytest
flakewhy -r 10 --factors shuffle       # only hunt order dependencies, 10 shuffled runs
flakewhy --format json -o flaky.json   # or --format markdown

flakewhy itself has no dependencies and doesn't need to be installed in your project. It runs pytest with your interpreter: --python, else $VIRTUAL_ENV, else ./.venv or ./venv, else python3 on PATH. That interpreter needs pytest 7 or newer.

option default
-r, --rounds N 5 runs per factor (total runs = 1 + rounds × factors)
--factors repeat,shuffle,tz,hashseed what to vary
--seed N 1 base seed for shuffles and hash seeds, so a hunt is repeatable
--tz Pacific/Kiritimati,America/St_Johns,Asia/Kathmandu,Pacific/Pago_Pago,UTC zones to try
--python PATH auto interpreter with your project and pytest
-C, --cwd DIR . project directory
--timeout S none kill a single pytest run after S seconds
--max-bisect-runs N 40 cap on check runs per flaky test
--format, -o text, stdout text, json or markdown
--github-summary off append a Markdown table to $GITHUB_STEP_SUMMARY

Exit codes: 0 no flaky tests found, 1 flaky tests found, 2 couldn't run (collection error, no tests, bad args).

In CI

Run it on a schedule rather than on every push; it runs your suite many times.

# .github/workflows/flakewhy.yml
on:
  schedule: [{cron: "0 3 * * 1"}]
  workflow_dispatch:
jobs:
  flakewhy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12"}
      - run: pip install -e . pytest "flakewhy @ git+https://github.com/hamarshehmhmd/flakewhy@v0.1.0"
      - run: flakewhy --github-summary -o flakewhy.txt
      - if: always()
        uses: actions/upload-artifact@v4
        with: {name: flakewhy, path: flakewhy.txt}

Try it on the example

examples/shop has eight tests with the usual bugs in them (leaked module state, local-time dates, set ordering, a random collision, plus one test that's just broken). This repo's CI runs flakewhy on it and checks each verdict.

git clone https://github.com/hamarshehmhmd/flakewhy && cd flakewhy
flakewhy -C examples/shop

How it works

  • A small pytest plugin (copied to a temp dir and loaded with -p) records every test's outcome to JSON and can run tests in an exact order or a seeded shuffle. Your project's files and config aren't touched.
  • Runs are serial with -p no:randomly -p no:xdist -p no:cacheprovider (plus a few other reorder/rerun plugins) and PYTEST_ADDOPTS cleared, so only one thing changes per run. Baseline runs use PYTHONHASHSEED=0 and no TZ.
  • For each test that flipped, flakewhy re-runs it alone and in chosen orders to confirm a cause, bisecting the tests that ran before it to find the culprit.

Limits

  • pytest only. Python 3.9+, tested on Linux and macOS (Windows isn't tested yet).
  • It can only explain what shows up in the runs it does. Rare flakes need more rounds (-r 20); the study above estimates ~170 reruns for 95% confidence a test isn't flaky at all.
  • A full hunt runs your suite about 21 times plus a few check runs per flaky test, serially. Point it at a slow suite's subset (-- tests/unit) or use --timeout.
  • Network, threads and timing all show up as nondeterministic. It doesn't freeze time or mock the network.
  • If leftover state (a file in ~, a DB row) already existed before the hunt started, every run fails and the test shows up as failing. leaks-between-runs only catches state that appears during the hunt, so run it from a clean state.
  • Group culprits: if no single earlier test breaks the victim, you get the smallest group bisection found.

Related tools

Development

pip install -e . pytest
pytest -q

License

flakewhy is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later).

Commercial licenses without the AGPL's obligations are available from the author: contact @hamarshehmhmd.

About

Find out why your pytest tests are flaky: test order (with the culprit test), timezone, hash seed or real nondeterminism, each confirmed with a one-line repro.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages