Find out why your pytest tests are flaky. flakewhy runs your suite under controlled variations (same order again, shuffled order, different timezones, different hash seeds), sees which tests flip, then re-runs just those tests to confirm the cause and hands you a one-line repro:
curl -fsSL https://raw.githubusercontent.com/hamarshehmhmd/flakewhy/main/install.sh | shFLAKY tests/test_pricing.py::test_price_in_usd
order-dependent: fails when another test runs before it (leaked state) (2/21 runs failed)
- fails when run after tests/test_settings.py::test_switch_to_eur
- passes when run alone (checked twice)
- confirmed: [culprit, then this test] fails twice
- failure: AssertionError: assert '2.50 EUR' == '2.50 USD'
repro: python3 -m pytest -p no:randomly tests/test_settings.py::test_switch_to_eur tests/test_pricing.py::test_price_in_usd
Knowing a test is flaky is the easy part. Retry plugins and CI dashboards will tell you that it flaked; you still lose an afternoon working out why, because it never fails when you run it on its own.
In Python the answer is usually boring. A study of 22,352 PyPI projects (Gruber et al., ICST 2021) found 59% of flaky tests were caused by test order dependency: some earlier test leaks state (a module global, an env var, a cache, a DB row) and a later one trips over it. Most of the rest came from randomness, network, time and unordered collections. Those causes leave fingerprints you can test for, so flakewhy does that instead of guessing.
| verdict | how flakewhy decides | what you get |
|---|---|---|
order-dependent |
fails in some orders, passes alone (twice), then bisects the tests that ran before it | the culprit test (or smallest group it found) and pytest culprit victim repro |
needs-other-test |
fails alone, passes after some other test | which test it secretly relies on |
timezone-dependent |
flips between TZ values (+14, −3:30, +5:45, −11, UTC), confirmed alone |
failing / passing zones, TZ=... pytest repro |
hashseed-dependent |
flips with PYTHONHASHSEED, confirmed alone |
a seed that fails, PYTHONHASHSEED=... pytest repro |
leaks-between-runs |
passed in the first run(s), failed in every run after, and now fails alone too | which files under your project and ~ changed during the hunt, e.g. a test writing your real ~/.app/settings.json |
nondeterministic |
flips with nothing changed (same order, TZ and seed) | failure rate; it's randomness, timing, threads or something outside |
failing |
fails in every run | listed separately: broken, not flaky |
A verdict is only given after the cause is reproduced on demand. If a test failed once and never again, it's reported as nondeterministic with confirmed: false rather than blamed on whatever happened to vary in that run.
cd your-project
flakewhy # whole suite: baseline + 5 rounds of repeat, shuffle, TZ, hash seed
flakewhy -- tests/test_api.py -k cart # anything after -- goes to pytest
flakewhy -r 10 --factors shuffle # only hunt order dependencies, 10 shuffled runs
flakewhy --format json -o flaky.json # or --format markdownflakewhy itself has no dependencies and doesn't need to be installed in your project. It runs pytest with your interpreter: --python, else $VIRTUAL_ENV, else ./.venv or ./venv, else python3 on PATH. That interpreter needs pytest 7 or newer.
| option | default | |
|---|---|---|
-r, --rounds N |
5 | runs per factor (total runs = 1 + rounds × factors) |
--factors |
repeat,shuffle,tz,hashseed |
what to vary |
--seed N |
1 | base seed for shuffles and hash seeds, so a hunt is repeatable |
--tz |
Pacific/Kiritimati,America/St_Johns,Asia/Kathmandu,Pacific/Pago_Pago,UTC |
zones to try |
--python PATH |
auto | interpreter with your project and pytest |
-C, --cwd DIR |
. |
project directory |
--timeout S |
none | kill a single pytest run after S seconds |
--max-bisect-runs N |
40 | cap on check runs per flaky test |
--format, -o |
text, stdout | text, json or markdown |
--github-summary |
off | append a Markdown table to $GITHUB_STEP_SUMMARY |
Exit codes: 0 no flaky tests found, 1 flaky tests found, 2 couldn't run (collection error, no tests, bad args).
Run it on a schedule rather than on every push; it runs your suite many times.
# .github/workflows/flakewhy.yml
on:
schedule: [{cron: "0 3 * * 1"}]
workflow_dispatch:
jobs:
flakewhy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.12"}
- run: pip install -e . pytest "flakewhy @ git+https://github.com/hamarshehmhmd/flakewhy@v0.1.0"
- run: flakewhy --github-summary -o flakewhy.txt
- if: always()
uses: actions/upload-artifact@v4
with: {name: flakewhy, path: flakewhy.txt}examples/shop has eight tests with the usual bugs in them (leaked module state, local-time dates, set ordering, a random collision, plus one test that's just broken). This repo's CI runs flakewhy on it and checks each verdict.
git clone https://github.com/hamarshehmhmd/flakewhy && cd flakewhy
flakewhy -C examples/shop- A small pytest plugin (copied to a temp dir and loaded with
-p) records every test's outcome to JSON and can run tests in an exact order or a seeded shuffle. Your project's files and config aren't touched. - Runs are serial with
-p no:randomly -p no:xdist -p no:cacheprovider(plus a few other reorder/rerun plugins) andPYTEST_ADDOPTScleared, so only one thing changes per run. Baseline runs usePYTHONHASHSEED=0and noTZ. - For each test that flipped, flakewhy re-runs it alone and in chosen orders to confirm a cause, bisecting the tests that ran before it to find the culprit.
- pytest only. Python 3.9+, tested on Linux and macOS (Windows isn't tested yet).
- It can only explain what shows up in the runs it does. Rare flakes need more rounds (
-r 20); the study above estimates ~170 reruns for 95% confidence a test isn't flaky at all. - A full hunt runs your suite about 21 times plus a few check runs per flaky test, serially. Point it at a slow suite's subset (
-- tests/unit) or use--timeout. - Network, threads and timing all show up as
nondeterministic. It doesn't freeze time or mock the network. - If leftover state (a file in
~, a DB row) already existed before the hunt started, every run fails and the test shows up asfailing.leaks-between-runsonly catches state that appears during the hunt, so run it from a clean state. - Group culprits: if no single earlier test breaks the victim, you get the smallest group bisection found.
- detect-test-pollution bisects to the polluting test once you already know the victim. flakewhy finds the victims first and does this for each one.
- pytest-randomly / pytest-random-order shuffle every run, which surfaces order bugs but doesn't explain them.
- pytest-flakefinder and pytest-repeat rerun tests many times to detect flakiness.
- pytest-rerunfailures retries failures, which keeps CI green but hides the cause.
pip install -e . pytest
pytest -qflakewhy is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later).
Commercial licenses without the AGPL's obligations are available from the author: contact @hamarshehmhmd.
