You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_draft_policy.py tests fail intermittently under a full live-Batfish suite run (not reproducible in isolation) #448
Three separate sightings now, each a different specific test in or near tests/test_draft_policy.py, each confirmed not caused by the PR it showed up under.
#442's own PR body (full live run against real Batfish): test_draft_policy.py::test_flipping_one_decision_produces_a_real_finding and test_parse_coverage.py::test_real_vendor_asa_caveat_states_how_much_was_read failed once. Both passed on re-run, twice each, on that branch and on main, and both passed in a combined live run of sixteen audit fixes together. Written up as "intermittent under a full live run, not caused by this PR."
My own review of #445, independently, before reading #442's note above: a full-suite run failed test_draft_policy.py::test_a_draft_decided_as_it_is_today_passes_the_real_check[rtr-us5-insecure]. Passed cleanly alone, passed cleanly as the whole file (46/46), and a second full-suite rerun came back completely clean (2253 passed, 13 skipped, 0 failed). #445's diff doesn't touch policy drafting or Batfish setup at all.
Three different tests, two different PRs, one shared trait: test_draft_policy.py (or something it depends on through real Batfish) is flaky specifically under a full-suite run, and reliable when run alone or as its own file. Nobody has reproduced a cause yet, only the pattern -- noted here as a tracking issue rather than a diagnosis, so the next person who hits it has somewhere to add a fourth data point instead of re-discovering the pattern from scratch.
Worth checking once there's enough data: resource contention with Batfish/Docker under a long sequential run, a shared fixture or temp path that isn't cleaned up between files, or ordering sensitivity if pytest's collection order changes between runs. Not yet narrowed down; this issue exists to collect sightings until it is.
No F-1/F-4 risk: this is CI noise, not a product behaviour. Low priority, not blocking.
Activity