Skip to content

test_draft_policy.py tests fail intermittently under a full live-Batfish suite run (not reproducible in isolation) #448

Open

Description

@patelankeet2

Three separate sightings now, each a different specific test in or near tests/test_draft_policy.py, each confirmed not caused by the PR it showed up under.

#442's own PR body (full live run against real Batfish): test_draft_policy.py::test_flipping_one_decision_produces_a_real_finding and test_parse_coverage.py::test_real_vendor_asa_caveat_states_how_much_was_read failed once. Both passed on re-run, twice each, on that branch and on main, and both passed in a combined live run of sixteen audit fixes together. Written up as "intermittent under a full live run, not caused by this PR."

My own review of #445, independently, before reading #442's note above: a full-suite run failed test_draft_policy.py::test_a_draft_decided_as_it_is_today_passes_the_real_check[rtr-us5-insecure]. Passed cleanly alone, passed cleanly as the whole file (46/46), and a second full-suite rerun came back completely clean (2253 passed, 13 skipped, 0 failed). #445's diff doesn't touch policy drafting or Batfish setup at all.

Three different tests, two different PRs, one shared trait: test_draft_policy.py (or something it depends on through real Batfish) is flaky specifically under a full-suite run, and reliable when run alone or as its own file. Nobody has reproduced a cause yet, only the pattern -- noted here as a tracking issue rather than a diagnosis, so the next person who hits it has somewhere to add a fourth data point instead of re-discovering the pattern from scratch.

Worth checking once there's enough data: resource contention with Batfish/Docker under a long sequential run, a shared fixture or temp path that isn't cleaned up between files, or ordering sensitivity if pytest's collection order changes between runs. Not yet narrowed down; this issue exists to collect sightings until it is.

No F-1/F-4 risk: this is CI noise, not a product behaviour. Low priority, not blocking.

Activity

  1. added
    bugSomething isn't working
    phase: 3Market readiness -- what stands between us and a product someone could buy.
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingphase: 3Market readiness -- what stands between us and a product someone could buy.tests

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions