Repository navigation
Research: how our policy check compares with formal verification (closes #423) - #431
Conversation
searchFilters is exhaustive and symbolic, so it is formal verification of one filter. Measured both sides: it finds a one-host, one-port hole in one query that 300 random spot checks missed, and it gives a green tick when the filter is applied to no interface at all. That second result is raised as #430.
patelankeet2
left a comment
There was a problem hiding this comment.
Reviewed like a research document should be: reproduced the measurements rather than reading them, and verified the citations live rather than trusting them.
Reproduced both measured examples from scratch, using the recipe at the end of the doc (second interface added, then the one-line change each case describes), not the fixtures you shipped:
unapplied filter, real pipeline:
AC-001 found Filter 'acl_in' is defined but never applied
PC-000 none No issues found by policy compliance
traceroute LAN -> 10.20.0.5 TCP/23: DELIVERED_TO_SUBNET
narrow hole, real Batfish:
searchFilters: 1 row, 10.10.10.77:49152 -> 10.20.0.5:8443 TCP (SYN), 0.24s
Both match the doc exactly. (For what it's worth, I'd tried to reproduce the unapplied-filter traceroute claim independently on a different PR a short while ago using the original single-interface fixture and got NO_ROUTE -- your reproduction note explaining that the second interface is what gives traceroute somewhere to go is the reason, and it's good that it's written down rather than left for the next person to hit the same wall.)
Checked the arithmetic rather than taking "about 6 x 10^-8" on faith: 1/254 x 1/65534 = 1/16,645,636 = 6.01 x 10^-8. Matches. 16.6 million expected trials at the stated 0.16s each is about 30.7 days, matches "around a month." Both numbers in the doc check out.
Verified the four citations with a real claim riding on them, live, not from memory:
- Brown et al., SIGCOMM 2023 (DOI confirmed, 10.1145/3603269.3604866) -- its own modelling-fidelity lesson is described as "most modeling issues stem from the diversity of configuration constructs rather than from router software bugs," which is exactly what you attribute to it.
- Jayaraman et al., SIGCOMM 2019 (SecGuru) -- confirmed Microsoft has run it "since 2013," matching your sentence precisely.
- Beckett et al., SIGCOMM 2017 (Minesweeper) -- confirmed as an SMT encoding of stable states under routing-protocol inputs, matching the table row.
- Fogel et al., NSDI 2015 -- confirmed as the original Batfish paper, declarative data-plane derivation, matching citation 9.
Did not individually re-verify the remaining five (HSA, VeriFlow, Atomic Predicates, NetKAT, Bryant's BDD paper) beyond confirming they're real, named papers in the table -- none of them carries a specific quantitative claim the way the four above do, so that felt like the right amount of checking rather than reflexively doing all nine.
On the two things you asked to be reviewed hardest:
- The limits section is fair. All three (path, model fidelity, everything outside one filter) are backed by a real measurement or a real cited finding, not asserted.
- Agree that control-plane verification is not worth building. The client has one firewall; Minesweeper's whole value is reasoning about which stable state survives a link or BGP-input failure, which doesn't apply here. Correctly scoped as a "not suggested," not a "maybe later."
The one thing #430 already turned into a real PR (#432, reviewed separately) closes exactly the gap this research found -- good that the loop closed rather than the finding sitting only in a doc.
No em-dashes, no AI-tool mentions.
Approving.
Closes #423. One new file,
docs/research/formal-methods.md. No code.Where we sit:
searchFiltersis exhaustive and symbolic (BDDs), so it is formal verification, of one filter. The closest industrial match is Microsoft's SecGuru, which checks Azure ACLs against contracts.Measured on the live container, both directions:
The second result is a real F-4 gap in my own check. It's raised as #430, and the fix is a separate PR so this one stays docs only.
To review: whether the limits section is fair, and whether you agree control-plane verification is not worth our time. Each measurement has a reproduction at the end of the doc.