Skip to content

Research: how our policy check compares with formal verification (closes #423) - #431

Merged
patelankeet2 merged 1 commit into
mainfrom
research/423-formal-methods
Nov 2, 2026
Merged

patelankeet2 merged 1 commit into
mainfrom
research/423-formal-methods

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Closes #423. One new file, docs/research/formal-methods.md. No code.

Where we sit: searchFilters is exhaustive and symbolic (BDDs), so it is formal verification, of one filter. The closest industrial match is Microsoft's SecGuru, which checks Azure ACLs against contracts.

Measured on the live container, both directions:

one-host, one-port hole     searchFilters: found in 0.14 s
                            300 random testFilters: 0 hits (expected ~16.6 million to find it)
ACL applied nowhere         PC-000 none (green tick) while traceroute LAN->server TCP/23 is DELIVERED

The second result is a real F-4 gap in my own check. It's raised as #430, and the fix is a separate PR so this one stays docs only.

To review: whether the limits section is fair, and whether you agree control-plane verification is not worth our time. Each measurement has a reproduction at the end of the doc.

searchFilters is exhaustive and symbolic, so it is formal verification of one
filter. Measured both sides: it finds a one-host, one-port hole in one query
that 300 random spot checks missed, and it gives a green tick when the filter
is applied to no interface at all. That second result is raised as #430.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed like a research document should be: reproduced the measurements rather than reading them, and verified the citations live rather than trusting them.

Reproduced both measured examples from scratch, using the recipe at the end of the doc (second interface added, then the one-line change each case describes), not the fixtures you shipped:

unapplied filter, real pipeline:
  AC-001  found  Filter 'acl_in' is defined but never applied
  PC-000  none   No issues found by policy compliance
  traceroute LAN -> 10.20.0.5 TCP/23: DELIVERED_TO_SUBNET

narrow hole, real Batfish:
  searchFilters: 1 row, 10.10.10.77:49152 -> 10.20.0.5:8443 TCP (SYN), 0.24s

Both match the doc exactly. (For what it's worth, I'd tried to reproduce the unapplied-filter traceroute claim independently on a different PR a short while ago using the original single-interface fixture and got NO_ROUTE -- your reproduction note explaining that the second interface is what gives traceroute somewhere to go is the reason, and it's good that it's written down rather than left for the next person to hit the same wall.)

Checked the arithmetic rather than taking "about 6 x 10^-8" on faith: 1/254 x 1/65534 = 1/16,645,636 = 6.01 x 10^-8. Matches. 16.6 million expected trials at the stated 0.16s each is about 30.7 days, matches "around a month." Both numbers in the doc check out.

Verified the four citations with a real claim riding on them, live, not from memory:

  • Brown et al., SIGCOMM 2023 (DOI confirmed, 10.1145/3603269.3604866) -- its own modelling-fidelity lesson is described as "most modeling issues stem from the diversity of configuration constructs rather than from router software bugs," which is exactly what you attribute to it.
  • Jayaraman et al., SIGCOMM 2019 (SecGuru) -- confirmed Microsoft has run it "since 2013," matching your sentence precisely.
  • Beckett et al., SIGCOMM 2017 (Minesweeper) -- confirmed as an SMT encoding of stable states under routing-protocol inputs, matching the table row.
  • Fogel et al., NSDI 2015 -- confirmed as the original Batfish paper, declarative data-plane derivation, matching citation 9.

Did not individually re-verify the remaining five (HSA, VeriFlow, Atomic Predicates, NetKAT, Bryant's BDD paper) beyond confirming they're real, named papers in the table -- none of them carries a specific quantitative claim the way the four above do, so that felt like the right amount of checking rather than reflexively doing all nine.

On the two things you asked to be reviewed hardest:

  • The limits section is fair. All three (path, model fidelity, everything outside one filter) are backed by a real measurement or a real cited finding, not asserted.
  • Agree that control-plane verification is not worth building. The client has one firewall; Minesweeper's whole value is reasoning about which stable state survives a link or BGP-input failure, which doesn't apply here. Correctly scoped as a "not suggested," not a "maybe later."

The one thing #430 already turned into a real PR (#432, reviewed separately) closes exactly the gap this research found -- good that the loop closed rather than the finding sitting only in a doc.

No em-dashes, no AI-tool mentions.

Approving.

@patelankeet2
patelankeet2 merged commit 7ef82b0 into main Nov 2, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the research/423-formal-methods branch November 2, 2026 01:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Research: formal verification methods compared with how Netwise checks policies

2 participants