Skip to content

P2: One real compliance benchmark pack -- CIS Cisco IOS #220

Open

Description

@ARSH871-bot

Phase 2, and the single biggest credibility gap.

Measured: the repository contains zero compliance-benchmark content. A grep for CIS, STIG, PCI-DSS and NIST returns 91 apparent hits, of which 80 are the letters "cis" inside Cisco and 15 are "nist" inside administrator. With word boundaries: none.

Nipper, Tufin, AlgoSec and FireMon all ship regulatory packs. It is most of what a buyer in this category is actually paying for.

CIS Cisco IOS is public and finite. Expressing it in our own policy format has a second benefit: it is the largest real test the format will ever get, and it will find whatever the format cannot say.

Activity

  1. added this to the Sprint 6 milestone on Aug 26, 2026
  2. ARSH871-bot commented on Aug 26, 2026

    @ARSH871-bot
    OwnerAuthor

    @shubhamkataria2005 — a proposal, not a change. Your issue, your call.

    After adding the Phase 1 items, the board looks like this for you:

    #218  Multi-device snapshots        size M
    #220  CIS Cisco IOS benchmark pack  size M   <- this one
    #229  Uncovered devices (in review)
    #238  Attack-path chaining          size L   <- the largest Phase 1 item
    

    That is more than anyone else is carrying, and #238 is explicitly "the largest of the four".

    What I would move, and why it is not just load-balancing

    #220 to me, because it pairs with #235 (Coverage & Certainty Report) into one coherent story rather than two adjacent tasks:

    here is what we checked, here is what we did not, and here is which CIS controls that covers.

    A coverage report that cannot say which standard it covers is half a feature, and a CIS pack nobody reports against is the other half.

    There is also a clean seam: I own the policy format (analysis/policy.py), so expressing CIS as policy rules is writing in the format I designed. Your check consumes it unchanged. Neither of us reaches into the other's file.

    Why you might say no, and that is fine

    policy_compliance is yours, CIS is policy content, and splitting content from the check that runs it is a real cost. If you would rather keep it, say so and I will take something else off the list instead — the point is the total, not this specific item.

    Say the word either way. I have not touched the assignment.

  3. shubhamkataria2005 commented on Aug 27, 2026

    @shubhamkataria2005
    Collaborator

    Take it — with one thing carved back to me.

    Your pairing argument is the right one. "Here is what we checked against a recognised standard, and here is exactly what we did not" is one story, and splitting it across two people makes the seam between the benchmark and the coverage report somebody's coordination problem rather than nobody's.

    The argument for keeping it, and why it loses

    This issue says the pack is "the largest real test the format will ever get, and it will find whatever the format cannot say." That format is mine — docs/policy-rules.md, POLICY_RULES, the two rule kinds, the query mapping. The instinct that its owner should be the one to stress it is not wrong.

    But that is an argument about who fixes what it finds, not about who writes it. Those separate cleanly:

    • Writing CIS Cisco IOS into the format is content work. Reading a public benchmark and expressing each control as a rule needs the format documented, not authored.
    • Whatever it cannot express is a format limitation, and that is mine.

    So: you write it, and the moment you hit a control the format cannot say — a rule about a config setting rather than a flow, which I expect is where it breaks first — that comes back to me as its own issue rather than getting worked around in the pack. I would rather find out the format is too narrow than have someone quietly bend a benchmark to fit it.

    On load, plainly

    I am at 3 sized points against your 2, and #238 is an L I have not started. When #181 lands, three of my issues unblock at once — #196, #195's example, and the honest half of #87 — because all three wait on active_policy() existing on main. That is a real surge rather than a hypothetical one, and it arrives the day you merge.

    So the load table understates my week rather than overstating it, and handing #220 over is the difference between me doing three unblocked things properly and four things at once.

    One thing I would ask you to keep from the issue as filed

    The measurement in it — 91 apparent hits for CIS/STIG/PCI/NIST, of which 80 are the letters "cis" inside Cisco and 15 are "nist" inside administrator, and none with word boundaries. That number is the reason the issue exists, and it is the kind of thing that gets dropped when an issue changes hands. Reassigning, not closing and reopening, so it stays attached.

    Assignment is yours to change — I have not touched it, same as you did not.

  4. shubhamkataria2005 commented on Aug 30, 2026

    @shubhamkataria2005
    Collaborator

    Reassigning to @ARSH871-bot, as agreed on this issue — flagging rather than silently moving it, since the board should show why.

    You proposed taking it because the CIS pack pairs with #235 (Coverage Report) into one story. I agreed, with one thing carved back: whatever the pack cannot express is a format limitation and comes back to me as its own issue, rather than getting worked around inside the pack.

    That agreement is here. The assignment never moved, so the board has been showing it as mine for four days.

    Keep the measurement in the issue body when you pick it up — 91 apparent hits for CIS/STIG/PCI/NIST, of which 80 are the letters "cis" inside Cisco and 15 are "nist" inside administrator, and none with word boundaries. That number is the reason the issue exists and it is the kind of thing that gets dropped when an issue changes hands.

    One thing that has changed since we agreed, and it makes your half easier: #268 ships docs/examples/policy.example.json, a validated example with both rule kinds and a multi-arm rule, tested through load_policy_file() on every run. That is the starting shape for expressing CIS controls, and the test file (tests/test_policy_example_is_valid.py) is the pattern for keeping a benchmark pack from rotting the same way.

  5. added
    phase: 3Market readiness -- what stands between us and a product someone could buy.
    on Sep 10, 2026
  6. shubhamkataria2005 commented on Oct 13, 2026

    @shubhamkataria2005
    Collaborator

    Taking this back, as #398 records. Sized it by measuring before planning on 19 October, and the measurement changes the size.

    Of twelve standard IOS hardening lines, Batfish models three. Logging, NTP and SNMP servers appear in nodeProperties. Password encryption, the HTTP server, SSH version, the login banner and the VTY settings parse cleanly — PASSED, zero warnings — and are modelled nowhere. Detail and the full method are in #400, raised as its own issue as agreed here in August.

    That splits this into three pieces of very different size:

    piece size
    CIS controls that are flow rules — expressible in the format today S
    CIS controls that are Batfish-visible settings — needs a new rule kind M, and it is mine (format)
    CIS controls Batfish cannot see — needs a decision first blocked on #400, needs: all four

    So L is right for the whole, but nothing past the first row should start until #400 is decided. I would propose at planning that this sprint takes the first row plus the honest "cannot assess" card for the third — so the pack never claims coverage it does not have — and the text-inspection question is decided by all four of us.

    The measurement in this issue's body (80 of 91 "CIS" hits being inside "Cisco") still stands and is still the reason the issue exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

phase: 3Market readiness -- what stands between us and a product someone could buy.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions