config-auditor

Checks Cisco IOS running-configs against a set of security rules and reports what failed, where, and why it matters.

Rules live in a JSON file, not in the code, so a new check is a new JSON object — no Python changes required. Parsing is done with regular expressions and the Python standard library only; there are no third-party dependencies.

Requirements

Python 3.10 or newer. Nothing else.

Usage

python3 audit.py samples/
python3 audit.py device1.cfg device2.cfg
python3 audit.py samples/ --json
python3 audit.py samples/ --rules my-rules.json
python3 audit.py samples/ --fail-on high
python3 audit.py configs/ --fail-on high --fail-on-not-evaluated

Which files get audited

Naming a file and pointing at a directory are two different acts, and the tool treats them differently.

A file named on the command line is always audited, whatever its name or contents. The user has said this is a configuration; the tool is in no position to argue, and refusing would leave no way to check a fragment or an oddly named export.

A file merely found inside a directory has to earn its place. Directories are searched recursively, so a tree of per-site backups is audited in full, but a found file is only picked up when it:

• ends in .cfg, .conf or .txt, and
• carries a recognisable IOS command — version 15.2, hostname, interface GigabitEthernet0/1, line vty 0 4, the show running-config header, and a few others — within the first 256 KB, or anywhere in a file smaller than that.

The limit is on bytes rather than lines so that no real configuration can hide its first recognisable command behind a long comment or banner header, while a large text file the tool stumbles over still costs a bounded read.

Directories whose name starts with a dot are not entered at all, which keeps .git and .venv out of the search.

The second condition exists because .txt is a weak claim. Without it, pointing the tool at a project directory audits its licence, changelog and notes as though they were configurations and reports confident nonsense about each. The patterns are anchored at both ends where the command occupies a whole line, so a sentence of prose beginning "interface between the two parts…" is not mistaken for an interface declaration.

Anything passed over is named in the report, in the text summary and in the JSON skipped list, with the reason. A guess that quietly dropped a real configuration would be the same false confidence this tool exists to remove, so the guess is made in the open. A rejected file does not fail the run; it is an expected outcome of searching a directory, not an error.

Files are decoded as strict UTF-8. A file that cannot be decoded is reported and skipped rather than scanned with replacement characters — an undecodable byte can sit in the middle of the very command a rule looks for, and replacing it silently would turn a missed match into a clean report. The other files are still audited, and the run ends with exit code 2.

The check that decides whether to look at a file decodes leniently, unlike the audit itself. It settles a routing question rather than a factual one, so a configuration containing a bad byte passes the search, reaches the strict read, and is reported as unreadable instead of vanishing from the results.

Exit codes

Code	Meaning
0	nothing at or above the failure threshold
1	at least one finding at or above the failure threshold
2	the tool could not run, or could not read part of its input

--fail-on {high,medium,low} sets that threshold; the default is low, which means any finding at all fails the run.

The switch changes only the exit code. Every finding is still printed in full, whatever the threshold — a run with --fail-on high reports the medium and low findings it saw, it just does not fail because of them. That keeps the report honest while letting a pipeline block on serious problems and merely record the rest.

--fail-on-not-evaluated adds a second condition: the run also fails when any rule could not be evaluated at all.

It exists because a green pipeline is a claim, and without this switch the tool can make that claim too cheaply. A rule with a block scope is skipped when the configuration holds no matching section, so a configuration captured in part — no line vty block in it — passes every rule that was left, reports no findings, and exits 0. Nothing was wrong with what was checked; the problem is how little was checked. Turning the switch on makes a half-run audit fail instead of quietly passing, which is what you want in CI, where nobody reads the output of a green build.

Exit code 2 wins over 1: if a file could not be read and findings were reported, the run exits 2, because "these results are incomplete" outranks "these results contain problems".

Example

samples/branch-router.cfg
------------------------------------------------------------------------------
MEDIUM  line  106  vty-no-access-class  [line vty 5 15]
        VTY lines have no access-class
        Without an inbound ACL the management plane accepts connections from
        any source that can reach the device address, which turns every
        reachable network into an attack surface.
        Fix: access-class MGMT-HOSTS in

LOW     line    -  no-service-password-encryption
        'service password-encryption' is not enabled
        ...
        Fix: service password-encryption

2 finding(s): 0 high, 1 medium, 1 low

A dash instead of a line number means the finding is about something that is missing from the configuration rather than something that is present in it.

Findings versus checks that did not apply

The tool distinguishes "this is wrong" from "this I could not check", and never lets the second silently look like a pass.

A rule with a block scope can only be judged inside the sections its scope selects. When a configuration contains no such section — a device with no line vty block, or a config that was captured in part — the rule is not reported as satisfied and not reported as violated. It is listed separately:

samples/partial-capture.cfg
------------------------------------------------------------------------------
no findings

Not evaluated:
  vty-no-access-class  no section matching /^line\s+vty\b/
  vty-telnet-enabled   no section matching /^line\s+vty\b/

These entries are not findings. A device without remote management is not less secure for it, so calling that a violation would be wrong. They do not count towards any total, and they only affect the exit code if you ask for that with --fail-on-not-evaluated; they exist so that "no findings" cannot be mistaken for "everything was checked". JSON output carries them per file and as a not_evaluated count in the summary.

Truncated configurations

An IOS running-config ends with end. When a file's last non-empty line is something else, the report opens that file's section with a warning:

warning: this configuration does not end with 'end', so it appears to be
truncated. Anything below the cut was not seen, and these results may be
incomplete.

This is one check at the level of the file, not a question each rule answers for itself. The parser only records the fact — has_end_marker, also present per file in the JSON output as a boolean so a pipeline can branch on it — and the report is what interprets it.

It deliberately does not affect the exit code. Truncation can only remove content, never add it: a must_not_match rule cannot pass falsely because of a cut, and a must_match rule turns a cut into a loud finding rather than a quiet pass. The one way a truncated file can slip through silently is a scope disappearing entirely, and --fail-on-not-evaluated already covers that. A heuristic ("this file looks truncated") should not drive a gate that a fact ("this rule did not run") already drives.

Rules

rules.json holds a list of rule objects:

{
  "id": "vty-telnet-enabled",
  "title": "Telnet is permitted on VTY lines",
  "severity": "high",
  "scope": { "type": "block", "start": "^line\\s+vty\\b" },
  "check": "must_not_match",
  "pattern": "^\\s*transport\\s+input\\b.*\\b(?:telnet|all)\\b",
  "why": "Telnet carries credentials and session data in clear text...",
  "fix": "transport input ssh"
}

Field	Meaning
id	unique machine-readable identifier
title	one-line description shown in the report
severity	high, medium or low
scope	{"type": "global"} for the whole file, or {"type": "block", "start": "<regex>"} to check inside each matching section
check	must_not_match — a match is a finding. must_match — the absence of a match is a finding
pattern	the regular expression to apply
ignore_case	optional, defaults to false
why	why this is a problem
fix	the command that resolves it
reference	optional pointer to the corresponding item in a recognised standard

Two axes — where to look and what counts as a violation — cover both "this must not be here" and "this must be here" checks. Block scope matters because a missing command has to be judged per section: a device where line vty 0 4 has an access-class and line vty 5 15 does not is misconfigured, and a whole-file search would not notice.

Rules are validated on load. A missing field, an unknown severity or a broken regex is reported with the offending rule id before any configuration is read.

References

reference is free text, printed after the fix as Ref: and carried into the JSON output. It is meant for the matching item in a benchmark such as the CIS Cisco IOS Benchmark.

Include the benchmark and its version, because CIS numbering shifts between releases and between platforms — a bare section number is ambiguous:

"reference": "CIS Cisco IOS 15 Benchmark v4.1.1 — 1.2.3"

None of the shipped rules currently carry a reference. The numbers were left out rather than guessed: an inaccurate pointer to a standard is worse than no pointer, because it looks authoritative and gets quoted onward.

How it works

audit.py            entry point
auditor/cli.py      argument parsing, orchestration, exit codes
auditor/rules.py    rules.json -> validated Rule objects
auditor/parser.py   config text -> numbered lines and indented blocks
auditor/engine.py   lines + rules -> findings, and rules that did not apply
auditor/report.py   results -> text or JSON

Data flows one way: the CLI loads rules, reads each file, hands the text to the parser, hands the parsed result to the engine, and hands the outcome to the renderer. The engine neither reads files nor prints anything, which is what makes it straightforward to test.

The parser treats indentation as the only structural marker, which is how IOS configurations are laid out: an unindented command opens a block and the indented lines beneath it are its body. A ! closes the current block.

run() returns an AuditResult with two lists: the findings, and the rules that could not be evaluated. Keeping them apart in the data model, rather than only in the report, is what stops a skipped check from being counted as a pass. FileResult wraps that result with the things that are true of the file rather than of the audit — its path, and whether it carries an end marker.

The split between stating a fact and interpreting it runs through the whole design: the parser records has_end_marker, the engine records that a rule had no section to look at, and the report is the only place that turns either into a sentence for a human.

Tests

python3 -m unittest discover -s tests

Every push and pull request runs the same command on GitHub Actions, against both Python 3.10 — the lowest version this README promises — and the current stable release, so the floor is tested and not just assumed. The badge at the top reflects those runs.

The suite covers the rule loader, the engine, the renderers and the CLI's exit codes, and includes regression tests for the formatting cases in samples/tricky-formatting.cfg: extra whitespace between keywords, the pre-12.x logging <ip> syntax, community strings that differ only in case, transport input all, and configuration-like text inside a banner.

Sample configurations

File	Purpose
clean-core-switch.cfg	passes every rule
legacy-access-switch.cfg	fails almost everything (11 findings)
branch-router.cfg	mostly hardened, 2 findings, one of them in only one of two VTY blocks
tricky-formatting.cfg	formatting edge cases that naive patterns get wrong

Known limitations

• IOS only. IOS-XE configurations mostly work; NX-OS and IOS-XR do not.
• logging host <hostname> is not recognised as a syslog destination; the pattern expects an IP address.
• The parser does not know about banner delimiters, so banner text is visible to global rules. Global patterns are anchored tightly enough that this does not currently cause false positives, but a rule that allows leading whitespace would be affected.
• Rules cannot express conditions across sections ("if A is set, B must also be set"). That would need a new check type rather than changes to the engine.
• A global rule is always considered evaluated, even against a partial configuration. Only block scopes can report that they had nothing to look at; the truncation warning covers the file as a whole instead.
• The end check is a heuristic. Output of show running-config | section ... is a legitimate fragment and will be flagged, and a file cut off exactly after its end line will not be.

Deliberately not done

These are known, weighed and left alone, not oversights.

A separate exit code for unevaluated rules. --fail-on-not-evaluated returns 1, the same code as a finding, so a pipeline cannot tell "problems were found" from "not everything ran" without reading the JSON. A third code would distinguish them, at the cost of one more thing every caller has to know. The JSON already carries the distinction for anyone who needs it.

Repeated paths are audited twice. audit.py configs/ configs/sw01.cfg reports that file twice and counts it twice in the totals, which a shell glob makes easy to trigger. Deduplicating means picking a canonical form for paths — resolving symlinks, or not — and that choice deserves more thought than the annoyance warrants.

The GitHub Actions are a major version behind. actions/checkout@v4 and actions/setup-python@v5 target Node 20, which GitHub has deprecated, so the runner forces them onto Node 24 and prints a warning on every run. Nothing is broken and the workflow is four lines; it will be bumped when something actually requires it.

Result containers are not hashable. Line and Block are values and can go in a set; AuditResult, FileResult and ParsedConfig hold lists and cannot. The consistency worth having is that value objects behave like values, not that every dataclass gets the same treatment.

Credits

Built with Claude Code as a project for learning Python. A walkthrough of the design and of every language feature used, written in Czech, is in VYSVETLENI.md.