Detection Engineering Lab

A rule does not count as coverage until it fires on a real attack and stays quiet on normal activity.

Most detection projects stop at "I wrote a rule for technique X, so X is covered." That is a claim, not a fact. In real SOCs a large share of rules never fire the way people think, because a log field got renamed, the tool logs the command somewhere the rule is not looking, or the rule was copied from a blog and never tested against real data. Nobody notices until an incident walks straight past the rule.

This lab proves coverage instead of claiming it. It replays attacker techniques and normal activity, runs the detection rules over both, and only marks a technique as covered when its rule fires on the attack and stays silent on the benign traffic. Then it treats detections like code: a small CI gate fails the build if a rule that used to catch an attack quietly stops catching it.

The whole demo runs with the Python standard library. No install, no Docker, no API keys. It runs the same on Windows, macOS, and Linux, and covers Linux, Windows, cloud, and network detections.

---

Try it in 30 seconds

python3 demo/run_demo.py

You get this (real output, nothing faked). Coverage spans Linux, Windows, cloud, and network:

  DETECTION COVERAGE  (proven vs claimed)
  --------------------------------------------------------------------------
  STATUS   TECHNIQUE   SCENARIO                    ATTACK  BENIGN
  --------------------------------------------------------------------------
  BROKEN   T1003.001   LSASS credential dump            0       0
  BROKEN   T1105       Ingress tool transfer            0       0
  NOISY    T1552.001   Reading credential files         1       1
  PROVEN   T1053.003   Cron persistence with pay…       1       0
  PROVEN   T1059.001   PowerShell encoded command       1       0
  PROVEN   T1059.004   Download and run a script        1       0
  PROVEN   T1070.001   Clearing Windows event lo…       1       0
  PROVEN   T1070.002   Clearing system logs             1       0
  PROVEN   T1071.004   DNS tunneling                   25       3
  PROVEN   T1078.004   AWS root console login           1       0
  PROVEN   T1110       SSH password brute force         8       2
  PROVEN   T1547.001   Registry Run key persiste…       1       0
  PROVEN   T1548.003   Sudoers file tampering           1       0
  --------------------------------------------------------------------------
  Claimed: 13   Proven: 10   Noisy: 1   Broken: 2   =>  76.9% real coverage

A basic count says 13 out of 13 techniques are covered. The truth is 10. One rule is too noisy to trust and two are silently broken. The demo also writes:

• reports/coverage.html (open it in a browser)
• reports/coverage.md
• reports/navigator-layer.json (upload at mitre-attack.github.io/attack-navigator to see the matrix light up green, orange, red)

make demo     # same thing
make test     # run the tests (standard library only)

---

Why this is different from "I set up a SIEM"

Every SIEM ships with default rules. You can download Sigma rules by the hundred and a Navigator heat map will happily paint your coverage green. All of that shows claimed coverage. Almost nobody checks whether the rules actually fire.

This lab makes the claim earn its color. For every rule it runs two experiments:

Experiment	What it replays	The rule must
Fire test	the matching attack	alert
Quiet test	normal admin, CI, and cloud activity	stay silent

Only rules that pass both count. That gives three honest outcomes:

• PROVEN  fired on the attack, quiet on benign. Real coverage.
• NOISY   fired on the attack but also on normal activity. False positives, cannot trust it.
• BROKEN never fired on its own attack. Claimed coverage that is not real.

---

The rules that fail (on purpose)

The lab ships one noisy rule and two broken ones so you can see what the prover catches. These are the real bugs detection engineers hit every week, on both Linux and Windows.

NOISY, T1552.001 (reading credential files, Linux). The rule flags any read of a file under .ssh/. That also matches a normal admin running cat ~/.ssh/known_hosts, so it fires on benign activity. A rule that cries wolf gets muted, and then it is not really coverage. The fix is to narrow it to private keys and credential files, not the whole .ssh folder.

BROKEN, T1105 (tool download with wget, Linux). The rule keys on the process image ending in /wget. But when an attacker runs bash -c "wget http://.../tool", the logged process image is /bin/bash, and wget only shows up in the command line. The rule looks in the wrong field and never fires. You would swear you had coverage. The fix is to match the command line, not the image. See rules/linux_wget_ingress.yml.

BROKEN, T1003.001 (LSASS credential dump, Windows). The rule keys on the file name procdump.exe. An attacker who renames the tool to p64.exe walks straight past it. The fix is to match the original file name, which Sysmon still reports after a rename, or the LSASS access itself. See rules/win_lsass_dump.yml. This renamed-binary trick is one of the most common ways real detections get bypassed.

All three are the kind of thing a coverage heat map hides and this lab surfaces.

---

Detections as code: catch regressions in CI

The first run records a baseline.json of what is proven today. After that:

python3 demo/run_demo.py --gate

fails with a non-zero exit code if any technique that used to be proven stops being proven. Wire that into CI (see .github/workflows/detections-ci.yml) and a pull request that quietly breaks a detection fails the build, the same way a broken unit test would. Rules drift over time; this stops the drift from being silent.

$ python3 demo/run_demo.py --gate     # after someone breaks a rule
DETECTION REGRESSION: these techniques were proven and no longer are:
  - T1059.004: PROVEN -> BROKEN
A rule stopped catching an attack it used to catch. Failing the build.

---

How it works

  rules/*.yml              real Sigma detection rules
  emulation/attacks/*      attacker techniques as log events   (rule must fire)
  emulation/benign/*       normal activity as log events        (rule must stay quiet)
        │
        ▼
  engine/sigma.py          a small Sigma engine: log source match, field
                           modifiers (contains/startswith/endswith/re),
                           and/or/not conditions, count() aggregation
        │
        ▼
  engine/prover.py         fire test + quiet test  ->  PROVEN / NOISY / BROKEN
        │
        ▼
  engine/report.py         console + Markdown + HTML
  engine/navigator.py      MITRE ATT&CK Navigator layer (color coded)

Everything is plain files on disk. Open any rule to see the logic, open any .jsonl to see the exact events being tested. Nothing is hidden.

---

Project structure

detection-engineering-lab/
├── engine/            the Sigma engine, the prover, and the reporters (pure stdlib)
├── rules/             13 real Sigma rules across Linux, Windows, cloud, network
├── emulation/
│   ├── attacks/       one attacker scenario per technique
│   ├── benign/        normal Linux, Windows, cloud, and network activity
│   └── manifest.yml   maps scenario -> technique -> the rule that should catch it
├── demo/run_demo.py   the offline demo and the CI gate
├── tests/             tests that run with the standard library
├── infra/             the full lab: real Splunk + cross-platform ingest + a self-contained option
├── .github/workflows/ detections-as-code CI gate
└── baseline.json      what is proven today (used by the CI gate)

---

Want the real thing? (Splunk, on any OS)

The demo uses the lab's own tiny engine so it can run anywhere. If you want a real SIEM, infra/ brings up Splunk in Docker (Windows, macOS, or Linux) and is open by design: Splunk is the hub, and anything that speaks to its HTTP Event Collector or a forwarder can feed it. There are two ways to run it:

make infra-up            # Splunk only: point your own hosts at it (Windows Sysmon, Linux auditd, CloudTrail, Zeek)
make infra-selfcontained # Splunk plus a container that streams the lab's events in, no extra host needed
make infra-verify        # one command to confirm it is up and accepting events
make infra-down          # stop and wipe

From there you convert the Sigma rules into Splunk searches with sigma-cli, run real Atomic Red Team techniques on a Windows or Linux host, and import the ATT&CK Navigator layer. Full cross-platform steps are in infra/README.md.

Short version: the demo proves the idea in seconds with no install; the full lab is closer to a real SOC and runs on whatever machine you have.

---

Honest limitations

• The emulated events are hand-written to be realistic, not captured from a live host. The full lab in infra/ is where you flow in real telemetry.
• The bundled Sigma engine supports the common rule features, not every Sigma option. For production you convert the rules to your SIEM and let the SIEM run them; this engine exists so the demo can prove the idea with zero setup.
• The Windows detections are written and tested offline here. To feed them real Windows telemetry you need a Windows host running Sysmon. On Apple Silicon the clean way to add one is a small cloud Windows box, not a local VM.
• This proves a rule works against a known attack. It does not prove a rule will catch every variant of that attack. It raises the floor, it is not a ceiling.

---

Tech

Python 3 (standard library only for the demo and tests), Sigma rule format, MITRE ATT&CK and Navigator, Sysmon and Zeek log formats, Splunk and Atomic Red Team for the full lab, GitHub Actions for the CI gate. Runs on Windows, macOS, and Linux.