Adversarial Robustness of ML-based IDS under a Realizability Constraint

Computer Networks case study — SRM 21CSC302J, Topic 10. Reproduces an IEEE paper (Gopa & Grace Shalini, ICADS 2026) on adversarial robustness + explainability for ML-based intrusion detection, then extends it.

The question

The paper attacks a Random Forest IDS with FGSM/PGD by perturbing all 41 NSL-KDD features freely. But a feature vector is not traffic. An adversarial example is only a real threat if some actual packet sequence, sent by a real attacker over TCP/IP, produces that vector.

By the mutability taxonomy in nsl_kdd_mutability_taxonomy.md:

Class	Count	Meaning
A — freely mutable	1	src_bytes
B — constrained	9	integer / raise-only / self-incriminating
C — non-mutable	31	victim-controlled, third-party window aggregates, or dead constants

So we re-run the attacks under a realizability mask and measure how much of the reported degradation survives. That is a networking critique of an ML result, not another ML tweak.

Status

• Repo scaffold, venv, pinned deps
• NSL-KDD obtained and verified (see data/README.md)
• Step 1 — RF baseline: 77.90% clean test accuracy
• Step 2 — FGSM/PGD under a transfer threat model
• Step 3 — adversarial training defence
• Step 4 — SHAP + stability (rho=0.9995, top-10 overlap 10/10)
• Step 5 — realizability mask (the contribution)
• Adaptive defense evaluation (attacker retrains against the defended model) — defense holds
• Improved IDS: 89.61% on KDDTest+ (src/ids_model.py)
• Evasion-rate evaluation: 31% → 5% (results/evasion.csv)
• Offline demo app: python demo/app.py (see demo/README.md)
• Live capture: lab coded + unit-tested (lab/); run it on a machine with working Docker

See STATUS_REPORT.md for the full, current results.

Headline results

Baseline RF, full KDDTest+ (n=22,544), clean accuracy 0.7790:

attack	unconstrained	constrained	degradation surviving
FGSM	−10.24 pts	−3.83 pts	37.4%
PGD	−24.13 pts	−4.24 pts	17.6%

The attacker moves 9 of 122 encoded dimensions (7.4%) once the mask is applied.

SHAP explains why the constrained attack is so much weaker: 80.7% of the model's importance mass sits in Class C features the attacker cannot control, and only 8.5% is attacker-controllable. The model is leaning on evidence that cannot be forged over the wire.

Two methodological findings worth reporting:

1. The published PGD setting (α=0.01, 5 iters) travels at most 0.05 in L-inf but claims ε=0.2 — it never reaches its own budget and is strictly weaker than its own FGSM. Saturation occurs at α·iters ≥ ε; we use α=0.02, 10 iters.
2. A linear substitute model makes FGSM leak the label: its input gradient is always ±w, so the perturbation direction encodes the class (measured: 93/93 shared dimensions push in opposite directions per class). A model adversarially trained on those examples scores a meaningless 100%. We use a nonlinear MLP substitute and the substitute's predicted labels instead.

Setup

Python 3.12 (not 3.13+ — ART/SHAP wheel availability).

uv venv .venv --python 3.12
uv pip install --python ./.venv/Scripts/python.exe -r requirements.txt

Get the data as described in data/README.md.

Run

./.venv/Scripts/python.exe src/model.py

Every number that goes in the report comes from results/metrics.csv, regenerated by src/experiment.py. No hand-typed numbers.

Layout

data/     NSL-KDD (git-ignored; see data/README.md)
src/
  preprocess.py   load, encode, scale, FIXED column order + raw->encoded index map
  model.py        RF train/eval/save
  attacks.py      FGSM/PGD via IBM ART
  mask.py         realizability mask  <- the novel bit
  shap_analysis.py
  experiment.py   constrained vs unconstrained -> results/metrics.csv
results/  metrics.csv, plots/, shap/
tests/    mask correctness (a frozen feature must not move the model input)