diff --git a/CLAUDE.md b/CLAUDE.md index f7029c8..b37954e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,22 +8,44 @@ This repo follows the 7-stage pipeline tracked in `docs/status.yaml`. Use `/proj The original design brief is `docs/prep-n-research.md`. It marks open questions as `{RESEARCH}` (must be answered with cited justification) and `{GRILL}` (must be nailed down in PRD / eng review). Do not silently resolve them. ## Commands -- Install: `uv sync` -- Test: `uv run pytest -q` +- Install: `uv sync` · also needs Zeek, Suricata, Wireshark (`brew install zeek suricata wireshark`) +- Test: `uv run pytest -q` (tests invoke Zeek/Suricata for real — see Conventions) - Lint: `uv run ruff check . && uv run ruff format .` -- Run: `uv run flabel ` +- Run: `uv run flabel --offline ` (bare `flabel ` is a Phase 2 stub) + +## Delivery phases +- **Phase 1 (current): Tier 2 only** — Suricata + Zeek reading the capture file. No lab. +- **Phase 2: Tier 1** — PANW NGFW via capture replay. Blocked on an unverified feasibility + question (can a cloud VM-Series see replayed traffic at all?) — PRD §13 Q16. Plan Phase 2 + only after that reachability spike. +- The CLI contract is already final: `--offline` is permanent, Phase 2 adds no flags. ## Architecture -- FILL_IN after /project:plan (entry point, module boundaries, where logic vs IO lives) +Entry point `src/flabel/cli.py` (argparse, zero runtime deps). One module per pipeline stage: +`ingest` → `zeek` → `suricata` → `correlate` → `labels`, with `rules/{fetch,admit,snapshot}` +managing content-addressed ruleset snapshots. Full contracts in `docs/spec.md` §3–§4. +- **Pure** (no `subprocess`/`urllib`/`socket`): `models`, `errors`, `config`, `rules/admit`, + `correlate`, `labels`, `provenance`, `notice`. A test enforces this. +- **I/O**: `ingest`, `zeek`, `suricata`, `rules/fetch`. `rules/fetch` is the *only* network I/O. +- `models.py` holds every dataclass and imports nothing from the package. ## Conventions - Write a test alongside every new function. Build test-first (`/tdd`). - Small functions; type hints on public interfaces. -- Config via environment, never hardcoded. Copy `.env.example` → `.env` (gitignored) for the GCP project ID and device endpoints; refer to them as `${GCP_PROJECT}` etc. in committed files. +- **Tools real, network stubbed.** Zeek/Suricata/`editcap` are invoked for real in tests — a mock + would encode our assumptions about tool behaviour, which is what needs verifying. Rule-feed + endpoints and the PANW device are never contacted. +- **Zeek is always invoked with `-D`.** Verified: without it `uid` differs every run and + reproducibility is impossible. Step 5 has a regression test that fails if it's dropped. +- Config via environment, never hardcoded. Copy `.env.example` → `.env` (gitignored) for the GCP + project ID and device endpoints; refer to them as `${GCP_PROJECT}` etc. in committed files. ## Guardrails -- **Label trustworthiness is the top quality bar.** Every verdict carries its source and provenance; never emit a label whose origin can't be traced. -- Public repo. Never commit secrets, capture data (`*.pcap`), device credentials, or internal host/project identifiers. +- **Label trustworthiness is the top quality bar.** Every verdict carries its source and provenance; + never emit a label whose origin can't be traced. `docs/spec.md` §13 lists the hard never-dos. +- **The benign canary is the standing FP review.** Wholesale-admitted sources have no per-rule gate, + so a benign fixture producing any label fails the build. Don't weaken it to make a test pass. +- Public repo. Never commit secrets, capture data, device credentials, or internal identifiers. + `.gitignore` has a deliberately narrow `tests/fixtures/**` exception — keep it narrow. - `archive/` is local-only (gitignored) — GCP teardown manifests with IAM and network detail. -- Never call external APIs or real devices in tests (mock them). - Don't edit files outside the current PLAN.md step without asking. diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..cd1d3d4 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,175 @@ +# PLAN — flabel Phase 1 + +**Stage 4, Part B.** Built from `docs/spec.md`. **Phase 1 only** — Phase 2 is planned after its reachability spike. + +Test-first throughout (`/tdd`). No code is written in this stage. + +## Shape of the work + +``` +Step 1 toolchain + CI ── blocks everything + │ +Step 2 models, errors, config ── blocks everything downstream + │ + ├─ Step 3 ingest ─┐ + ├─ Step 4 rules ─┤ all four touch disjoint files + ├─ Step 5 zeek ─┤ → safe to build in parallel + └─ Step 6 suricata ─┘ + │ + ├─ Step 7 correlate ─┐ pure; parallel with each other + └─ Step 8 labels ─┘ + │ +Step 9 cli (integration) + │ +Step 10 canaries + reproducibility gates +``` + +**Parallel groups:** {3, 4, 5, 6} and {7, 8}. Cap at 2–3 worktrees at a time. Every other step is sequential. + +| Step | Issue | Parallel-safe | +| :-: | :-: | :-: | +| 1 Toolchain and CI | [#15](https://github.com/DeepTempo/flabel/issues/15) | | +| 2 Foundations | [#16](https://github.com/DeepTempo/flabel/issues/16) | | +| 3 Ingest | [#17](https://github.com/DeepTempo/flabel/issues/17) | ⟂ | +| 4 Rules | [#18](https://github.com/DeepTempo/flabel/issues/18) | ⟂ | +| 5 Zeek | [#19](https://github.com/DeepTempo/flabel/issues/19) | ⟂ | +| 6 Suricata | [#20](https://github.com/DeepTempo/flabel/issues/20) | ⟂ | +| 7 Correlate | [#21](https://github.com/DeepTempo/flabel/issues/21) | ⟂ | +| 8 Labels | [#22](https://github.com/DeepTempo/flabel/issues/22) | ⟂ | +| 9 CLI | [#23](https://github.com/DeepTempo/flabel/issues/23) | | +| 10 Canaries | [#24](https://github.com/DeepTempo/flabel/issues/24) | | + +**Why step 1 is first:** the testing decision is *tools real, network stubbed* — Zeek, Suricata, and `editcap` are invoked for real in tests. Until CI can run them, **nothing else is testable**, so this is a hard prerequisite rather than scaffolding to do later. + +--- + +## Step 1 — Toolchain and CI + +**Files:** `.github/workflows/ci.yml`, `Dockerfile.toolchain` (or a CI install block), `docs/dev-setup.md`, `.gitignore` + +**Changes:** Provide pinned Zeek 6+/8.x, Suricata 8.x, and Wireshark (`editcap`, `capinfos`) to CI, plus the `zeek/foxio/ja4` package via `zkg`. Record exact versions — pinning is a precondition for Goal 2, since reproducibility across unpinned tool versions is meaningless. Document local setup (`brew install zeek suricata wireshark`). Add `.flabel/` to `.gitignore`. + +**Test that proves it:** CI runs a job asserting `zeek --version`, `suricata --version`, `editcap --version` all succeed and match the pinned versions, then runs `pytest -q`. **CI fails if zero `requires_tools` tests executed** — a skipped integration suite must never look like a passing one. + +**Depends on:** nothing. **Blocks:** everything. + +--- + +## Step 2 — Foundations: models, errors, config + +**Files:** `src/flabel/models.py`, `src/flabel/errors.py`, `src/flabel/config.py`, `data/sources.toml`, `tests/test_models.py`, `tests/test_config.py` + +**Changes:** All frozen dataclasses from spec §4, in one module that imports nothing from the package. Typed exceptions mapped to exit codes (spec §12). Source-registry loader with validation. `data/sources.toml` populated with the ten sources and their authoritative class/basis/licence assignments from spec §5. + +**Test that proves it:** `SourceSpec.may_label` is `False` exactly for `identify`; `label_basis` is `indicator-reference` exactly for `ioc-name`; unknown `source_class` or `admission_basis` raises; `metadata-filter` on a source without ET metadata raises; every exception type maps to exactly one exit code, and every exit code in spec §12 is reachable. **Plus the architectural guard:** a test asserting no pure module's source contains `subprocess`, `urllib`, or `socket`. + +**Depends on:** 1. **Blocks:** 3, 4, 5, 6, 7, 8. + +--- + +## Step 3 — Ingest and normalization ⟂ + +**Files:** `src/flabel/ingest.py`, `tests/test_ingest.py`, `tests/fixtures/make_awkward.py` + +**Changes:** Format sniffing by magic bytes (never extension), gzip decompression, flabel's own record-header walk for packet count and truncation offset, `editcap -F pcap` conversion, multi-datalink dominant-type selection. Emits `NormalizedCapture` with everything provenance needs. + +`make_awkward.py` extends the existing canary generator to emit the nasty inputs: truncated pcap, truncated pcapng, multi-datalink pcapng, bad header, gzipped variants. + +**Test that proves it:** each awkward fixture produces its specified outcome — truncated pcap gives `input_status: partial` with a correct offset; truncated pcapng and bad header raise `CaptureError` and create **no** output; multi-datalink keeps the dominant type and records the discards; gzip is transparent. Round-trip: a plain pcap normalizes to a byte-identical file. + +**Depends on:** 2. **Parallel with:** 4, 5, 6. + +--- + +## Step 4 — Ruleset fetch, admission, snapshots ⟂ + +**Files:** `src/flabel/rules/{__init__,fetch,admit,snapshot}.py`, `tests/test_admit.py`, `tests/test_snapshot.py`, `tests/fixtures/rules/*.rules` + +**Changes:** `fetch.py` is the only network I/O in the package, behind an interface a test can point at local files. `admit.py` implements the per-source policy: wholesale, or ET metadata filter on `confidence High` **and** `signature_severity Major|Critical`, counting each exclusion reason separately. `snapshot.py` writes content-addressed immutable snapshots with a manifest, and loads them. + +**Test that proves it:** on committed rule fixtures, `fetched == admitted + sum(excluded)` exactly; a `confidence Low` rule and a rule with no `confidence` key are excluded into *different* counters; `#alert` lines never admit; `ja3.hash` / `ja4.hash` rules are counted separately. Snapshot id is stable across writes of identical content and changes when content changes; `rules.rules` is sorted so fetch order cannot affect the id; `load_snapshot(None)` returns the newest; a missing snapshot raises. **Reports the real ET Open admitted counts, closing issue #11.** + +**Depends on:** 2. **Parallel with:** 3, 5, 6. + +--- + +## Step 5 — Zeek invocation and parsing ⟂ + +**Files:** `src/flabel/zeek.py`, `data/json-logs.zeek`, `tests/test_zeek.py` + +**Changes:** Single invocation `zeek -C -D -r json-logs.zeek`. The Zeek script adds JSON log filters for `conn` and `ssl` so one pass yields both TSV (retained) and JSON (parsed) and they cannot disagree. Parse `conn_json.log` into `Flow`, join `ssl_json.log` for `ja4`/`ja4s`/`server_name` on `uid`. Retain all TSV logs; strip the `_json` files from retained output. + +**Test that proves it:** on `benign.pcap`, exactly two flows with the expected tuples. **Determinism gate: two runs produce identical `uid`s** — this is the regression test for the verified spike-3 finding, and it fails if `-D` is ever dropped. `packet_filter.log` is confirmed non-reproducible and excluded. A TLS fixture yields a populated `ja4`. Non-zero exit produces a `tool_failures[]` entry rather than an exception escaping. + +**Depends on:** 2. **Parallel with:** 3, 4, 6. + +--- + +## Step 6 — Suricata invocation and parsing ⟂ + +**Files:** `src/flabel/suricata.py`, `tests/test_suricata.py`, `tests/fixtures/rules/synthetic.rules` + +**Changes:** Invoke with `-S /rules.rules` so only snapshot rules load and no ambient system ruleset leaks in; `--runmode single` for a deterministic alert set; JA3/JA4 fingerprinting enabled by `--set`. Parse `eve.json` alert records into `Detection`, resolving each SID's originating source from the snapshot manifest. **Drop detections from `identify`-class sources before they can become labels**, counting them. + +**Test that proves it:** a synthetic rule matching `benign.pcap` produces exactly one parsed `Detection` with correct sid/rev/classtype/tuple/timestamp. A synthetic **`ja4.hash`** rule matching a TLS fixture also produces a detection — proving the JA4 labelling *capability* independent of whether content exists (US-14). An `identify`-source rule that fires yields **zero** detections and increments `identify_alerts_suppressed` (US-16). Two runs produce the same alert set. + +**Depends on:** 2 (and step 4's snapshot writer for a real snapshot; a hand-built snapshot directory suffices to keep them parallel). **Parallel with:** 3, 4, 5. + +--- + +## Step 7 — Correlation ⟂ + +**Files:** `src/flabel/correlate.py`, `tests/test_correlate.py` + +**Changes:** Pure. Tuple match in either direction, time-window disambiguation on multiple candidates, `UnmatchedDetection` with a reason when zero or still-ambiguous. Consolidate to one `Label` per flow with sorted `sources` and `best_tier`. Implement the unmatched gate: silent at zero, warn above zero, fail above the threshold. + +**Test that proves it:** synthetic detections and flows only — no tools needed. One flow, one detection → one label. Two detections on one flow → one label with two sources. Port reuse with two candidate flows → resolved by time containment; detection outside both windows → `ambiguous_flow_match`. Tuple absent → `no_flow_match`. Threshold: 1 unmatched in 200 passes, 1 in 50 fails. `best_tier` is the minimum, not the maximum. + +**Depends on:** 2. **Parallel with:** 8. + +--- + +## Step 8 — Labels, provenance, NOTICE ⟂ + +**Files:** `src/flabel/labels.py`, `src/flabel/provenance.py`, `src/flabel/notice.py`, `tests/test_labels.py`, `tests/test_provenance.py` + +**Changes:** Build `SourceEntry` values, deriving `label_basis` from source class and carrying `admission_basis` and `licence`. Canonical serialisation per spec §10. Assemble the run block including every loss-condition field. Emit `NOTICE` for sources that actually asserted a label. + +**Test that proves it:** canonical output is byte-identical across two serialisations of the same data, and the `labels` array sorts by `(ts_first, uid)` regardless of input order. **Required-fields check: every `SourceEntry` carries every field the spec §4 table demands** — this is the automated form of Goal 1, with no "where applicable" escape. `ioc-name` sources yield `indicator-reference`; all other labelling classes yield `direct`. `NOTICE` lists a GPL/CC-BY source that asserted a label and omits a source that asserted none. Every loss-condition field exists in the run block. + +**Depends on:** 2. **Parallel with:** 7. + +--- + +## Step 9 — CLI and orchestration + +**Files:** `src/flabel/cli.py`, `tests/test_cli.py` + +**Changes:** argparse surface from spec §12 including the `rules` subcommands. Wire ingest → zeek → suricata → correlate → labels into the run directory (`{capture-name}_{datetime}/`). The default path is the `Coming Soon (TM)` stub. Map exceptions to exit codes. + +**Test that proves it:** end-to-end `--offline` on `benign.pcap` writes a run directory with `zeek/`, `labels.json`, and `NOTICE`, and exits 0. **The stub path prints `Coming Soon (TM)`, names `--offline`, creates no directory, and exits 3** (US-22). Re-running creates a sibling directory and leaves the first untouched; sorted names are chronological. A hard failure writes **no** `labels.json` — never a partial one. `--ruleset-snapshot nonexistent` exits 1. A labelling run makes no network call (asserted by a socket guard). + +**Depends on:** 3, 4, 5, 6, 7, 8. + +--- + +## Step 10 — Canaries and reproducibility gates + +**Files:** `tests/integration/test_canaries.py`, `tests/integration/test_reproducibility.py`, `tests/fixtures/README.md`, `.github/workflows/ci.yml` + +**Changes:** Wire Goal 5 and Goal 2 into CI as build-failing gates. Source the malicious canary and record its origin and licence. + +**Test that proves it:** **benign canary produces zero labels — any label fails the build** (Goal 5, and the standing FP review for every wholesale-admitted source including `pawpatrules`). Malicious canary produces at least one label. **Reproducibility: two full `--offline` runs against the same capture and pinned snapshot are identical after canonicalisation**, excluding only `started_at`/`finished_at`/`duration_seconds` and `packet_filter.log` (Goal 2). Fault-injection test for every Phase 1 loss condition in spec §11. + +**Depends on:** 9. **Note:** the malicious canary is the one unresolved input (spec §14); if it slips, the benign canary and reproducibility gates land without it and the sensitivity test follows. + +--- + +## Definition of done for Phase 1 + +- All ten steps merged, each behind a green `/project:verify`. +- CI green on `main` with the toolchain container, and **executing** the `requires_tools` tests rather than skipping them. +- Goals 1, 2, 3, 4, and 5 each have a passing automated test. +- `flabel --offline` labels a real public capture end to end. +- The default path prints `Coming Soon (TM)` and exits 3. +- Issue #11 closed with real admitted-rule counts. diff --git a/docs/spec.md b/docs/spec.md new file mode 100644 index 0000000..571be0e --- /dev/null +++ b/docs/spec.md @@ -0,0 +1,503 @@ +# Specification — flabel Phase 1 + +**Stage 4, Part A.** Derived from `docs/prd.md` v0.4 and `docs/eng-review.md`. Scope: **Phase 1 (Tier 2 / open-source screening) only.** Phase 2 is specified after its reachability spike. + +The bar for this document: hand it to a stranger and they build the same thing. + +--- + +## 1. Vocabulary + +Terms are used exactly as defined here, in code and in output. + +| Term | Meaning | +| :-- | :-- | +| **capture** | The input file as supplied by the operator: `pcap`, `pcapng`, optionally gzipped. | +| **normalized capture** | The single `pcap` file derived from the capture that every consumer reads. Never the original artifact. | +| **run** | One invocation of flabel against one capture. Produces exactly one run directory. | +| **run directory** | `{capture-name}_{datetime}/` — the complete, self-contained output of a run. | +| **flow** | A Zeek connection, identified by its `uid`. The authoritative unit of identity in the system. | +| **detection** | One alert from one source. Not yet tied to a flow. | +| **label** | A verdict about one flow, carrying every detection that asserted it. | +| **source** | A named ruleset or feed (e.g. `et/open`, `abuse.ch/urlhaus`). | +| **source class** | `signature`, `ioc-dest`, `ioc-name`, or `identify`. Determines `label_basis` and whether the source may label at all. | +| **admission basis** | `metadata-filter` or `wholesale`. How a source's rules were gated. Orthogonal to source class. | +| **label basis** | `direct` (this flow is the malicious activity) or `indicator-reference` (this flow referenced a malicious indicator). | +| **ruleset snapshot** | An immutable, content-addressed directory of filtered rules plus a manifest. The unit of reproducibility. | +| **canary** | A fixture capture with a known-correct expected outcome. Benign → zero labels; malicious → at least one. | +| **loss condition** | An enumerated way a run can under-report. Each has a named field in the run block and one fault-injection test. | + +--- + +## 2. Constraints and invariants + +These hold everywhere. A violation is a bug, not a trade-off. + +1. **Zero runtime dependencies.** Standard library only. `tomllib` (3.11+) for config; `argparse` for the CLI. Dev dependencies (pytest, ruff) are unrestricted. +2. **Only `flabel rules update` performs network I/O.** A labelling run that attempts a network connection is a defect. This is what makes Goal 2 achievable. +3. **Zeek is always invoked with `-D`.** Verified: without it, `uid` differs on every run and reproducibility is impossible. +4. **All consumers read the same normalized capture.** They cannot disagree about input. +5. **Absence is never a signal.** Every enumerated loss condition is reported in the run block. Silence means nothing happened, never "something happened and we didn't say." +6. **A computed fingerprint is never a verdict.** Labels come only from rule matches. +7. **Phase 2 must be additive.** No schema version change, no consumer change. Any Phase 1 design that would force one is wrong. +8. **`identify`-class sources can never produce a label.** Enforced in code and asserted in a test. + +### Testing line: tools real, network stubbed + +`CLAUDE.md` says never call external APIs in tests. That rule targets **external services and devices**, and it stands: the PANW device (Phase 2) and rule-feed endpoints are never contacted from a test. + +Local CLI tools are a different category — Zeek, Suricata, and `editcap` are hermetic, deterministic, and versioned. **They are hard test dependencies and are invoked for real.** There are no mocks and no golden-file substitutes for them, because a mock would encode our assumptions about tool behaviour, which is exactly what needs verifying. + +| Boundary | In tests | +| :-- | :-- | +| Zeek, Suricata, `editcap`, `capinfos` | **Invoked for real.** Suite cannot run without them. | +| Rule-feed HTTP endpoints | **Stubbed** — `fetch` reads local fixture files. | +| PANW device (Phase 2) | **Never contacted.** `[LAB]` criteria only. | + +Consequence: CI must provide the toolchain before any other step is testable. That is step 1 of the plan. + +--- + +## 3. Module layout and responsibilities + +``` +src/flabel/ + __init__.py __version__ + models.py all dataclasses; imported by everything, imports nothing + errors.py typed exceptions -> exit codes + config.py load + validate the source registry + ingest.py format sniff, decompress, convert, validate + zeek.py invoke Zeek; parse conn.log + ssl.log + suricata.py invoke Suricata; parse eve.json + rules/ + __init__.py + fetch.py the ONLY network I/O in the package + admit.py per-source admission policy (pure) + snapshot.py hash, write, load snapshots + correlate.py detections -> flows (pure) + labels.py build labels, canonical serialisation (pure) + provenance.py assemble the run block (pure) + notice.py emit NOTICE attribution (pure) + cli.py argument parsing, orchestration, exit codes +data/ + sources.toml the source registry (shipped with the package) + json-logs.zeek Zeek script adding JSON filters +``` + +**`models.py` is a refinement of the approved layout.** Every module codes against shared dataclasses rather than each owning its own, which is what allows steps 4–7 to be built in parallel without importing one another. + +**Pure modules** (`models`, `errors`, `config`, `admit`, `correlate`, `labels`, `provenance`, `notice`) must not import `subprocess`, `urllib`, or `socket`. Enforced by a test that greps the module sources — a cheap architectural guard that survives refactoring. + +--- + +## 4. Data models + +All are frozen dataclasses in `models.py`. + +```python +# --- configuration ------------------------------------------------------- +SourceClass = Literal["signature", "ioc-dest", "ioc-name", "identify"] +AdmissionBasis = Literal["metadata-filter", "wholesale"] +LabelBasis = Literal["direct", "indicator-reference"] + +@dataclass(frozen=True) +class SourceSpec: + name: str # "et/open" + url: str + licence: str # SPDX id, or "unstated" + source_class: SourceClass + admission_basis: AdmissionBasis + enabled: bool = True + + @property + def may_label(self) -> bool: # False iff source_class == "identify" + @property + def label_basis(self) -> LabelBasis | None: # None iff not may_label + +# --- ruleset snapshot ---------------------------------------------------- +@dataclass(frozen=True) +class SourceAdmission: + name: str + licence: str + source_class: SourceClass + admission_basis: AdmissionBasis + rules_fetched: int + rules_admitted: int + rules_excluded_no_confidence: int + rules_excluded_low_confidence: int + rules_excluded_low_severity: int + ja4_rules_admitted: int + ja3_rules_admitted: int + fetched_at: str # ISO-8601 UTC + +@dataclass(frozen=True) +class SnapshotManifest: + snapshot_id: str # sha256(rules.rules)[:16] + created_at: str + flabel_version: str + sources: tuple[SourceAdmission, ...] + total_admitted: int + total_ja4_admitted: int + +# --- pipeline ------------------------------------------------------------ +@dataclass(frozen=True) +class Flow: + uid: str + src_ip: str; src_port: int + dst_ip: str; dst_port: int + proto: str + ts_first: float; ts_last: float + ja4: str | None = None + ja4s: str | None = None + server_name: str | None = None + +@dataclass(frozen=True) +class Detection: + source: str # SourceSpec.name + tier: int # always 2 in Phase 1 + sid: int + rev: int + classtype: str | None + app_proto: str | None + threat: str # the rule's msg + ts: float # alert timestamp, capture timeline + src_ip: str; src_port: int + dst_ip: str; dst_port: int + proto: str + +@dataclass(frozen=True) +class SourceEntry: # one asserting detection on a label + tier: int + source: str + sid: int + rev: int + ruleset: str # snapshot_id + admission_basis: AdmissionBasis + licence: str + classtype: str | None + label_basis: LabelBasis + threat: str + +@dataclass(frozen=True) +class Label: + flow: Flow + verdict: Literal["malicious"] + best_tier: int # min(tier); lower is higher trust + sources: tuple[SourceEntry, ...] + +@dataclass(frozen=True) +class UnmatchedDetection: + detection: Detection + reason: Literal["no_flow_match", "ambiguous_flow_match"] +``` + +### `labels.json` document + +```json +{ + "schema_version": "1.0", + "run": { "...": "see §9" }, + "labels": [ "...Label..." ], + "unmatched_detections": [ "...UnmatchedDetection..." ] +} +``` + +`schema_version` is `"1.0"` and **does not change when Phase 2 adds tier-1 entries to `sources[]`** (Goal 6). + +--- + +## 5. Source registry + +`data/sources.toml`, shipped with the package, overridable with `--sources`. + +```toml +[[source]] +name = "et/open" +url = "https://rules.emergingthreats.net/open/suricata-7.0/emerging.rules.tar.gz" +licence = "MIT" +source_class = "signature" +admission_basis = "metadata-filter" + +[[source]] +name = "abuse.ch/feodotracker" +url = "https://sslbl.abuse.ch/blacklist/..." +licence = "CC0-1.0" +source_class = "ioc-dest" # matches a C2 destination -> the flow IS malicious +admission_basis = "wholesale" + +[[source]] +name = "abuse.ch/urlhaus" +licence = "CC0-1.0" +source_class = "ioc-name" # matches a looked-up name -> reference +admission_basis = "wholesale" + +[[source]] +name = "oisf/trafficid" +licence = "MIT" +source_class = "identify" # may_label == False +admission_basis = "wholesale" +``` + +**Authoritative class assignments:** + +| Source | Licence | Class | Basis | Labels? | +| :-- | :-- | :-- | :-- | :-: | +| `et/open` | MIT | `signature` | metadata-filter | direct | +| `stamus/lateral` | GPL-3.0-only | `signature` | wholesale | direct | +| `malsilo/win-malware` | MIT | `signature` | wholesale | direct | +| `the-hunters-ledger/open` | CC-BY-4.0 | `signature` | wholesale | direct | +| `pawpatrules` | CC-BY-SA-4.0 | `signature` | wholesale | direct | +| `abuse.ch/feodotracker` | CC0-1.0 | `ioc-dest` | wholesale | direct | +| `abuse.ch/sslbl-c2` | CC0-1.0 | `ioc-dest` | wholesale | direct | +| `sslbl/ssl-fp-blacklist` | CC0-1.0 | `ioc-dest` | wholesale | direct | +| `abuse.ch/urlhaus` | CC0-1.0 | `ioc-name` | wholesale | **indicator-reference** | +| `oisf/trafficid` | MIT | `identify` | wholesale | **never** | + +Excluded entirely and absent from the registry: `tgreen/hunting`, `etnetera/aggressive`, `ptresearch/attackdetection`, `ptrules/open`, `sslbl/ja3-fingerprints`, and all commercial sources. + +Validation on load: unknown `source_class` or `admission_basis` is a hard failure; `metadata-filter` is permitted only where ET-style metadata exists. + +--- + +## 6. Admission policy — `rules/admit.py` + +```python +def admit(spec: SourceSpec, rule_lines: Iterable[str]) -> tuple[list[str], SourceAdmission] +``` + +Pure. Given a source and its fetched rule text, return the admitted rules plus counts. + +- `admission_basis == "wholesale"` → admit every `alert` line. Count JA3/JA4 rules by presence of `ja3.hash` / `ja4.hash`. +- `admission_basis == "metadata-filter"` → admit only where **both** hold: + - `metadata: ... confidence High ...` + - `metadata: ... signature_severity Major|Critical ...` + + Rules with no `confidence` key are excluded and counted separately from rules with `confidence Low`/`Medium` — the distinction feeds issue #10. +- Commented-out rules (`#alert`) are never admitted. +- Every exclusion increments exactly one counter; `fetched == admitted + sum(excluded)` is asserted. + +--- + +## 7. Ruleset snapshots — `rules/snapshot.py` + +``` +.flabel/rules// + manifest.json SnapshotManifest + rules.rules concatenated admitted rules, sorted by (source, sid) + raw/.rules as fetched, for audit +``` + +```python +def write_snapshot(root: Path, admitted: Mapping[str, list[str]], + admissions: Sequence[SourceAdmission]) -> SnapshotManifest +def load_snapshot(root: Path, snapshot_id: str | None) -> tuple[Path, SnapshotManifest] +def list_snapshots(root: Path) -> list[SnapshotManifest] +``` + +- `snapshot_id = sha256(rules.rules bytes)[:16]`. Self-verifying: rewriting the file changes the id. +- `rules.rules` is written **sorted by (source, sid)** so the id depends on content, not fetch order. +- `load_snapshot(root, None)` returns the most recently created snapshot. +- A missing or unreadable snapshot is a hard failure (`SnapshotError` → exit 1). +- `.flabel/` is gitignored. + +--- + +## 8. Tool invocation + +### Zeek — `zeek.py` + +One invocation per run: + +``` +zeek -C -D -r /json-logs.zeek +``` + +- `-C` ignore checksum errors; `-D` deterministic seeds (**mandatory**). +- `json-logs.zeek` adds a JSON `Log::add_filter` for `conn` and `ssl` writing `conn_json.log` / `ssl_json.log`. **One pass produces both formats, so TSV and JSON cannot disagree.** +- TSV logs are the retained artifact in `zeek/`. The `_json` files are parse input and are removed from the retained output. +- Parsed: `conn_json.log` → `Flow`; `ssl_json.log` → `ja4`, `ja4s`, `server_name` joined on `uid`. All other logs retained unparsed. +- `packet_filter.log` carries a wall-clock stamp, is never reproducible, and is excluded from any reproducibility comparison. +- Non-zero exit or an OOM kill → `tool_failures[]` entry; the run fails. + +```python +def run_zeek(capture: Path, outdir: Path) -> tuple[dict[str, Flow], ZeekRunInfo] +``` + +### Suricata — `suricata.py` + +``` +suricata -r -S /rules.rules -l \ + --set app-layer.protocols.tls.ja3-fingerprints=yes \ + --set app-layer.protocols.tls.ja4-fingerprints=yes \ + --runmode single +``` + +- `-S` loads **only** the snapshot rules, replacing any system ruleset — no ambient state. +- `--runmode single` for determinism of the alert set. +- Parsed from `eve.json`: records with `event_type == "alert"` → `Detection`, taking `alert.signature_id`, `alert.rev`, `alert.category`, `alert.signature`, `alert.metadata`, `app_proto`, `timestamp`, and the 5-tuple. +- The originating source for each SID is resolved from the snapshot manifest, since `eve.json` does not carry it. +- Detections whose source has `may_label == False` are **dropped before correlation** and counted in `identify_alerts_suppressed`. + +```python +def run_suricata(capture: Path, snapshot: Path, outdir: Path) -> tuple[list[Detection], SuricataRunInfo] +``` + +### Ingest — `ingest.py` + +```python +def normalize(capture: Path, workdir: Path) -> NormalizedCapture +``` + +Order of operations: +1. **Sniff by magic bytes**, never by extension: gzip `1f 8b`; pcap `a1b2c3d4`/`d4c3b2a1` (and nanosecond variants); pcapng `0a0d0d0a`. +2. Decompress gzip to a temporary file. +3. **Validate by walking record headers** — flabel's own walk, because no tool in the dependency set reports a truncation offset. Yields `packets_read` and, if the final record is short, `truncated_at_offset`. +4. **Unreadable header** → `CaptureError`, hard failure, no output directory. +5. **Truncated pcap** → proceed; `input_status = "partial"`. +6. **Truncated pcapng** → hard failure telling the operator to repair with `editcap`; a partial pcapng block cannot be converted safely. +7. **pcapng** → `editcap -F pcap`. If it reports multiple link types, determine the dominant type by packet count, split with `editcap`, keep only the dominant, and record `discarded_link_types` and `discarded_packets` with `input_status = "partial"`. +8. Record every transformation in provenance. + +--- + +## 9. Correlation — `correlate.py` + +```python +def correlate(detections: Sequence[Detection], flows: Mapping[str, Flow], + threshold: float = 0.01) -> CorrelationResult +``` + +Pure. For each detection: + +1. Candidate flows are those matching the 5-tuple in either direction. +2. **Zero candidates** → `UnmatchedDetection(reason="no_flow_match")`. +3. **One candidate** → matched. +4. **Multiple candidates** (port reuse within one capture) → select the flow whose `[ts_first, ts_last]` window contains the detection `ts`. If exactly one qualifies, matched; otherwise `UnmatchedDetection(reason="ambiguous_flow_match")`. **A detection is never assigned to a flow by guess.** + +Then consolidate: one `Label` per flow, `sources` sorted, `best_tier = min(tier)`. + +**Gate:** zero unmatched is silent; any unmatched warns; unmatched / total detections above `threshold` (default `0.01`) fails the run. Phase 2 configures its own, looser threshold rather than relaxing this default. + +--- + +## 10. Canonical output — `labels.py` + +Reproducibility depends entirely on this being exact. + +- `labels` sorted by `(flow.ts_first, flow.uid)`. +- `sources` within a label sorted by `(tier, source, sid, rev)`. +- `unmatched_detections` sorted by `(ts, source, sid)`. +- `json.dump(..., sort_keys=True, indent=2, ensure_ascii=False)`, trailing newline. +- Timestamps: ISO-8601 UTC with microsecond precision and a `Z` suffix. One format everywhere. +- Floats never emitted where a string is expected; no locale-dependent formatting. + +**Excluded from a reproducibility comparison** — and nothing else: `run.started_at`, `run.finished_at`, `run.duration_seconds`, and `zeek/packet_filter.log`. + +### Run block + +```python +{ + "flabel_version": str, "schema_version": "1.0", + "started_at": str, "finished_at": str, "duration_seconds": float, + "mode": "offline", # Phase 1 is always this + "tiers_attempted": [2], "tiers_unavailable": [1], + "input": {"path": str, "sha256": str, "format": "pcap|pcapng|pcap.gz|pcapng.gz", + "bytes": int, "input_status": "complete|partial", + "packets_read": int, + "truncated_at_offset": int | None, + "discarded_link_types": [str], "discarded_packets": int, + "normalization": [str]}, + "ruleset": {"snapshot_id": str, "sources": [...SourceAdmission...], + "total_admitted": int, "total_ja4_admitted": int}, + "tools": {"zeek": str, "zeek_flags": ["-C", "-D"], "suricata": str, + "editcap": str, "ja4_zeek_package": str}, + "counts": {"flows": int, "detections": int, "labels": int, + "unmatched": int, "unmatched_ratio": float, + "identify_alerts_suppressed": int}, + "loss_conditions": {...}, # §11 + "tool_failures": [ ... ], + "warnings": [str] +} +``` + +### `NOTICE` — `notice.py` + +Lists every source that asserted at least one label in this run, with its licence and required attribution. Sources present in the snapshot but which asserted nothing are not listed. + +--- + +## 11. Loss conditions + +Each has a field and exactly one fault-injection test. This closed list is what Goal 3 is checked against. + +| Condition | Field | Fault injection | +| :-- | :-- | :-- | +| Input truncated | `input.input_status`, `packets_read`, `truncated_at_offset` | truncate a fixture mid-record | +| Multi-datalink discard | `input.discarded_link_types`, `discarded_packets` | fixture with two link types | +| Detection uncorrelatable | `counts.unmatched`, `unmatched_detections[]` | detection with a tuple absent from `conn.log` | +| Ambiguous flow match | `unmatched_detections[].reason` | two flows, same tuple, detection outside both windows | +| Tool non-zero exit / OOM | `tool_failures[]` | point at a non-existent binary | +| Snapshot missing | hard failure, exit 1 | `--ruleset-snapshot nonexistent` | +| `identify` alert suppressed | `counts.identify_alerts_suppressed` | rule from an `identify` source that fires | + +--- + +## 12. CLI contract — `cli.py` + +``` +flabel Phase 1: stub. Prints "Coming Soon (TM)", + names --offline, writes nothing, exit 3. +flabel --offline Runs the Tier 2 pipeline. + --ruleset-snapshot default: newest available + --output-dir default: cwd + --rules-dir default: ./.flabel/rules + --sources default: packaged data/sources.toml + --unmatched-threshold default: 0.01 +flabel rules update [--sources ] [--rules-dir ] +flabel rules list [--rules-dir ] +``` + +**Exit codes** + +| Code | Meaning | +| :-: | :-- | +| 0 | Success. Labels written. Covers both complete and partial input — `run.input.input_status` distinguishes them. | +| 1 | Failure. No labels written. | +| 2 | Usage error (argparse). | +| 3 | Not implemented — the Phase 1 default path only. | + +Partial input is deliberately **not** a distinct code: truncated captures are common, and a non-zero exit would make every ordinary `set -e` script treat a successful run as a failure. + +stderr carries progress and warnings; stdout is reserved and currently unused by the pipeline. `errors.py` maps each exception type to exactly one exit code. + +--- + +## 13. Explicit non-behaviours + +flabel **must never**: + +- assert that a flow is benign, or emit any verdict other than `malicious`; +- emit a label from a fingerprint value alone, without a rule match; +- emit a label attributable to an `identify`-class source; +- assign a detection to a flow by guess when the match is ambiguous; +- perform network I/O outside `flabel rules update`; +- invoke Zeek without `-D`; +- overwrite or modify a previous run directory; +- write a partial `labels.json` on a hard failure — either a complete run directory exists or none does; +- report full coverage when any loss condition fired; +- contact the PANW device (Phase 1 has no Tier 1 code path beyond the stub); +- commit, transmit, or copy capture data anywhere outside the run directory. + +--- + +## 14. Open items carried into build + +Not blocking the plan; each has an owner. + +| Item | Where | +| :-- | :-- | +| Malicious canary capture must be sourced (origin + licence recorded) | `tests/fixtures/README.md`, PRD Q8 | +| Exact ET Open admitted-rule counts | issue #11, measured by step 4 | +| Untagged-ET-rule policy | issue #10 | +| JA4 rule content | issue #13 | +| Stakeholders, target release, metric review dates | PRD Q1, Q2, Q10 | diff --git a/docs/status.yaml b/docs/status.yaml index 7ba22de..3b9b97b 100644 --- a/docs/status.yaml +++ b/docs/status.yaml @@ -4,8 +4,15 @@ project: "flabel" created: "2026-08-11" github_repo: "DeepTempo/flabel" notion_url: "https://app.notion.com/p/3b92a84a5230812aa64aef638e389725" -prd_gdoc_url: "https://docs.google.com/document/d/1WwKukQ71L6JQWkSpKvoR4lgwB96w3x1WHAHzZ30ul2E/edit" # v0.2; supersedes v0.1 doc 1q_muS6Am... (delete manually — Drive MCP has no delete) -current_stage: plan +prd_gdoc_url: "https://docs.google.com/document/d/1WwKukQ71L6JQWkSpKvoR4lgwB96w3x1WHAHzZ30ul2E/edit" +# STALE: the Doc above is PRD v0.2; docs/prd.md is v0.4. Regenerate it (Drive MCP cannot edit in +# place, so it means creating a new Doc and updating this URL). Also: the superseded v0.1 Doc +# 1q_muS6AmCuvcNva4C1WtRE3_KXArJPxnapttsU6UQIE needs deleting by hand — no delete tool exists. +current_stage: scaffold +next_action: "Merge PR #25, then run /project:build — it does the scaffold check first, then Step 1 (#15, toolchain+CI), which blocks every other step." +blocked_on: + - "Malicious canary capture must be sourced (origin + licence recorded) — needed by Step 10 (#24) only; steps 1-9 are unblocked." + - "PRD Q1/Q2/Q10: stakeholders, target release, success-metric review dates. Left TBD deliberately, not invented." # NOTE: issue numbers are not in stage order. Two creates hit a transient TLS # error and were retried, and GitHub never reuses issue numbers. This mapping is # authoritative — trust it over the numeric order in the GitHub issue list. @@ -13,9 +20,9 @@ stages: research: { status: completed, issue: 1, artifact: docs/research.md, completed: "2026-08-11" } prd: { status: completed, issue: 2, artifact: docs/prd.md, completed: "2026-08-11" } eng_review: { status: completed, issue: 6, artifact: docs/eng-review.md, completed: "2026-08-11" } - plan: { status: pending, issue: 3, artifact: PLAN.md, completed: null } + plan: { status: completed, issue: 3, artifact: PLAN.md, completed: "2026-08-11" } scaffold: { status: pending, issue: 4, artifact: .github/workflows/ci.yml, completed: null } - build: { status: pending, issue: 7, artifact: null, completed: null, steps_done: 0, steps_total: 0 } + build: { status: pending, issue: 7, artifact: null, completed: null, steps_done: 0, steps_total: 10 } verify: { status: pending, issue: 5, artifact: null, completed: null } log: - "2026-08-11 init — repo scaffolded, 7 stage issues filed, Notion row created" @@ -23,3 +30,5 @@ log: - "2026-08-11 stage(research) completed — recommends Suricata over Snort, replay only for PANW, and JA4-as-enrichment instead of JA3/JA4 labelling" - "2026-08-11 stage(prd) completed — labels.json schema settled (one entry per flow, sources[], Zeek uid join, malicious-only); timestamped run dirs; 13 user stories" - "2026-08-11 stage(eng-review) completed — 3 Critical / 9 High findings; PRD v0.4 splits delivery: Phase 1 = Tier 2 only, Tier 1 to Phase 2; NGFW default stubbed, --offline retained" + - "2026-08-11 stage(plan) completed — docs/spec.md + PLAN.md; 10 Phase 1 steps (#15-#24), parallel groups {3,4,5,6} and {7,8}; toolchain CI is step 1 because tests invoke Zeek/Suricata for real" + - "2026-08-12 session handoff — CLAUDE.md architecture filled in from the plan; PR #25 open and unmerged; local branches stage/{research,prd,eng-review} are merged and prunable"