From 2bd1d409da31eb87bb3b963d306640c21eaec428 Mon Sep 17 00:00:00 2001 From: Craig yourname Date: Tue, 11 Aug 2026 13:59:16 -0700 Subject: [PATCH 1/6] stage(research): approved research brief Part A of Stage 1. Records the four decisions from the grill: - PANW VM-Series is a Tier 1 trust source; replay architecture accepted - trust modelled per-source, not per-rule (so ruleset curation is the entire false-positive defence) - free rulesets preferred, 30-day delay acceptable, search broadened to government and professionally-curated sources - licence status must be recorded, not satisfied; unlicensed is acceptable - validation is trust-by-construction (logged as an explicit assumption) Also notes that designing and building the lab environment is deferred to plan/build, not dropped. Refs #1 --- docs/research-brief.md | 83 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 docs/research-brief.md diff --git a/docs/research-brief.md b/docs/research-brief.md new file mode 100644 index 0000000..5800788 --- /dev/null +++ b/docs/research-brief.md @@ -0,0 +1,83 @@ +# Research Brief — flabel + +**Stage 1 gate.** Approved by Craig on: 2026-08-11 · Issue: #1 + +## Objective + +Determine which detection sources, rulesets, and fingerprint feeds `flabel` can trust enough to produce malicious-flow labels for detection-model training — and confirm that the replay-based architecture required by a Tier 1 NGFW is workable — so that a PRD can specify the `labels.json` schema and pipeline without guessing. + +## Decisions already made (not open for research) + +These came out of the grill and bound the research: + +| Decision | Detail | +| --- | --- | +| NGFW is in | Palo Alto Networks VM-Series is a **Tier 1 trust source** for v1. The replay architecture it requires is accepted, not re-litigated. | +| Trust is **per-source**, not per-rule | A label carries its source's tier. Per-rule confidence metadata is *not* modelled in the output. | +| Consequence | Because trust isn't modelled per rule, **ruleset curation is the entire false-positive defence.** Research must answer *which rules are admitted*, not *how confident each rule is*. | +| Ruleset budget | Free sources preferred. A **30-day delay is acceptable** for high-confidence free rulesets. | +| Source breadth | Look beyond the OSS projects themselves: government agencies, highly regarded third parties, and rulesets curated by long-time security professionals (including top public repos). | +| Licensing | **Licence status must be recorded, not satisfied.** A high-quality source with no explicit licence is acceptable — note it as unlicensed/unstated. Do not exclude a trustworthy source for lacking a licence, and do not silently assume one. | +| Validation | **Trust by construction.** No ground-truth corpus validation in scope — see Out of scope. | + +## Key questions to answer + +### A. Content-inspection sources and rulesets +1. **Snort or Suricata?** Judge on technical merit *and* — likely decisive — the volume of high-confidence rulesets actually available for each. +2. Which specific free rulesets clear the "high confidence" bar? Candidates include ET Open (metadata-filtered), Talos registered (30-day delay), government-published sets, reputable third-party and professionally-curated repos. Each needs **source, provenance, licence status, update cadence, and why it is trusted** — where "licence status" may legitimately be *unstated/unlicensed*, recorded as such. +3. Given per-source tiering, what **rule-admission criteria** yield a uniformly high-confidence Tier 2 set? (Rule metadata such as `signature_severity`, `confidence`, and `deployment` is machine-readable and may serve as the filter.) +4. What does PANW VM-Series contribute as Tier 1 — what threat metadata does it expose, and how are detections queried programmatically? +5. Stretch: is there a free/OSS L7 application-detection equivalent to PANW/FortiGate? + +### B. Encrypted traffic — JA3/JA4 +6. Which JA3/JA4 feeds come from **highly trusted sources** (multiple; industry-standard, government, research-grade)? Provenance and update cadence for each. +7. How are multiple feeds **collated and deconflicted** into one list — specifically, what happens when two feeds disagree about the same fingerprint? +8. **Can a threat *name* be derived from a JA3/JA4 match**, or is the verdict only "matches a known-bad fingerprint"? This determines how usable these labels are for training. +9. What is the known **collision / false-positive behaviour** of JA3/JA4 fingerprinting — and does that make it a labelling source in its own right, or enrichment only? +10. Can Zeek compute both JA3 and JA4 natively, and at what version / with which plugins? + +### C. Architecture and replay fidelity +11. **Replay fidelity:** does `tcpreplay --topspeed` preserve what the inspection engine needs? Rewriting inter-packet timing can affect reassembly, flow timeouts, and rate-based rules. Are packet drops a risk — a dropped packet is a *missing label*. +12. **Tap vs routed** deployment for PANW: which detections are possible in tap/sniffer mode, and does tap silently disable a class of rules? +13. **Correlation:** how is the PANW log query bounded by the replay window, what clock-sync accuracy does that demand, and how are device detections (stamped at replay time) mapped back to the original capture's flows? +14. Which **pcap formats** can each component ingest (pcap, pcapng, gzipped, size limits)? This defines "all commonly supported formats." +15. What are the 2–3 viable overall architectures, and which is recommended? + +### D. Trust tiers +16. What evidence justifies assigning a source to Tier 1/2/3, and which tier does each source land in? + +## Constraints + +- **Stack:** Python 3.12 + uv + pytest + ruff (already scaffolded). Test-first via `/tdd`. +- **Environment:** GCP project `${GCP_PROJECT}` — Ubuntu host (flabel), PANW VM-Series (inline inspection), Snort/Suricata host. NTP sync required across all hosts. +- **Public repo:** no secrets, credentials, capture data, or internal identifiers may be committed. +- **Output contract:** `{input-pcap-name}/zeek/` (all Zeek logs) + `{input-pcap-name}/labels.json`. +- **Per-label data:** threat name, full flow tuple (src/dst IP, src/dst port, protocol), detection timestamp, detection source. +- **Primary quality bar:** label trustworthiness over label volume. Every verdict must be traceable to its origin. +- **Deliverable:** an environment diagram compatible with both draw.io and mermaid. + +## Exit criteria + +Research is done when: + +1. Snort vs Suricata is recommended, with justification covering both technical merit and high-confidence ruleset volume. +2. Every proposed ruleset and feed is documented with source, provenance, **licence status** (an explicit licence, or an explicit "unstated" — both acceptable, neither may be left blank or assumed), update cadence, and a written justification for trusting it. +3. Rule-admission criteria for a uniformly high-confidence Tier 2 set are defined. +4. Every detection source has a tier assignment with justification. +5. The JA3/JA4 questions are answered: threat-name derivability, collation/deconfliction approach, and a recommendation on labelling source vs enrichment. +6. Replay fidelity, tap-mode limits, and the clock/correlation approach are assessed, with risks ranked. +7. A pcap-format support matrix per component exists. +8. 2–3 viable architectures are compared, with a recommendation. +9. Top 5 risks are ranked, each with a concrete de-risking action. +10. Sources are cited; uncertainty is flagged rather than guessed. +11. 3–5 open questions are raised for Craig. + +## Out of scope + +- **Label validation against a ground-truth corpus.** Decided: trust by construction, resting on source tiers plus curated high-confidence rulesets. *Documented assumption:* the trustworthiness claim will be argued from provenance, not measured — so `/project:verify` can confirm the pipeline ran correctly but not that its verdicts are right. Revisit at eng-review if the claim needs to be defensible to an external consumer of the labels. +- Exact `labels.json` schema, confidence representation, and versioning — PRD. +- Whether benign flows are labelled at all (negative class) — PRD. +- Actual GCP provisioning and device configuration — **deferred, not dropped.** Designing *and building* the lab environment (Ubuntu host, PANW VM-Series, Snort/Suricata host, tap wiring, NTP) is a required project deliverable, along with the draw.io- and mermaid-compatible diagram. The environment gets designed at `/project:plan` (as spec + numbered steps) and built at `/project:build`. Research only needs to surface the constraints that shape that design — tap-vs-routed, clock accuracy, replay drops — not the build itself. +- FortiGate as the NGFW; PANW VM-Series is chosen. +- Paid rulesets, unless the research finds nothing free clears the trust bar — in which case flag it explicitly rather than lowering the bar. +- TLS decryption / MITM inspection — impossible on after-the-fact captures; fingerprinting is the substitute. From 7f8b831ac5c3f7f8264670513d962c2d7087b774 Mon Sep 17 00:00:00 2001 From: Craig yourname Date: Tue, 11 Aug 2026 14:09:54 -0700 Subject: [PATCH 2/6] =?UTF-8?q?stage(research):=20complete=20=E2=80=94=20d?= =?UTF-8?q?ocs/research-brief.md=20+=20docs/research.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings that change the design as briefed: 1. Suricata over Snort 3, decided on free high-confidence ruleset volume plus a machine-readable licence/provenance index (OISF/suricata-intel-index) and per-rule ET confidence metadata usable as a Tier 2 admission filter. 2. Replay only for PANW. Zeek and Suricata read pcaps natively; only PANW VM-Series has no offline ingest. Keeping Tier 2/3 offline makes them deterministic and confines replay-fidelity risk to the Tier 1 path. 3. JA4 as enrichment, not a label. The canonical free JA3 feed (abuse.ch SSLBL) is frozen at 2021-08-03 and self-declares untested FP posture; JA3 is collision-prone and broken by ClientHello shuffling; no free trustworthy JA4 verdict feed exists and no threat name is derivable. Also flags FoxIO License 1.1 (non-commercial) on the JA4+ suite — plain JA4 is BSD 3-Clause and safe. Also documents a required pcap ingest normalization stage: Zeek cannot read pcapng at all, Suricata only partially. Refs #1 --- docs/research.md | 261 +++++++++++++++++++++++++++++++++++++++++++++++ docs/status.yaml | 5 +- 2 files changed, 264 insertions(+), 2 deletions(-) create mode 100644 docs/research.md diff --git a/docs/research.md b/docs/research.md new file mode 100644 index 0000000..0224cee --- /dev/null +++ b/docs/research.md @@ -0,0 +1,261 @@ +# Research — flabel + +**Stage 1, Part B.** Driven by the approved `docs/research-brief.md` · Issue #1 · 2026-08-11 + +--- + +## Headline recommendation + +**Run Suricata and Zeek directly against the capture file. Replay only for PANW.** + +The brief's flow sends everything through the inline device via replay. That isn't necessary: Suricata and Zeek both read a pcap natively (`suricata -r`, `zeek -r`), and only PANW VM-Series has no offline pcap ingestion — Palo Alto's own guidance is to build a virtual-wire pair and replay at it from an external server ([LIVEcommunity](https://live.paloaltonetworks.com/t5/general-topics/replay-pcap/td-p/36261)). + +That split matters because **replay is the single largest threat to label fidelity.** Rewritten packet timing can disturb reassembly and rate-based rules, and a dropped packet is a *missing label*. Keeping the Tier 2 and Tier 3 paths offline makes them deterministic and byte-for-byte reproducible, and confines the nondeterminism to the one source that cannot avoid it. It also means a Tier 2/Tier 3-only run needs **no lab environment at all** — useful for testing and for captures processed before the lab exists. + +**Second recommendation: drop JA3 in favour of JA4, and treat fingerprint hits as enrichment rather than labels.** Evidence below — this is the most significant finding in this research and it contradicts the brief's assumption that JA3/JA4 feeds can supply trustworthy labels. + +--- + +## A. Content-inspection sources and rulesets + +### A1. Snort 3 or Suricata? → **Suricata** + +Your stated tiebreaker was volume of high-confidence rulesets, and Suricata wins it decisively — but on a subtler axis than raw rule count. + +| | Suricata | Snort 3 | +| --- | --- | --- | +| Free ruleset ecosystem | ~30 sources indexed centrally (below) | Talos registered + community | +| High-confidence *selection* mechanism | Per-rule `confidence` / `signature_severity` metadata | Policy tiers (connectivity / balanced / security / max-detect) | +| Free-tier delay | ET Open: none | Registered: 30 days behind subscriber, no zero-days | +| Rule compatibility | Reads Snort rules (imperfectly) | Cannot read Suricata-native keywords | +| Native pcap ingest | Yes (`-r`) | Yes (`-r`) | + +Two decisive points. First, Suricata has a **central, machine-readable index of free rule sources** — `OISF/suricata-intel-index` — with a declared licence per source, which is exactly the provenance record your brief demands and Snort has no equivalent of. Second, Suricata-native rules can express protocol fields Snort cannot, so equivalent detections are more precise with fewer false positives ([comparison](https://www.decryptiondigest.com/blog/snort-vs-suricata-ids-ips-comparison)). One study measured Suricata running ~4,600 more enabled rules than Snort on the same traffic. + +Snort's counter-argument is real and worth recording: Talos LightSPD is maintained by **dedicated paid researchers**, whereas ET Open is largely community-produced ([Snort blog](https://blog.snort.org/2020/12/soft-release-lightspd-new-rules-package.html)). If uniform curation mattered more than breadth, Snort would win. Under per-source tiering — where we curate the input set ourselves — breadth plus filterable metadata is more useful. + +**Nothing prevents running both.** They'd both be Tier 2, and deduplication would be needed. Not recommended for v1. + +### A2. Free rule sources, with licence status + +From `OISF/suricata-intel-index` (authoritative, machine-readable). **Licence status recorded as required — including the non-standard cases.** + +**Recommended for Tier 2 admission:** + +| Source | Vendor | Licence | Notes | +| --- | --- | --- | --- | +| `et/open` | Proofpoint | **MIT** | Primary set. Filter by metadata — see A3. | +| `oisf/trafficid` | OISF | **MIT** | Traffic identification, not threat detection. Enrichment. | +| `abuse.ch/feodotracker` | abuse.ch | **CC0-1.0** | Botnet C2 IPs, actively maintained. | +| `abuse.ch/urlhaus` | abuse.ch | **CC0-1.0** | Malware-distribution URLs. | +| `abuse.ch/sslbl-c2` | abuse.ch | **CC0-1.0** | C2 servers by blacklisted certificate. | +| `malsilo/win-malware` | malsilo | **MIT** | Windows malware artifacts. Small, focused. | +| `stamus/lateral` | Stamus Networks | **GPL-3.0-only** | Lateral movement. Copyleft — rules only, no linking concern. | +| `the-hunters-ledger/open` | The Hunters Ledger | **CC-BY-4.0** | Attribution required. Derived from malware investigations. | +| `pawpatrules` | pawpatrules | **CC-BY-SA-4.0** | Share-alike. Broad scope; needs FP review. | + +**Excluded, with reasons:** + +| Source | Licence | Why excluded | +| --- | --- | --- | +| `tgreen/hunting` | GPLv3 | Self-described **hunting / anomaly-detection** ruleset. Your brief explicitly excludes threat-hunting rules in favour of detection. Policy exclusion, not a quality judgement. | +| `etnetera/aggressive` | MIT | Self-described "**aggressive** IP blacklist". Incompatible with a low-FP bar. | +| `ptresearch/attackdetection`, `ptrules/open` | **Custom** (non-standard) | Two concerns. The licence is non-standard — acceptable per your guidance if noted, and it is noted. But the vendor, **Positive Technologies, has been under US sanctions since April 2021**. Against a requirement for "highly trusted sources," a sanctioned entity is a provenance problem independent of rule quality, and plausibly a compliance one for DeepTempo. **Recommend exclude; flagging as your call.** | +| `et/pro`, `scwx/*`, `stamus/nrd-*` | Commercial | Require paid subscription. `scwx/malware` is notable — self-described "**high-fidelity, high-priority**" — and is the closest thing to a purpose-built low-FP set. Worth pricing if free sources underdeliver. | + +**Government sources:** no government body publishes a maintained, general-purpose Suricata/Snort ruleset. CISA and allied agencies publish Snort signatures *inside individual advisories*, which are point-in-time IOCs rather than a feed. Harvesting them would mean scraping advisories — real work, low yield, no update cadence. **Recommend: not a v1 source.** This is a gap against your "government sources" ask, and I'd rather say so than pad the list. + +### A3. Tier 2 rule-admission criteria → filter on ET metadata + +This is where per-source tiering gets its false-positive defence. + +ET introduced a `confidence` metadata tag in 2022 specifically to express **false-positive likelihood** — "High" confidence means minimal FP likelihood. Coverage has grown from 30% to **over 70% of the ruleset, with newer rules at 100%** ([Proofpoint](https://www.proofpoint.com/us/blog/threat-insight/emerging-threats-updates-improve-metadata-including-mitre-attck-tags), [ET wiki](https://community.emergingthreats.net/t/signature-metadata/96)). `signature_severity` runs Informational → Critical. + +**Proposed admission rule:** + +``` +admit if metadata.confidence == High + and metadata.signature_severity in (Major, Critical) + and source in the recommended table above +``` + +Two consequences to decide at PRD time: + +- **The <30% of ET Open rules lacking a `confidence` tag are excluded** by this rule. Fail-closed is the right default when trustworthiness is paramount, but it discards untagged rules that may be good. +- The admitted rule set must be **snapshotted and recorded per run** (source + version + date), because the filter's output changes as ET revises metadata. Without that, two runs of "the same" flabel produce differently-grounded labels. + +### A4. PANW VM-Series as Tier 1 + +- **Tap mode works for our purpose.** In tap mode the firewall cannot block or reset, so every security profile action is set to `alert` — which is precisely what a labeller wants. It still performs App-ID and threat identification, and writes to the threat log ([PANW docs](https://docs.paloaltonetworks.com/pan-os/11-0/pan-os-networking-admin/configure-interfaces/tap-interfaces)). +- **But Palo Alto's own replay guidance is a virtual-wire pair**, not tap. Vwire is inline-but-transparent and is the configuration their community recommends for replaying a pcap at the device. Your brief prefers tap. **Unresolved:** whether tap-plus-mirror and vwire yield identical detection sets. This needs an empirical check on the actual device — see Open Questions. +- **Threat name is available.** PAN-OS threat logs carry the threat/content name, and the XML API retrieves Threat-type logs with filter expressions equivalent to the Monitor tab, including a `receive_time` field ([Retrieve Logs](https://docs.paloaltonetworks.com/ngfw/api/pan-os-xml-api-request-types-and-actions/retrieve-logs)). `kevinsteves/pan-python` is the mature Python client. **Not yet verified:** the exact filter syntax for a bounded `receive_time` range, and whether log-write latency requires a settling delay before querying. Verify on-device. + +### A5. Free L7 equivalent to PANW App-ID → partial, not equivalent + +Zeek plus Suricata's app-layer protocol detection covers protocol identification well. Neither reproduces App-ID's application-level catalogue. `oisf/trafficid` adds some identification. **Conclusion: no free equivalent; PANW's Tier 1 contribution is genuinely distinct.** That is an argument *for* keeping the NGFW, consistent with your decision. + +--- + +## B. Encrypted traffic — JA3/JA4 + +This section changed my view of the brief's design. Reporting it plainly. + +### B1. The canonical free JA3 feed is abandoned and self-declares an FP problem + +abuse.ch SSLBL's JA3 fingerprint blacklist is the source the brief implicitly assumes. Fetched directly: + +- **Most recent listed fingerprint: `2021-08-03`.** Five years stale. The *file* regenerates every 5 minutes, so automated freshness checks that watch file mtime would report it healthy — the *content* is frozen. +- The page carries this warning: **"These fingerprints have not been tested against known good traffic yet and may cause a significant amount of FPs!"** +- Licence is **CC0-1.0** — unrestricted, commercial use fine. Licensing is not the problem; provenance quality is. + +A feed that is both abandoned and self-declared as untested against benign traffic cannot be a labelling source for ML training data under a per-source trust model with no per-rule filter. + +### B2. JA3 is structurally degraded, independent of feed quality + +- Chrome and Firefox **shuffle ClientHello extension order**, which changes the JA3 hash for the same client. Stable JA3 values can no longer be assumed. +- JA3's limited attribute set produces **collisions** — unrelated clients sharing a fingerprint, so benign traffic can match a "malicious" JA3 ([Fingerprint.com](https://fingerprint.com/blog/limitations-ja3-fingerprinting-accurate-device-identification/)). + +For training data this failure mode is the expensive one: a collision mislabels *benign* traffic as malicious, teaching the model the wrong thing. + +### B3. JA4 is the successor, is maintained, and is partly licence-restricted + +- **Maintained:** `zkg install zeek/foxio/ja4`, v0.18.8. Zeek 5+ supported, Zeek 6+ for QUIC. Zeek published a how-to in January 2026 ([zeek.org](https://zeek.org/2026/01/how-to-use-ja4-network-fingerprints-in-zeek/)). +- JA4 sorts extensions, so it is **resistant to the shuffling that breaks JA3**. +- **Licence split matters:** + - **JA4 (TLS client) — BSD 3-Clause**, with FoxIO explicitly claiming no patent rights. + - **JA4+ (JA4S, JA4H, JA4X, JA4L, JA4SSH, JA4T, …) — FoxIO License 1.1: non-commercial only.** Internal use securing your own company is permitted; selling it in a product requires an OEM licence from FoxIO. + +**flabel produces training data for detection models.** If those models ship in a DeepTempo product, using JA4+ plausibly constitutes monetization. **Recommendation: use only plain JA4 (BSD), avoid the JA4+ extensions.** That keeps the project unambiguously clear of the restriction. This is a legal question, not an engineering one — flagged for your decision, and it is exactly the licence-status issue your correction to the brief anticipated. + +### B4. Can a threat *name* come from a fingerprint? → **No, not reliably** + +A fingerprint match yields "this client matches a known-bad fingerprint." Malware-family attribution only exists if the feed supplies it, and the one CC0 feed that did is frozen at 2021. `ja4db` (FoxIO) catalogues fingerprint→application mappings for identification, not malicious verdicts. **Answer: a Tier 3 label would carry no trustworthy threat name.** + +### B5. Recommendation: Tier 3 is **enrichment, not a label** + +Combining B1–B4: no maintained, trustworthy, free malicious-fingerprint feed exists; JA3 is collision-prone and unstable; JA4 is sound as a *fingerprint* but has no reputable free malicious-verdict feed behind it; and no threat name is derivable. + +**Recommend:** compute JA4 for every TLS connection and record it in the Zeek output and alongside labels as an *attribute*, but **do not emit a malicious label on fingerprint match alone.** This preserves all the analytic value — a model can learn from JA4 as a feature, and analysts can pivot on it — without asserting a verdict the evidence doesn't support. + +This contradicts the brief, which treats JA3/JA4 as a second labelling source. It's your call to overrule; I'd rather flag it now than produce labels I can't defend. + +### B6. Feed deconfliction + +Largely moot if B5 is accepted — with no feed supplying verdicts, there is nothing to deconflict. Should you keep fingerprint labelling, the required design is: per-feed provenance retained per fingerprint, never silently merged; disagreement resolved by recording *all* asserting feeds rather than voting; and a snapshot date pinned per run. + +--- + +## C. Architecture, replay fidelity, and formats + +### C1. Three viable architectures + +**Approach A — Offline only (no NGFW).** `zeek -r` + `suricata -r`, merge, emit. One host, no lab, no clocks, fully deterministic and reproducible. **Excluded by your Tier 1 decision**, but it is the correct v0/test configuration and the fallback if the lab is delayed. + +**Approach B — Hybrid: offline OSS + replay for PANW only. ⭐ Recommended.** Zeek and Suricata read the file directly; only PANW gets a replay. Confines all replay-fidelity and clock-correlation risk to the Tier 1 path, and Tier 2/3 stay reproducible. Costs one extra concept: two ingest paths for one capture. + +**Approach C — Full replay (as briefed).** Everything past the tap. Matches the original design, and is arguably more "realistic". But it makes the Suricata path nondeterministic and drop-prone **for no benefit**, since Suricata reads the file perfectly well. Not recommended. + +### C2. Replay fidelity — the top risk + +- `--topspeed` explicitly trades timing accuracy for speed; tcpreplay's own docs note that batching packets for throughput costs accuracy ([tcpreplay man](https://tcpreplay.appneta.com/wiki/tcpreplay-man.html)). +- Rewritten timing can affect stateful reassembly, flow timeouts, and any rate-based rule. **A dropped or reordered packet is a missing label, not a wrong one** — and a missing label is invisible in the output. +- **De-risking:** replay at a controlled rate rather than `--topspeed`; verify packet counts sent vs. seen; run the same capture twice and diff the detection sets — instability across identical runs quantifies the problem directly. I could not find published measurements of `--topspeed`-induced missed alerts; this needs empirical measurement in the lab, not a literature answer. + +### C3. Clock and correlation + +Bounding the PANW log query by the replay window requires that flabel's clock and the firewall's agree. NTP is necessary but I found **no published figure** for the required accuracy — it depends on log-write latency and how tightly the window is drawn. Practical approach: record replay start/end from flabel, pad the query window generously, then filter returned records by flow tuple rather than relying on time alone. Time bounds the query; the tuple does the matching. + +**Unsolved and worth PRD attention:** detections are stamped at *replay* time, but labels must reference the *capture's* original timeline. The mapping is not 1:1 under `--topspeed`, because the replay compresses the capture's duration. Correlation should therefore be tuple-driven, with replay-time used only to scope the query. + +### C4. pcap format support matrix + +| Component | pcap | pcapng | Notes | +| --- | --- | --- | --- | +| Zeek | Yes | **No** | `zeek -r` on pcapng produces parser errors; conversion via `editcap -F pcap` required ([Zeek community](https://community.zeek.org/t/analysing-pcapng-files-from-wireshark-traffic-captured-with-zeek-or-spicy/6959)) | +| Suricata | Yes | **Partial** | Reads pcapng 1.0; breaks on multi-interface files with differing datalinks ([Feature #432](https://redmine.openinfosecfoundation.org/issues/432)) | +| tcpreplay | Yes | Partial | Same multi-datalink caveat | +| PANW | n/a | n/a | No file ingest at all — replay only | + +**This kills "all commonly supported formats" as a naive requirement.** Since pcapng is what Wireshark produces by default, flabel needs an **ingest normalization stage**: detect format, convert pcapng → pcap with `editcap`, split multi-datalink captures first (`frame.interface_id` / `frame.dlt`), and record the conversion in provenance. Gzipped input needs decompression too. This is a real work item the brief didn't anticipate. + +### C5. Trust tier assignments + +| Tier | Source | Justification | +| --- | --- | --- | +| **1** | PANW VM-Series | Commercially curated signatures, named threats, App-ID coverage with no free equivalent. Your decision. | +| **2** | Suricata + metadata-filtered ET Open and the CC0/MIT sources in A2 | Per-rule `confidence: High` is a vendor-declared low-FP assertion; admission filter is machine-checkable and snapshottable. | +| **3** | JA4 fingerprints | **Recommend: enrichment only, not a labelling tier** (see B5). No maintained trustworthy verdict feed; no derivable threat name. | + +--- + +## Existing tools and maintenance status + +| Tool | Status | Role | +| --- | --- | --- | +| Zeek | Actively maintained; 8.x current | Logs, flow `uid`, JA4 host | +| `zeek/foxio/ja4` | Active, v0.18.8, Zeek 5+/6+ | JA4 computation | +| Suricata | Active, 8.x stable, 9.0 in dev | Tier 2 engine | +| `suricata-update` | Active, ships the source index | Ruleset fetch + filter | +| `OISF/suricata-intel-index` | Active | Licence/provenance record per source | +| tcpreplay | Maintained (AppNeta) | Replay to PANW | +| Wireshark `editcap` | Active | pcapng → pcap normalization | +| `kevinsteves/pan-python` | Mature; verify recent activity before adopting | PAN-OS XML API client | +| abuse.ch CC0 feeds (feodotracker, urlhaus, sslbl-c2) | Active, 5-min regeneration | Tier 2 IOC rules | +| abuse.ch SSLBL **JA3** list | **Effectively abandoned — newest entry 2021-08-03** | Do not use | + +--- + +## Top 5 risks, ranked + +1. **Replay infidelity silently drops labels (Tier 1).** A missing label is invisible — nothing in the output says "we lost a packet". *De-risk:* Approach B confines it to the PANW path; assert sent-vs-seen packet counts; diff repeat runs; avoid `--topspeed` in favour of a controlled rate. +2. **Trust-by-construction is unfalsifiable.** With per-source tiering and no validation corpus (your decision), the trustworthiness claim rests entirely on ruleset curation and cannot be measured. If a consumer of these labels ever asks "what's your false-positive rate?", there is no answer. *De-risk:* snapshot rulesets per run so labels are at least *reproducible* and auditable; revisit at eng-review. +3. **JA3/JA4 labelling would inject false positives into training data.** Abandoned feed plus collision-prone fingerprints. *De-risk:* adopt B5 — enrichment, not labels. +4. **JA4+ licensing exposure.** FoxIO License 1.1 forbids monetization; flabel feeds product models. *De-risk:* restrict to BSD-licensed plain JA4; get a legal read before touching JA4+. +5. **Correlating PANW detections back to capture flows.** Replay-time stamps, `--topspeed` time compression, and possible port reuse within a capture. *De-risk:* tuple-driven matching with time only scoping the query; define the unmatchable-detection behaviour explicitly (drop vs. emit unmatched) at PRD. + +--- + +## What I need from you to make a full recommendation + +1. A ruling on **JA4 as enrichment vs. label** (B5) — it changes the schema and the whole Tier 3 story. +2. A ruling on **Positive Technologies** sources given the sanctions status (A2). +3. Whether **JA4+ licensing** needs legal review, or whether restricting to plain JA4 settles it (B3). +4. Whether the **lab environment is a v1 prerequisite** or whether an Approach-A offline mode ships first while the lab is built. + +--- + +## Open questions + +1. **Does PANW tap mode produce the same detections as a virtual-wire pair?** Your brief prefers tap; Palo Alto's replay guidance says vwire. Needs an on-device A/B with an identical capture. If they differ, tap may silently forfeit a class of detections. +2. **Should flabel emit labels at all when the lab is unavailable** — i.e. is a Tier 2/3-only run a valid, clearly-marked output, or an error? Affects whether offline mode is a first-class feature. +3. **What is the required clock accuracy** between flabel and the firewall, and does the threat log need a settling delay before querying? Empirical, not documented. +4. **Are untagged ET Open rules (the <30% without `confidence` metadata) excluded or manually reviewed?** Fail-closed loses coverage; reviewing them is ongoing manual work. +5. **Is `scwx/malware` worth pricing?** It is the only ruleset explicitly marketed as high-fidelity/low-FP, and your bar is unusually demanding. You said free-with-30-day-delay is acceptable; this asks whether that's a preference or a hard constraint. + +--- + +## Sources + +- [Snort vs Suricata IDS/IPS 2026: Performance, Rule Sets](https://www.decryptiondigest.com/blog/snort-vs-suricata-ids-ips-comparison) +- [A Comparative Analysis of Snort 3 and Suricata (Univ. of Portsmouth)](https://pure.port.ac.uk/ws/portalfiles/portal/79753845/A_Comparative_Analysis_of_Snort_3_and_Suricata.pdf) +- [Emerging Threats Updates Improve Metadata, Including MITRE ATT&CK Tags — Proofpoint](https://www.proofpoint.com/us/blog/threat-insight/emerging-threats-updates-improve-metadata-including-mitre-attck-tags) +- [Signature Metadata — Emerging Threats wiki](https://community.emergingthreats.net/t/signature-metadata/96) +- [OISF suricata-intel-index (rule source licences)](https://github.com/OISF/suricata-intel-index/blob/master/index.yaml) +- [What are the differences in the rule sets? — Snort FAQ](https://www.snort.org/faq/what-are-the-differences-in-the-rule-sets) +- [Soft Release: lightSPD, the new rules package for Snort 3](https://blog.snort.org/2020/12/soft-release-lightspd-new-rules-package.html) +- [SSLBL Blacklist — abuse.ch](https://sslbl.abuse.ch/blacklist/) +- [SSLBL Malicious JA3 Fingerprints — abuse.ch](https://sslbl.abuse.ch/ja3-fingerprints/) +- [The Limits of JA3 Fingerprinting — Fingerprint.com](https://fingerprint.com/blog/limitations-ja3-fingerprinting-accurate-device-identification/) +- [JA3 vs JA4: TLS Fingerprinting for Bot Detection in 2026 — VoidMob](https://voidmob.com/blog/ja3-vs-ja4-tls-fingerprinting-bot-detection-2026) +- [FoxIO-LLC/ja4 — README and licensing](https://github.com/FoxIO-LLC/ja4/blob/main/README.md) +- [FoxIO License FAQ](https://github.com/FoxIO-LLC/ja4/blob/main/License%20FAQ.md) +- [How to Use JA4 Network Fingerprints in Zeek (Jan 2026)](https://zeek.org/2026/01/how-to-use-ja4-network-fingerprints-in-zeek/) +- [JA4+ Zeek package](https://packages.zeek.org/packages/view/65d88958-d5f0-11ee-8674-0a598146b5c6) +- [Tap Interfaces — PAN-OS docs](https://docs.paloaltonetworks.com/pan-os/11-0/pan-os-networking-admin/configure-interfaces/tap-interfaces) +- [How to Configure a Palo Alto Networks Device for Tap Mode](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClMzCAK) +- [Retrieve Logs — PAN-OS XML API](https://docs.paloaltonetworks.com/ngfw/api/pan-os-xml-api-request-types-and-actions/retrieve-logs) +- [Replay pcap — Palo Alto LIVEcommunity](https://live.paloaltonetworks.com/t5/general-topics/replay-pcap/td-p/36261) +- [tcpreplay man page](https://tcpreplay.appneta.com/wiki/tcpreplay-man.html) +- [Suricata Feature #432: PCAP-NG support](https://redmine.openinfosecfoundation.org/issues/432) +- [Analysing PCAPNG files with Zeek — Zeek community](https://community.zeek.org/t/analysing-pcapng-files-from-wireshark-traffic-captured-with-zeek-or-spicy/6959) +- [HowTo handle PcapNG files — Netresec](https://www.netresec.com/?page=Blog&month=2012-12&post=HowTo-handle-PcapNG-files) diff --git a/docs/status.yaml b/docs/status.yaml index f2f030e..96d6c1e 100644 --- a/docs/status.yaml +++ b/docs/status.yaml @@ -5,12 +5,12 @@ created: "2026-08-11" github_repo: "DeepTempo/flabel" notion_url: "https://app.notion.com/p/3b92a84a5230812aa64aef638e389725" prd_gdoc_url: "" # Google Doc PRD (set by /project:prd); "" = not created yet -current_stage: research +current_stage: prd # NOTE: issue numbers are not in stage order. Two creates hit a transient TLS # error and were retried, and GitHub never reuses issue numbers. This mapping is # authoritative — trust it over the numeric order in the GitHub issue list. stages: - research: { status: pending, issue: 1, artifact: docs/research.md, completed: null } + research: { status: completed, issue: 1, artifact: docs/research.md, completed: "2026-08-11" } prd: { status: pending, issue: 2, artifact: docs/prd.md, completed: null } eng_review: { status: pending, issue: 6, artifact: docs/eng-review.md, completed: null } plan: { status: pending, issue: 3, artifact: PLAN.md, completed: null } @@ -20,3 +20,4 @@ stages: log: - "2026-08-11 init — repo scaffolded, 7 stage issues filed, Notion row created" - "2026-08-11 init — repo recreated so main is the default branch; old repo renamed to DeepTempo/flabel-old and archived" + - "2026-08-11 stage(research) completed — recommends Suricata over Snort, replay only for PANW, and JA4-as-enrichment instead of JA3/JA4 labelling" From 1ac582bb80837207f7226dca4a70ce6980312f6a Mon Sep 17 00:00:00 2001 From: Craig yourname Date: Tue, 11 Aug 2026 14:53:28 -0700 Subject: [PATCH 3/6] =?UTF-8?q?stage(research):=20resolve=20review=20notes?= =?UTF-8?q?=20=E2=80=94=20decisions=20folded=20in?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Craig reviewed docs/research.md and left inline notes; all 24 resolved. Corrections to my own findings: - Fixed a real bug in the proposed Tier 2 admission filter. The global rule (confidence == High AND severity in Major,Critical) would have excluded 100% of the IOC feeds (abuse.ch, malsilo, pawpatrules), which carry no ET confidence taxonomy — admitting ET Open alone. Replaced with a per-source policy: metadata filter for signature rulesets, wholesale admission plus snapshot provenance for IOC feeds. - Reversed the 'drop JA3' conclusion. ET Open maintains its own emerging-ja3.rules with created_at dates through 2026_03_13, carrying confidence and signature_severity metadata, MIT-licensed, ~100+ rules targeting malware C2. Having found abuse.ch SSLBL frozen at 2021 I generalised wrongly to 'no trustworthy free JA3 source'. ET's rules are kept and confidence-filtered; only the abandoned SSLBL source is dropped. - Narrowed the ClientHello-shuffling critique: it degrades JA3 for browser identification, but ET's rules fingerprint malware TLS stacks, which are typically fixed. - Clarified that '>70% metadata coverage' is not '70% admitted'. Decisions recorded: Suricata; Approach B; vwire not tap; no --topspeed; Positive Technologies excluded; JA4+ approved with Legal engaged; lab is a v1 requirement; NGFW default with --offline for Tier 2 only; millisecond clock accuracy; pcapng supported via editcap normalization; encrypted detection lives in Tier 2 as Suricata rule content via native ja3.hash / ja4.hash keywords; JA4 as enrichment until ET ships ja4 rules. Adds commercial pricing as requested: ET Pro ~$900/sensor/year list (~$750 reseller); Secureworks and Stamus are quote-only. Answers the fingerprint-aging question: JA4/JA3 hashes are deterministic and never reassigned, so aging is about the population of software sharing a fingerprint, not expiry. Filed #10 (untagged ET rules) and #11 (admission-filter measurement). Refs #1 --- docs/research.md | 297 +++++++++++++++++++++++++++-------------------- 1 file changed, 169 insertions(+), 128 deletions(-) diff --git a/docs/research.md b/docs/research.md index 0224cee..0662a10 100644 --- a/docs/research.md +++ b/docs/research.md @@ -1,236 +1,275 @@ # Research — flabel **Stage 1, Part B.** Driven by the approved `docs/research-brief.md` · Issue #1 · 2026-08-11 +**Status:** findings reviewed by Craig; all decisions resolved and folded in below. --- -## Headline recommendation +## Decisions (resolved 2026-08-11) + +| # | Decision | +| --- | --- | +| 1 | **Suricata**, not Snort 3. | +| 2 | **Approach B** — Zeek and Suricata read the capture file directly; replay only for PANW. | +| 3 | PANW deployed as a **virtual-wire pair**, not tap mode. | +| 4 | **No `--topspeed`.** Replay at a controlled rate. | +| 5 | **Positive Technologies sources excluded** (sanctions + non-standard licence). | +| 6 | **JA4+ approved** — use the highest-fidelity option; Legal engaged on FoxIO License 1.1. | +| 7 | **Lab environment is a v1 requirement.** | +| 8 | NGFW **required by default**; a Tier 2-only run is available behind an **`--offline`** flag. | +| 9 | Clock sync accuracy target: **millisecond**. | +| 10 | **pcapng supported** via `editcap -F pcap` normalization, with multi-datalink caveats noted. | +| 11 | Encrypted-traffic detection lives in **Tier 2**, as Suricata rule content — not a separate trust tier. | +| 12 | **ET Open's JA3 rules kept** (confidence-filtered); the abandoned abuse.ch SSLBL JA3 source dropped. | +| 13 | Untagged ET Open rules **excluded for now**; enhancement issue filed to revisit. | +| 14 | Admission-filter impact **measured at build time**, not estimated now. | +| 15 | No free/OSS NGFW-equivalent line of inquiry — closed as not worth pursuing. | + +--- -**Run Suricata and Zeek directly against the capture file. Replay only for PANW.** +## Headline recommendation -The brief's flow sends everything through the inline device via replay. That isn't necessary: Suricata and Zeek both read a pcap natively (`suricata -r`, `zeek -r`), and only PANW VM-Series has no offline pcap ingestion — Palo Alto's own guidance is to build a virtual-wire pair and replay at it from an external server ([LIVEcommunity](https://live.paloaltonetworks.com/t5/general-topics/replay-pcap/td-p/36261)). +**Run Suricata and Zeek directly against the capture file. Replay only for PANW, over a virtual-wire pair, at a controlled rate.** -That split matters because **replay is the single largest threat to label fidelity.** Rewritten packet timing can disturb reassembly and rate-based rules, and a dropped packet is a *missing label*. Keeping the Tier 2 and Tier 3 paths offline makes them deterministic and byte-for-byte reproducible, and confines the nondeterminism to the one source that cannot avoid it. It also means a Tier 2/Tier 3-only run needs **no lab environment at all** — useful for testing and for captures processed before the lab exists. +The brief's flow sent everything through the inline device. That isn't necessary: Suricata and Zeek both read a pcap natively (`suricata -r`, `zeek -r`), and only PANW VM-Series has no offline pcap ingestion — Palo Alto's own guidance is a virtual-wire pair fed by an external replay server ([LIVEcommunity](https://live.paloaltonetworks.com/t5/general-topics/replay-pcap/td-p/36261)). -**Second recommendation: drop JA3 in favour of JA4, and treat fingerprint hits as enrichment rather than labels.** Evidence below — this is the most significant finding in this research and it contradicts the brief's assumption that JA3/JA4 feeds can supply trustworthy labels. +That split matters because **replay is the single largest threat to label fidelity.** Rewritten packet timing can disturb reassembly and rate-based rules, and a dropped packet is a *missing label* — invisible in the output. Keeping the Tier 2 path offline makes it deterministic and byte-for-byte reproducible, and confines nondeterminism to the one source that cannot avoid it. It also makes the `--offline` mode a natural consequence of the architecture rather than a bolted-on special case. --- ## A. Content-inspection sources and rulesets -### A1. Snort 3 or Suricata? → **Suricata** +### A1. Suricata, not Snort 3 -Your stated tiebreaker was volume of high-confidence rulesets, and Suricata wins it decisively — but on a subtler axis than raw rule count. +The stated tiebreaker was volume of high-confidence rulesets, and Suricata wins it — on a subtler axis than raw rule count. | | Suricata | Snort 3 | | --- | --- | --- | -| Free ruleset ecosystem | ~30 sources indexed centrally (below) | Talos registered + community | +| Free ruleset ecosystem | ~30 sources indexed centrally | Talos registered + community | | High-confidence *selection* mechanism | Per-rule `confidence` / `signature_severity` metadata | Policy tiers (connectivity / balanced / security / max-detect) | | Free-tier delay | ET Open: none | Registered: 30 days behind subscriber, no zero-days | -| Rule compatibility | Reads Snort rules (imperfectly) | Cannot read Suricata-native keywords | | Native pcap ingest | Yes (`-r`) | Yes (`-r`) | +| Native TLS fingerprint matching | **`ja3.hash` and `ja4.hash` rule keywords** | No equivalent | -Two decisive points. First, Suricata has a **central, machine-readable index of free rule sources** — `OISF/suricata-intel-index` — with a declared licence per source, which is exactly the provenance record your brief demands and Snort has no equivalent of. Second, Suricata-native rules can express protocol fields Snort cannot, so equivalent detections are more precise with fewer false positives ([comparison](https://www.decryptiondigest.com/blog/snort-vs-suricata-ids-ips-comparison)). One study measured Suricata running ~4,600 more enabled rules than Snort on the same traffic. - -Snort's counter-argument is real and worth recording: Talos LightSPD is maintained by **dedicated paid researchers**, whereas ET Open is largely community-produced ([Snort blog](https://blog.snort.org/2020/12/soft-release-lightspd-new-rules-package.html)). If uniform curation mattered more than breadth, Snort would win. Under per-source tiering — where we curate the input set ourselves — breadth plus filterable metadata is more useful. +Three decisive points. Suricata has a **central, machine-readable index of free rule sources** (`OISF/suricata-intel-index`) with a declared licence per source — exactly the provenance record the brief demands, and Snort has no equivalent. Suricata-native rules express protocol fields Snort cannot, so equivalent detections are more precise with fewer false positives ([comparison](https://www.decryptiondigest.com/blog/snort-vs-suricata-ids-ips-comparison)). And critically for this project, Suricata matches TLS fingerprints natively as rule content — see section B. -**Nothing prevents running both.** They'd both be Tier 2, and deduplication would be needed. Not recommended for v1. +Snort's counter-argument, recorded for the record: Talos LightSPD is maintained by dedicated paid researchers, whereas ET Open is largely community-produced ([Snort blog](https://blog.snort.org/2020/12/soft-release-lightspd-new-rules-package.html)). If uniform curation mattered more than breadth, Snort would win. -### A2. Free rule sources, with licence status +### A2. Rule sources and licence status -From `OISF/suricata-intel-index` (authoritative, machine-readable). **Licence status recorded as required — including the non-standard cases.** +From `OISF/suricata-intel-index` (authoritative, machine-readable). -**Recommended for Tier 2 admission:** +**Admitted:** -| Source | Vendor | Licence | Notes | +| Source | Vendor | Licence | Admission policy | | --- | --- | --- | --- | -| `et/open` | Proofpoint | **MIT** | Primary set. Filter by metadata — see A3. | -| `oisf/trafficid` | OISF | **MIT** | Traffic identification, not threat detection. Enrichment. | -| `abuse.ch/feodotracker` | abuse.ch | **CC0-1.0** | Botnet C2 IPs, actively maintained. | -| `abuse.ch/urlhaus` | abuse.ch | **CC0-1.0** | Malware-distribution URLs. | -| `abuse.ch/sslbl-c2` | abuse.ch | **CC0-1.0** | C2 servers by blacklisted certificate. | -| `malsilo/win-malware` | malsilo | **MIT** | Windows malware artifacts. Small, focused. | -| `stamus/lateral` | Stamus Networks | **GPL-3.0-only** | Lateral movement. Copyleft — rules only, no linking concern. | -| `the-hunters-ledger/open` | The Hunters Ledger | **CC-BY-4.0** | Attribution required. Derived from malware investigations. | -| `pawpatrules` | pawpatrules | **CC-BY-SA-4.0** | Share-alike. Broad scope; needs FP review. | - -**Excluded, with reasons:** - -| Source | Licence | Why excluded | +| `et/open` | Proofpoint | **MIT** | Metadata filter (A3). Includes `emerging-ja3.rules` — see B. | +| `oisf/trafficid` | OISF | **MIT** | Wholesale; identification only, contributes no verdicts | +| `abuse.ch/feodotracker` | abuse.ch | **CC0-1.0** | Wholesale (IOC feed) | +| `abuse.ch/urlhaus` | abuse.ch | **CC0-1.0** | Wholesale (IOC feed) | +| `abuse.ch/sslbl-c2` | abuse.ch | **CC0-1.0** | Wholesale (IOC feed) — certificate-based, unaffected by the JA3 issue | +| `sslbl/ssl-fp-blacklist` | abuse.ch | **CC0-1.0** | Wholesale (IOC feed) — certificate-based | +| `malsilo/win-malware` | malsilo | **MIT** | Wholesale; small and focused | +| `stamus/lateral` | Stamus Networks | **GPL-3.0-only** | Wholesale; copyleft applies to rules, no linking concern | +| `the-hunters-ledger/open` | The Hunters Ledger | **CC-BY-4.0** | Wholesale; attribution required | +| `pawpatrules` | pawpatrules | **CC-BY-SA-4.0** | Wholesale, pending FP review; share-alike | + +**Excluded:** + +| Source | Licence | Why | | --- | --- | --- | -| `tgreen/hunting` | GPLv3 | Self-described **hunting / anomaly-detection** ruleset. Your brief explicitly excludes threat-hunting rules in favour of detection. Policy exclusion, not a quality judgement. | -| `etnetera/aggressive` | MIT | Self-described "**aggressive** IP blacklist". Incompatible with a low-FP bar. | -| `ptresearch/attackdetection`, `ptrules/open` | **Custom** (non-standard) | Two concerns. The licence is non-standard — acceptable per your guidance if noted, and it is noted. But the vendor, **Positive Technologies, has been under US sanctions since April 2021**. Against a requirement for "highly trusted sources," a sanctioned entity is a provenance problem independent of rule quality, and plausibly a compliance one for DeepTempo. **Recommend exclude; flagging as your call.** | -| `et/pro`, `scwx/*`, `stamus/nrd-*` | Commercial | Require paid subscription. `scwx/malware` is notable — self-described "**high-fidelity, high-priority**" — and is the closest thing to a purpose-built low-FP set. Worth pricing if free sources underdeliver. | +| `sslbl/ja3-fingerprints` | CC0-1.0 | **Abandoned** — newest entry 2021-08-03; self-declared untested FP posture (B1) | +| `tgreen/hunting` | GPLv3 | Hunting/anomaly ruleset; brief excludes hunting in favour of detection | +| `etnetera/aggressive` | MIT | Self-described "aggressive" blacklist; incompatible with a low-FP bar | +| `ptresearch/attackdetection`, `ptrules/open` | **Custom** (non-standard) | Non-standard licence *and* vendor under US sanctions since April 2021. **Excluded by decision.** | -**Government sources:** no government body publishes a maintained, general-purpose Suricata/Snort ruleset. CISA and allied agencies publish Snort signatures *inside individual advisories*, which are point-in-time IOCs rather than a feed. Harvesting them would mean scraping advisories — real work, low yield, no update cadence. **Recommend: not a v1 source.** This is a gap against your "government sources" ask, and I'd rather say so than pad the list. +**Commercial options, priced as requested:** -### A3. Tier 2 rule-admission criteria → filter on ET metadata +| Source | Price | Notes | +| --- | --- | --- | +| `et/pro` | **~$900/sensor/year** list; ~$750 via the OPNsense reseller | The only publicly-priced option found. Proofpoint raised prices post-acquisition; older subscriptions grandfathered. | +| `scwx/enhanced`, `scwx/malware`, `scwx/security` | **Not published — quote only** | `scwx/malware` is self-described "high-fidelity, high-priority", the closest thing to a purpose-built low-FP set. Requires contacting Secureworks. | +| `stamus/nrd-*` | **Not published — quote only** | Newly-registered-domain feeds. Requires contacting Stamus. | -This is where per-source tiering gets its false-positive defence. +**Government sources:** no government body publishes a maintained, general-purpose Suricata/Snort ruleset. CISA and allied agencies publish signatures *inside individual advisories* — point-in-time IOCs with no feed or update cadence. Harvesting them means scraping advisories: real work, low yield. Not a v1 source. This is a stated gap against the "government sources" ask rather than a padded list. -ET introduced a `confidence` metadata tag in 2022 specifically to express **false-positive likelihood** — "High" confidence means minimal FP likelihood. Coverage has grown from 30% to **over 70% of the ruleset, with newer rules at 100%** ([Proofpoint](https://www.proofpoint.com/us/blog/threat-insight/emerging-threats-updates-improve-metadata-including-mitre-attck-tags), [ET wiki](https://community.emergingthreats.net/t/signature-metadata/96)). `signature_severity` runs Informational → Critical. +### A3. Tier 2 admission — a **per-source** policy -**Proposed admission rule:** +An earlier draft proposed one global filter: `confidence == High AND signature_severity in (Major, Critical) AND source in `. **That was wrong, and applying it would have silently deleted most of the admitted sources.** The abuse.ch, malsilo, and pawpatrules sources are IOC-match rulesets that don't carry ET's `confidence` taxonomy at all, so the condition evaluates false for every rule in them. The filter would have admitted ET Open alone. + +The correct design is a per-source policy, because the two source classes control false positives by different means: + +**Class 1 — signature rulesets (ET Open).** FP risk lives in the rule logic, and ET expresses it per rule. Filter: ``` -admit if metadata.confidence == High - and metadata.signature_severity in (Major, Critical) - and source in the recommended table above +admit if metadata.confidence == High + and metadata.signature_severity in (Major, Critical) ``` -Two consequences to decide at PRD time: +ET introduced `confidence` in 2022 specifically to express FP likelihood; coverage has grown from 30% to over 70% of the ruleset, with newer rules at 100% ([Proofpoint](https://www.proofpoint.com/us/blog/threat-insight/emerging-threats-updates-improve-metadata-including-mitre-attck-tags), [ET wiki](https://community.emergingthreats.net/t/signature-metadata/96)). + +**Class 2 — IOC feeds (abuse.ch, malsilo, and similar).** FP risk lives in the *indicator list*, not the rule logic — the rule is an exact match on a C2 IP, URL, or certificate. There is no per-rule confidence to filter on, and the curation happened upstream. Admit wholesale, and record the feed snapshot date as the provenance. -- **The <30% of ET Open rules lacking a `confidence` tag are excluded** by this rule. Fail-closed is the right default when trustworthiness is paramount, but it discards untagged rules that may be good. -- The admitted rule set must be **snapshotted and recorded per run** (source + version + date), because the filter's output changes as ET revises metadata. Without that, two runs of "the same" flabel produce differently-grounded labels. +**A clarification on my own earlier framing:** ">70% coverage" is not "we keep 70%". Coverage means the tag *exists*; admission additionally requires it to equal `High` **and** severity to be Major/Critical. The admitted fraction is materially smaller than 70%. **The exact figure is deferred to build-time measurement** (`suricata-update` with the filter applied, counted per source) rather than estimated from a partial sample. + +**Required either way:** the admitted rule set must be **snapshotted per run** — source, version, and date — because the filter's output changes as ET revises metadata. Without that, two runs of "the same" flabel produce differently-grounded labels. ### A4. PANW VM-Series as Tier 1 -- **Tap mode works for our purpose.** In tap mode the firewall cannot block or reset, so every security profile action is set to `alert` — which is precisely what a labeller wants. It still performs App-ID and threat identification, and writes to the threat log ([PANW docs](https://docs.paloaltonetworks.com/pan-os/11-0/pan-os-networking-admin/configure-interfaces/tap-interfaces)). -- **But Palo Alto's own replay guidance is a virtual-wire pair**, not tap. Vwire is inline-but-transparent and is the configuration their community recommends for replaying a pcap at the device. Your brief prefers tap. **Unresolved:** whether tap-plus-mirror and vwire yield identical detection sets. This needs an empirical check on the actual device — see Open Questions. -- **Threat name is available.** PAN-OS threat logs carry the threat/content name, and the XML API retrieves Threat-type logs with filter expressions equivalent to the Monitor tab, including a `receive_time` field ([Retrieve Logs](https://docs.paloaltonetworks.com/ngfw/api/pan-os-xml-api-request-types-and-actions/retrieve-logs)). `kevinsteves/pan-python` is the mature Python client. **Not yet verified:** the exact filter syntax for a bounded `receive_time` range, and whether log-write latency requires a settling delay before querying. Verify on-device. +- **Deployment: virtual-wire pair** (decided). Vwire is inline-but-transparent and is what Palo Alto's community recommends for replaying a capture at the device. Tap mode would also have worked functionally — it forces every security-profile action to `alert`, which is what a labeller wants — but vwire avoids the unresolved question of whether tap forfeits a class of detections. +- **Threat name is available.** PAN-OS threat logs carry the threat/content name, and the XML API retrieves Threat-type logs with Monitor-tab-equivalent filter expressions including `receive_time` ([Retrieve Logs](https://docs.paloaltonetworks.com/ngfw/api/pan-os-xml-api-request-types-and-actions/retrieve-logs)). `kevinsteves/pan-python` is the mature Python client. +- **Still to verify on-device:** exact filter syntax for a bounded `receive_time` range, and whether log-write latency requires a settling delay before querying. -### A5. Free L7 equivalent to PANW App-ID → partial, not equivalent +### A5. Free L7 equivalent to App-ID -Zeek plus Suricata's app-layer protocol detection covers protocol identification well. Neither reproduces App-ID's application-level catalogue. `oisf/trafficid` adds some identification. **Conclusion: no free equivalent; PANW's Tier 1 contribution is genuinely distinct.** That is an argument *for* keeping the NGFW, consistent with your decision. +Closed by decision — not pursued further. For the record: Zeek plus Suricata's app-layer detection covers protocol identification, but nothing free reproduces App-ID's application catalogue. PANW's Tier 1 contribution is genuinely distinct, which argues for keeping the NGFW. --- -## B. Encrypted traffic — JA3/JA4 +## B. Encrypted traffic detection -This section changed my view of the brief's design. Reporting it plainly. +**Design:** encrypted-traffic detection is **Tier 2 Suricata rule content**, not a separate trust tier. Suricata matches TLS fingerprints natively via the **`ja3.hash` and `ja4.hash` rule keywords** ([Suricata JA3/JA4 keywords](https://docs.suricata.io/en/latest/rules/ja-keywords.html)), enabled by setting `app-layer.protocols.tls.ja{3,4}-fingerprints` (auto-enabled when a loaded rule requires it). This means no separate matching engine and no separate feed pipeline — fingerprint detections inherit the same admission filter, snapshot provenance, and tiering as every other Tier 2 rule. -### B1. The canonical free JA3 feed is abandoned and self-declares an FP problem +### B1. The abandoned source: abuse.ch SSLBL JA3 — **dropped** -abuse.ch SSLBL's JA3 fingerprint blacklist is the source the brief implicitly assumes. Fetched directly: +Verified directly: -- **Most recent listed fingerprint: `2021-08-03`.** Five years stale. The *file* regenerates every 5 minutes, so automated freshness checks that watch file mtime would report it healthy — the *content* is frozen. -- The page carries this warning: **"These fingerprints have not been tested against known good traffic yet and may cause a significant amount of FPs!"** -- Licence is **CC0-1.0** — unrestricted, commercial use fine. Licensing is not the problem; provenance quality is. +- **Newest listed fingerprint: `2021-08-03`.** The *file* regenerates every 5 minutes, so an mtime-based freshness check would report it healthy — the *content* is frozen. +- The page warns: **"These fingerprints have not been tested against known good traffic yet and may cause a significant amount of FPs!"** +- Licence is CC0-1.0. Licensing was never the problem; provenance quality is. -A feed that is both abandoned and self-declared as untested against benign traffic cannot be a labelling source for ML training data under a per-source trust model with no per-rule filter. +A feed that is both abandoned and self-declared as untested against benign traffic cannot supply labels for training data. -### B2. JA3 is structurally degraded, independent of feed quality +### B2. The maintained source: ET Open `emerging-ja3.rules` — **kept, confidence-filtered** -- Chrome and Firefox **shuffle ClientHello extension order**, which changes the JA3 hash for the same client. Stable JA3 values can no longer be assumed. -- JA3's limited attribute set produces **collisions** — unrelated clients sharing a fingerprint, so benign traffic can match a "malicious" JA3 ([Fingerprint.com](https://fingerprint.com/blog/limitations-ja3-fingerprinting-accurate-device-identification/)). +This corrects an earlier conclusion in this document. Having found SSLBL dead, an earlier draft generalized to "JA3 has no trustworthy free source." That was wrong. Inspecting `emerging-ja3.rules` directly: -For training data this failure mode is the expensive one: a collision mislabels *benign* traffic as malicious, teaching the model the wrong thing. +- Rules carry `created_at` dates from **2019_09_10 through 2026_03_13** — actively maintained. +- Every rule carries **`confidence` and `signature_severity` metadata**, so it plugs straight into the A3 Class 1 filter. Values range across `confidence Low` (excluded) to `confidence High` (admitted). +- ~100+ active rules, MIT-licensed, targeting malware C2 with `classtype:command-and-control` — Cobalt Strike Malleable C2, Remcos, Trickbot among them. +- Provenance is mixed: some rules credit abuse.ch/SSLBL, others cite malware-traffic-analysis.net or ET's own research. -### B3. JA4 is the successor, is maintained, and is partly licence-restricted +**This is a trustworthy free source, and it is one we already accept as Tier 2.** It is the only free fingerprint labelling content available today. -- **Maintained:** `zkg install zeek/foxio/ja4`, v0.18.8. Zeek 5+ supported, Zeek 6+ for QUIC. Zeek published a how-to in January 2026 ([zeek.org](https://zeek.org/2026/01/how-to-use-ja4-network-fingerprints-in-zeek/)). -- JA4 sorts extensions, so it is **resistant to the shuffling that breaks JA3**. -- **Licence split matters:** - - **JA4 (TLS client) — BSD 3-Clause**, with FoxIO explicitly claiming no patent rights. - - **JA4+ (JA4S, JA4H, JA4X, JA4L, JA4SSH, JA4T, …) — FoxIO License 1.1: non-commercial only.** Internal use securing your own company is permitted; selling it in a product requires an OEM licence from FoxIO. +### B3. On JA3's known weaknesses — narrower than they first appear -**flabel produces training data for detection models.** If those models ship in a DeepTempo product, using JA4+ plausibly constitutes monetization. **Recommendation: use only plain JA4 (BSD), avoid the JA4+ extensions.** That keeps the project unambiguously clear of the restriction. This is a legal question, not an engineering one — flagged for your decision, and it is exactly the licence-status issue your correction to the brief anticipated. +JA3's documented problems are real but apply unevenly: -### B4. Can a threat *name* come from a fingerprint? → **No, not reliably** +- **ClientHello extension shuffling** (Chrome, Firefox) changes the JA3 hash for the same client. This degrades JA3 for identifying **browsers**. ET's rules fingerprint **malware TLS stacks**, which are typically fixed — so the effect on these detections is much smaller than a general critique of JA3 implies. +- **Collisions** remain a genuine risk: JA3's limited attribute set means unrelated clients can share a fingerprint ([Fingerprint.com](https://fingerprint.com/blog/limitations-ja3-fingerprinting-accurate-device-identification/)). This is the expensive failure mode for training data, because a collision mislabels *benign* traffic as malicious. Mitigation is the `confidence High` filter — ET assigns confidence with FP likelihood explicitly in mind. -A fingerprint match yields "this client matches a known-bad fingerprint." Malware-family attribution only exists if the feed supplies it, and the one CC0 feed that did is frozen at 2021. `ja4db` (FoxIO) catalogues fingerprint→application mappings for identification, not malicious verdicts. **Answer: a Tier 3 label would carry no trustworthy threat name.** +### B4. JA4 — mechanism ready, free content not yet -### B5. Recommendation: Tier 3 is **enrichment, not a label** +- **Maintained:** `zkg install zeek/foxio/ja4`, v0.18.8. Zeek 5+ supported, Zeek 6+ for QUIC. Zeek published a how-to in January 2026 ([zeek.org](https://zeek.org/2026/01/how-to-use-ja4-network-fingerprints-in-zeek/)). +- JA4 **sorts extensions**, making it resistant to the shuffling that destabilizes JA3 — technically the better fingerprint. +- **But ET Open ships no JA4 rules yet**, and no free maintained JA4 malicious-verdict feed exists. `ja4db` (FoxIO) catalogues fingerprint→application mappings for identification, not malicious verdicts. Organizations currently build their own JA4 blocklists. + +**Therefore:** compute JA4 (and JA4+) via the Zeek plugin on every TLS connection and record it as an **attribute** in the Zeek output and alongside labels — valuable as a model feature and an analyst pivot — but **do not emit a label on a JA4 match**, because there is no verdict source to match against. **Promote JA4 to labelling the moment ET publishes `ja4.hash` rules**, at which point it flows through the existing Tier 2 path with no architectural change. + +**Licensing:** plain JA4 (TLS client) is **BSD 3-Clause** with no patent claims. The JA4+ suite (JA4S, JA4H, JA4X, JA4L, JA4SSH, JA4T) is **FoxIO License 1.1 — non-commercial**. JA4+ is approved for use on the highest-fidelity basis, with Legal engaged on the licence question. Recording the exposure precisely: internal use securing your own company is permitted; shipping it in a product requires an OEM licence from FoxIO. -Combining B1–B4: no maintained, trustworthy, free malicious-fingerprint feed exists; JA3 is collision-prone and unstable; JA4 is sound as a *fingerprint* but has no reputable free malicious-verdict feed behind it; and no threat name is derivable. +### B5. Does fingerprint aging matter? — **differently than for IP/domain IOCs** -**Recommend:** compute JA4 for every TLS connection and record it in the Zeek output and alongside labels as an *attribute*, but **do not emit a malicious label on fingerprint match alone.** This preserves all the analytic value — a model can learn from JA4 as a feature, and analysts can pivot on it — without asserting a verdict the evidence doesn't support. +A JA4 or JA3 hash is a deterministic function of the TLS ClientHello, so unlike an IP address it is never reassigned. What changes over time is the **population of software sharing a fingerprint**: -This contradicts the brief, which treats JA3/JA4 as a second labelling source. It's your call to overrule; I'd rather flag it now than produce labels I can't defend. +- Malware updating its TLS library **changes its fingerprint** → false negatives, not false positives. The old entry becomes inert rather than harmful. +- A fingerprint tied to a library version gets **adopted by more benign software** as that library spreads → a once-distinctive fingerprint becomes shared, and the verdict silently becomes wrong. This is the failure that matters. +- JA4's extension sorting makes it **more stable than JA3**, so drift is slower. -### B6. Feed deconfliction +**Implication:** age the *fingerprint→verdict assertion*, not the fingerprint, and track first-seen/last-seen per feed. Aging is less about expiry than about detecting when a fingerprint has become too common to carry a verdict. In practice ET handles this for us by revising rules and their confidence values — which is another argument for sourcing fingerprint verdicts through ET rather than a raw feed. -Largely moot if B5 is accepted — with no feed supplying verdicts, there is nothing to deconflict. Should you keep fingerprint labelling, the required design is: per-feed provenance retained per fingerprint, never silently merged; disagreement resolved by recording *all* asserting feeds rather than voting; and a snapshot date pinned per run. +### B6. Deconfliction + +Simple deconfliction is sufficient. With fingerprint verdicts arriving as ET rules rather than raw feeds, deduplication happens at the detection level like any other Suricata rule: retain per-rule provenance (SID, rev, ruleset snapshot), never silently merge, and record all asserting rules rather than voting. If additional fingerprint feeds are added later, keep per-feed provenance per fingerprint and pin a snapshot date per run. --- ## C. Architecture, replay fidelity, and formats -### C1. Three viable architectures +### C1. Chosen architecture — Approach B (hybrid) -**Approach A — Offline only (no NGFW).** `zeek -r` + `suricata -r`, merge, emit. One host, no lab, no clocks, fully deterministic and reproducible. **Excluded by your Tier 1 decision**, but it is the correct v0/test configuration and the fallback if the lab is delayed. +**Approach B — offline OSS + replay for PANW only. ✅ Chosen.** Zeek and Suricata read the file directly; only PANW receives a replay, over a virtual-wire pair. Confines all replay-fidelity and clock-correlation risk to the Tier 1 path; Tier 2 stays deterministic and reproducible. Costs one extra concept: two ingest paths for one capture. -**Approach B — Hybrid: offline OSS + replay for PANW only. ⭐ Recommended.** Zeek and Suricata read the file directly; only PANW gets a replay. Confines all replay-fidelity and clock-correlation risk to the Tier 1 path, and Tier 2/3 stay reproducible. Costs one extra concept: two ingest paths for one capture. +The `--offline` flag runs the Tier 2 path alone, with the output clearly marked as lacking Tier 1 coverage. NGFW is required by default. -**Approach C — Full replay (as briefed).** Everything past the tap. Matches the original design, and is arguably more "realistic". But it makes the Suricata path nondeterministic and drop-prone **for no benefit**, since Suricata reads the file perfectly well. Not recommended. +*Considered and rejected:* **Approach A** (offline only, no NGFW) — excluded because the lab is a v1 requirement, though it remains the natural test configuration and is effectively what `--offline` provides. **Approach C** (full replay, as briefed) — rejected because it makes the Suricata path nondeterministic and drop-prone for no benefit. ### C2. Replay fidelity — the top risk -- `--topspeed` explicitly trades timing accuracy for speed; tcpreplay's own docs note that batching packets for throughput costs accuracy ([tcpreplay man](https://tcpreplay.appneta.com/wiki/tcpreplay-man.html)). -- Rewritten timing can affect stateful reassembly, flow timeouts, and any rate-based rule. **A dropped or reordered packet is a missing label, not a wrong one** — and a missing label is invisible in the output. -- **De-risking:** replay at a controlled rate rather than `--topspeed`; verify packet counts sent vs. seen; run the same capture twice and diff the detection sets — instability across identical runs quantifies the problem directly. I could not find published measurements of `--topspeed`-induced missed alerts; this needs empirical measurement in the lab, not a literature answer. +**`--topspeed` is dropped by decision.** Its own documentation notes that batching packets for throughput costs timing accuracy ([tcpreplay man](https://tcpreplay.appneta.com/wiki/tcpreplay-man.html)), and rewritten timing can affect stateful reassembly, flow timeouts, and rate-based rules. Replay at a controlled rate instead. + +**A dropped or reordered packet is a missing label, not a wrong one** — and nothing in the output announces it. Required controls: + +- Assert packets sent vs. packets seen by the device; fail the run on mismatch rather than emitting a silently incomplete label set. +- Replay the same capture twice and diff the detection sets; instability across identical runs quantifies the problem directly. +- No published measurements of replay-induced missed alerts were found; this needs empirical measurement in the lab. ### C3. Clock and correlation -Bounding the PANW log query by the replay window requires that flabel's clock and the firewall's agree. NTP is necessary but I found **no published figure** for the required accuracy — it depends on log-write latency and how tightly the window is drawn. Practical approach: record replay start/end from flabel, pad the query window generously, then filter returned records by flow tuple rather than relying on time alone. Time bounds the query; the tuple does the matching. +**Target accuracy: millisecond**, via NTP across all hosts. That is tight enough that the query window can be drawn closely around the replay, but the correlation design should not depend on it: -**Unsolved and worth PRD attention:** detections are stamped at *replay* time, but labels must reference the *capture's* original timeline. The mapping is not 1:1 under `--topspeed`, because the replay compresses the capture's duration. Correlation should therefore be tuple-driven, with replay-time used only to scope the query. +- Record replay start/end from flabel, pad the query window, then **match returned records by flow tuple** rather than by time. Time bounds the query; the tuple does the matching. +- **Detections are stamped at replay time, but labels must reference the capture's original timeline.** Even at a controlled rate the mapping is not 1:1, since replay does not reproduce the capture's wall-clock duration. Tuple-driven correlation is therefore not just more robust — it's necessary. +- Port reuse within a single capture can make a tuple ambiguous. Behaviour for an unmatchable detection (drop vs. emit unmatched) must be defined at PRD. -### C4. pcap format support matrix +### C4. pcap format support — normalization required -| Component | pcap | pcapng | Notes | +| Component | pcap | pcapng | Handling | | --- | --- | --- | --- | -| Zeek | Yes | **No** | `zeek -r` on pcapng produces parser errors; conversion via `editcap -F pcap` required ([Zeek community](https://community.zeek.org/t/analysing-pcapng-files-from-wireshark-traffic-captured-with-zeek-or-spicy/6959)) | -| Suricata | Yes | **Partial** | Reads pcapng 1.0; breaks on multi-interface files with differing datalinks ([Feature #432](https://redmine.openinfosecfoundation.org/issues/432)) | -| tcpreplay | Yes | Partial | Same multi-datalink caveat | -| PANW | n/a | n/a | No file ingest at all — replay only | +| Zeek | Yes | **No** | Convert with `editcap -F pcap`; `zeek -r` on pcapng produces parser errors ([Zeek community](https://community.zeek.org/t/analysing-pcapng-files-from-wireshark-traffic-captured-with-zeek-or-spicy/6959)) | +| Suricata | Yes | **Partial** | Reads pcapng 1.0; breaks on multi-interface files with differing datalinks ([Feature #432](https://redmine.openinfosecfoundation.org/issues/432)). Feed it the normalized pcap for consistency. | +| tcpreplay | Yes | Partial | Same multi-datalink caveat; feed the normalized pcap | +| PANW | n/a | n/a | No file ingest — replay only. Not an issue. | -**This kills "all commonly supported formats" as a naive requirement.** Since pcapng is what Wireshark produces by default, flabel needs an **ingest normalization stage**: detect format, convert pcapng → pcap with `editcap`, split multi-datalink captures first (`frame.interface_id` / `frame.dlt`), and record the conversion in provenance. Gzipped input needs decompression too. This is a real work item the brief didn't anticipate. +**pcapng is supported, via a normalization stage** — not by hoping each component copes. Required behaviour: detect input format; decompress gzipped input; split multi-datalink captures first (`frame.interface_id` / `frame.dlt`); convert to pcap with `editcap -F pcap`; feed all three consumers the same normalized file so they see identical bytes; and **record the conversion in provenance**, since a converted capture is not the original artifact. -### C5. Trust tier assignments +### C5. Trust tiers -| Tier | Source | Justification | +| Tier | Source | Basis | | --- | --- | --- | -| **1** | PANW VM-Series | Commercially curated signatures, named threats, App-ID coverage with no free equivalent. Your decision. | -| **2** | Suricata + metadata-filtered ET Open and the CC0/MIT sources in A2 | Per-rule `confidence: High` is a vendor-declared low-FP assertion; admission filter is machine-checkable and snapshottable. | -| **3** | JA4 fingerprints | **Recommend: enrichment only, not a labelling tier** (see B5). No maintained trustworthy verdict feed; no derivable threat name. | +| **1** | PANW VM-Series (vwire) | Commercially curated signatures, named threats, App-ID coverage with no free equivalent | +| **2** | Suricata — ET Open (metadata-filtered, **including `emerging-ja3.rules`**) plus the CC0/MIT IOC feeds | Per-rule `confidence: High` is a vendor-declared low-FP assertion for signature rules; upstream curation plus snapshot provenance for IOC feeds | +| *Enrichment* | JA4 / JA4+ via Zeek | Recorded as an attribute, not a verdict — no free verdict source yet. Promote into Tier 2 when ET ships `ja4.hash` rules. | --- -## Existing tools and maintenance status +## Tools and maintenance status | Tool | Status | Role | | --- | --- | --- | -| Zeek | Actively maintained; 8.x current | Logs, flow `uid`, JA4 host | -| `zeek/foxio/ja4` | Active, v0.18.8, Zeek 5+/6+ | JA4 computation | -| Suricata | Active, 8.x stable, 9.0 in dev | Tier 2 engine | -| `suricata-update` | Active, ships the source index | Ruleset fetch + filter | +| Zeek | Actively maintained; 8.x current | Logs, flow `uid`, JA4 computation | +| `zeek/foxio/ja4` | Active, v0.18.8, Zeek 5+/6+ | JA4 / JA4+ enrichment | +| Suricata | Active, 8.x stable, 9.0 in dev | Tier 2 engine; native `ja3.hash` / `ja4.hash` | +| `suricata-update` | Active; ships the source index | Ruleset fetch, filter, snapshot | | `OISF/suricata-intel-index` | Active | Licence/provenance record per source | -| tcpreplay | Maintained (AppNeta) | Replay to PANW | +| ET Open `emerging-ja3.rules` | Active — rules created through 2026_03_13 | Tier 2 encrypted-traffic detection | +| tcpreplay | Maintained (AppNeta) | Controlled-rate replay to PANW | | Wireshark `editcap` | Active | pcapng → pcap normalization | | `kevinsteves/pan-python` | Mature; verify recent activity before adopting | PAN-OS XML API client | -| abuse.ch CC0 feeds (feodotracker, urlhaus, sslbl-c2) | Active, 5-min regeneration | Tier 2 IOC rules | -| abuse.ch SSLBL **JA3** list | **Effectively abandoned — newest entry 2021-08-03** | Do not use | +| abuse.ch CC0 feeds (feodotracker, urlhaus, sslbl-c2, ssl-fp-blacklist) | Active, 5-min regeneration | Tier 2 IOC rules | +| abuse.ch SSLBL **JA3** list | **Abandoned — newest entry 2021-08-03** | Excluded | --- ## Top 5 risks, ranked -1. **Replay infidelity silently drops labels (Tier 1).** A missing label is invisible — nothing in the output says "we lost a packet". *De-risk:* Approach B confines it to the PANW path; assert sent-vs-seen packet counts; diff repeat runs; avoid `--topspeed` in favour of a controlled rate. -2. **Trust-by-construction is unfalsifiable.** With per-source tiering and no validation corpus (your decision), the trustworthiness claim rests entirely on ruleset curation and cannot be measured. If a consumer of these labels ever asks "what's your false-positive rate?", there is no answer. *De-risk:* snapshot rulesets per run so labels are at least *reproducible* and auditable; revisit at eng-review. -3. **JA3/JA4 labelling would inject false positives into training data.** Abandoned feed plus collision-prone fingerprints. *De-risk:* adopt B5 — enrichment, not labels. -4. **JA4+ licensing exposure.** FoxIO License 1.1 forbids monetization; flabel feeds product models. *De-risk:* restrict to BSD-licensed plain JA4; get a legal read before touching JA4+. -5. **Correlating PANW detections back to capture flows.** Replay-time stamps, `--topspeed` time compression, and possible port reuse within a capture. *De-risk:* tuple-driven matching with time only scoping the query; define the unmatchable-detection behaviour explicitly (drop vs. emit unmatched) at PRD. - ---- - -## What I need from you to make a full recommendation - -1. A ruling on **JA4 as enrichment vs. label** (B5) — it changes the schema and the whole Tier 3 story. -2. A ruling on **Positive Technologies** sources given the sanctions status (A2). -3. Whether **JA4+ licensing** needs legal review, or whether restricting to plain JA4 settles it (B3). -4. Whether the **lab environment is a v1 prerequisite** or whether an Approach-A offline mode ships first while the lab is built. +1. **Replay infidelity silently drops labels (Tier 1).** A missing label is invisible. *Controls:* Approach B confines it to the PANW path; `--topspeed` dropped; assert sent-vs-seen packet counts and fail on mismatch; diff repeat runs. +2. **Trust-by-construction is unfalsifiable.** With per-source tiering and no validation corpus, the trustworthiness claim rests entirely on curation and cannot be measured — if a consumer asks for a false-positive rate, there is no answer. *Control:* snapshot rulesets per run so labels are at least reproducible and auditable. Flagged for eng-review. +3. **Correlating PANW detections back to capture flows.** Replay-time stamps, time compression, and port reuse within a capture. *Control:* tuple-driven matching with time only scoping the query; define unmatchable-detection behaviour at PRD. +4. **Fingerprint verdicts drift as software populations converge (B5).** A fingerprint can become shared by benign software, turning a valid verdict silently wrong. *Control:* source fingerprint verdicts through ET rules rather than raw feeds, so ET's revisions carry the aging burden; track ruleset snapshot dates. +5. **JA4+ licensing exposure.** FoxIO License 1.1 forbids monetization; flabel feeds product models. *Control:* Legal engaged; plain JA4 is BSD and unrestricted if a fallback is needed. --- ## Open questions -1. **Does PANW tap mode produce the same detections as a virtual-wire pair?** Your brief prefers tap; Palo Alto's replay guidance says vwire. Needs an on-device A/B with an identical capture. If they differ, tap may silently forfeit a class of detections. -2. **Should flabel emit labels at all when the lab is unavailable** — i.e. is a Tier 2/3-only run a valid, clearly-marked output, or an error? Affects whether offline mode is a first-class feature. -3. **What is the required clock accuracy** between flabel and the firewall, and does the threat log need a settling delay before querying? Empirical, not documented. -4. **Are untagged ET Open rules (the <30% without `confidence` metadata) excluded or manually reviewed?** Fail-closed loses coverage; reviewing them is ongoing manual work. -5. **Is `scwx/malware` worth pricing?** It is the only ruleset explicitly marketed as high-fidelity/low-FP, and your bar is unusually demanding. You said free-with-30-day-delay is acceptable; this asks whether that's a preference or a hard constraint. +Resolved items removed. Remaining: + +1. **What is the exact admitted-rule count per source** once the A3 per-source filter is applied? Deferred to build-time measurement with `suricata-update`; needed to know whether Tier 2 coverage is adequate. +2. **Does PANW's threat log need a settling delay** before the API query returns all records for a completed replay, and what is the exact bounded-`receive_time` filter syntax? On-device verification. +3. **What is the behaviour for a detection that cannot be matched to a capture flow** — drop it, or emit it unmatched with a flag? PRD decision. +4. **Are `scwx/malware` or `stamus/nrd-*` worth pursuing** given they are quote-only? Requires contacting sales; ET Pro at ~$900/sensor/year is the only known figure. +5. **Does `pawpatrules` pass an FP review?** Admitted wholesale on a share-alike licence with broad scope; it is the least-vetted of the admitted sources. --- @@ -240,22 +279,24 @@ Bounding the PANW log query by the replay window requires that flabel's clock an - [A Comparative Analysis of Snort 3 and Suricata (Univ. of Portsmouth)](https://pure.port.ac.uk/ws/portalfiles/portal/79753845/A_Comparative_Analysis_of_Snort_3_and_Suricata.pdf) - [Emerging Threats Updates Improve Metadata, Including MITRE ATT&CK Tags — Proofpoint](https://www.proofpoint.com/us/blog/threat-insight/emerging-threats-updates-improve-metadata-including-mitre-attck-tags) - [Signature Metadata — Emerging Threats wiki](https://community.emergingthreats.net/t/signature-metadata/96) +- [ET Open rule index (suricata-7.0)](https://rules.emergingthreats.net/open/suricata-7.0/rules/) +- [ET Open `emerging-ja3.rules`](https://rules.emergingthreats.net/open/suricata-7.0/rules/emerging-ja3.rules) - [OISF suricata-intel-index (rule source licences)](https://github.com/OISF/suricata-intel-index/blob/master/index.yaml) +- [JA3/JA4 Keywords — Suricata docs](https://docs.suricata.io/en/latest/rules/ja-keywords.html) - [What are the differences in the rule sets? — Snort FAQ](https://www.snort.org/faq/what-are-the-differences-in-the-rule-sets) - [Soft Release: lightSPD, the new rules package for Snort 3](https://blog.snort.org/2020/12/soft-release-lightspd-new-rules-package.html) +- [ET Pro Ruleset — Proofpoint](https://www.proofpoint.com/us/resources/data-sheets/et-pro-ruleset) +- [Proofpoint ET Pro Ruleset 1yr subscription — OPNsense shop](https://shop.opnsense.com/product/proofpoint-et-pro-ruleset-1yr-subscription/) - [SSLBL Blacklist — abuse.ch](https://sslbl.abuse.ch/blacklist/) - [SSLBL Malicious JA3 Fingerprints — abuse.ch](https://sslbl.abuse.ch/ja3-fingerprints/) - [The Limits of JA3 Fingerprinting — Fingerprint.com](https://fingerprint.com/blog/limitations-ja3-fingerprinting-accurate-device-identification/) -- [JA3 vs JA4: TLS Fingerprinting for Bot Detection in 2026 — VoidMob](https://voidmob.com/blog/ja3-vs-ja4-tls-fingerprinting-bot-detection-2026) - [FoxIO-LLC/ja4 — README and licensing](https://github.com/FoxIO-LLC/ja4/blob/main/README.md) - [FoxIO License FAQ](https://github.com/FoxIO-LLC/ja4/blob/main/License%20FAQ.md) - [How to Use JA4 Network Fingerprints in Zeek (Jan 2026)](https://zeek.org/2026/01/how-to-use-ja4-network-fingerprints-in-zeek/) - [JA4+ Zeek package](https://packages.zeek.org/packages/view/65d88958-d5f0-11ee-8674-0a598146b5c6) - [Tap Interfaces — PAN-OS docs](https://docs.paloaltonetworks.com/pan-os/11-0/pan-os-networking-admin/configure-interfaces/tap-interfaces) -- [How to Configure a Palo Alto Networks Device for Tap Mode](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClMzCAK) - [Retrieve Logs — PAN-OS XML API](https://docs.paloaltonetworks.com/ngfw/api/pan-os-xml-api-request-types-and-actions/retrieve-logs) - [Replay pcap — Palo Alto LIVEcommunity](https://live.paloaltonetworks.com/t5/general-topics/replay-pcap/td-p/36261) - [tcpreplay man page](https://tcpreplay.appneta.com/wiki/tcpreplay-man.html) - [Suricata Feature #432: PCAP-NG support](https://redmine.openinfosecfoundation.org/issues/432) - [Analysing PCAPNG files with Zeek — Zeek community](https://community.zeek.org/t/analysing-pcapng-files-from-wireshark-traffic-captured-with-zeek-or-spicy/6959) -- [HowTo handle PcapNG files — Netresec](https://www.netresec.com/?page=Blog&month=2012-12&post=HowTo-handle-PcapNG-files) From f774e2961a92d559004693fac8036cb7fdb61df4 Mon Sep 17 00:00:00 2001 From: Craig yourname Date: Tue, 11 Aug 2026 15:31:03 -0700 Subject: [PATCH 4/6] =?UTF-8?q?stage(prd):=20complete=20=E2=80=94=20docs/p?= =?UTF-8?q?rd.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRD written to the standard template, published as a Google Doc, source of truth kept as markdown in the repo. Settles what research deferred — the labels.json schema: - one entry per flow, sources[] retaining every asserting detection - max_tier for trust filtering without walking the array - Zeek uid as the authoritative join key, tuple retained for PANW correlation and readability - malicious flows only; states explicitly that an unlabelled flow is unlabelled, NOT verified-benign - schema version on the file Edge-case behaviour decided: - uncorrelatable detections emitted in unmatched_detections[] with a reason, never dropped or guessed into a flow - timestamped run directories with a 'latest' pointer, so a re-run preserves prior labels for cross-snapshot comparison (changes the output layout the original brief specified) - truncated input processed and stamped partial; unreadable header fails hard - zero detections is a successful run with an empty labels array Goals are framed as verifiable under trust-by-construction — provenance completeness, reproducibility, zero silent-loss paths — rather than a false-positive rate this design cannot measure. 13 user stories (P0-P2) across two personas, acceptance criteria per story, 16 test cases, 9 risks, 10 open questions. Closes #2 --- docs/prd.md | 417 +++++++++++++++++++++++++++++++++++++++++++++++ docs/status.yaml | 7 +- 2 files changed, 421 insertions(+), 3 deletions(-) create mode 100644 docs/prd.md diff --git a/docs/prd.md b/docs/prd.md new file mode 100644 index 0000000..79d0875 --- /dev/null +++ b/docs/prd.md @@ -0,0 +1,417 @@ +# Product Requirements Document + +| | | +| :-: | :-: | +| **Document Title** | flabel — Malicious Flow Labeling for Packet Captures | +| **Author** | Craig | +| **Last Updated** | 2026-08-11 | +| **Status** | Draft | +| **Stakeholders** | TBD — Craig (author/PM). Legal required as approver for the FoxIO License 1.1 / JA4+ question. Remaining reviewers and approvers to be named. | +| **Target Release** | TBD | + +## Revision History + +| | | | | +| :-: | :-: | :-: | :-: | +| **Date** | **Author** | **Version** | **Change Summary** | +| 2026-08-11 | Craig | 0.1 | Initial draft | + +## 1. Problem Statement + +Training a network detection model requires labeled flow data: examples of traffic known to be malicious, tied to the flows that carried them. Packet captures are abundant and cheap to collect; **trustworthy labels for them are neither.** + +The options available today are each inadequate for the purpose: + +- **Manual analyst labeling** produces high-quality verdicts but does not scale to the volume a model needs. +- **Public labeled datasets** are fixed corpora — they cannot label *our* captures, and their labeling methodology is often undocumented or dated. +- **Running an IDS and taking its alerts at face value** scales, but silently inherits every false positive in whatever ruleset happened to be loaded. For training data this is the expensive failure: a false positive teaches the model that benign traffic is malicious, and the error is invisible downstream. + +The people affected are the engineers building DeepTempo's detection models, who currently have no repeatable way to turn an arbitrary capture into labels they can defend. The cost of inaction is models trained on ground truth that is either too small to be useful or too noisy to be trusted — and, critically, **no way to tell which**, because label provenance is not recorded. + +`flabel` addresses this by processing a capture through detection sources of *known, documented* trust and emitting labels that each carry their origin — the source, the exact rule, and the ruleset snapshot that produced them. + +> **Evidence note:** this statement is grounded in the project brief (`docs/prep-n-research.md`) and the Stage 1 research findings, not in customer interviews or support data. Stakeholders should treat the problem framing as internally asserted rather than externally validated. + +## 2. Goals + +| | Goal | How it is verified | +| :-: | :-- | :-- | +| **Goal 1** | **Every label is traceable to its origin.** A label records its tier, source, the specific rule or signature that fired, and the ruleset snapshot in use. | 100% of emitted labels carry a complete provenance block. Machine-checkable on the output. | +| **Goal 2** | **Runs are reproducible.** The same capture processed with the same ruleset snapshot in `--offline` mode produces identical labels. | Byte-identical `labels.json` across two runs, ignoring run-metadata timestamps. | +| **Goal 3** | **Nothing is lost silently.** Truncated input, packet-count mismatch during replay, and detections that cannot be correlated to a flow are all surfaced in the output. | Every such condition has a corresponding field in the `run` block; zero silent-drop paths. | +| **Goal 4** | **One label per flow.** Detections from multiple tiers against the same flow consolidate into a single entry retaining all asserting sources. | No duplicate flow identity in `labels`. | +| **Goal 5** | **Usable without the lab.** A Tier 2-only run is available and clearly marked as lacking Tier 1 coverage. | `--offline` completes successfully with no NGFW reachable and stamps reduced coverage in the output. | + +**Explicitly not a goal:** a measured false-positive rate. Per the trust-by-construction decision (Stage 1), label quality is argued from source provenance and ruleset curation, not measured against a ground-truth corpus. See Risks. + +## 3. Non-Goals + +- **Not a real-time IDS or IPS.** flabel processes captures after the fact. It never blocks, resets, or otherwise acts on live traffic. +- **Not a TLS decryption or MITM tool.** Encrypted traffic is handled by fingerprint- and certificate-based detection only; decryption is impossible on an after-the-fact capture. +- **Not an assertion of benign.** flabel labels malicious flows. It does not and cannot certify that any flow is safe. +- **Not a general pcap forensics or analysis platform.** Zeek logs are emitted because they are needed for correlation and enrichment, not as a product surface. +- **Not a false-positive measurement system.** flabel does not estimate its own accuracy. +- **No graphical interface.** Command-line only. +- **Not a ruleset authoring tool.** flabel consumes third-party rulesets; it does not write detection rules. + +## 4. Out of Scope + +| | | | +| :-: | :-: | :-: | +| **Item** | **Reason** | **Future Phase?** | +| Label validation against a ground-truth corpus | Trust-by-construction decided at Stage 1; trustworthiness argued from provenance | TBD — flagged for eng-review | +| JA4 as a labeling source | ET Open ships no `ja4.hash` rules yet; no free maintained JA4 verdict feed exists. JA4 is computed as enrichment only | Yes — promotes into Tier 2 with no architectural change once ET publishes JA4 rules | +| Snort 3 as the Tier 2 engine | Suricata selected on free high-confidence ruleset volume and native fingerprint keywords | No | +| FortiGate as the NGFW | PANW VM-Series selected as Tier 1 | No | +| Free/OSS L7 equivalent to PANW App-ID | No free equivalent exists; line of inquiry closed | No | +| Paid rulesets (ET Pro ~$900/sensor/yr, Secureworks, Stamus) | Free sources with a 30-day delay are acceptable for v1 | TBD — revisit if free-source coverage proves inadequate | +| Government-published rule feeds | No agency publishes a maintained general-purpose ruleset; advisory signatures are point-in-time IOCs with no cadence | No | +| Positive Technologies rulesets | Non-standard licence and vendor under US sanctions since 2021 | No | +| PANW tap-mode deployment | Virtual-wire pair selected; avoids the unresolved question of whether tap forfeits detections | No | +| Benign / negative-class labels | flabel cannot assert benign; absence of a label is not a verdict | No | +| `abuse.ch` SSLBL JA3 fingerprint feed | Abandoned — newest entry 2021-08-03, self-declared untested FP posture | No | + +## 5. Background & Context + +**Prior work.** Stage 1 research (`docs/research.md`, driven by `docs/research-brief.md`) evaluated detection sources, rulesets, fingerprinting methods, and replay mechanics. The original design brief is `docs/prep-n-research.md`. Three research findings changed the design as originally conceived: + +1. **Only PANW requires replay.** Suricata and Zeek read capture files natively. The original design sent all traffic through the inline device; confining replay to the one source that needs it removes the largest source of label nondeterminism from the Tier 2 path. +2. **The canonical free JA3 feed is abandoned**, but ET Open independently maintains its own JA3 ruleset — actively updated through 2026, carrying per-rule confidence metadata, MIT-licensed. Fingerprint detection therefore belongs in Tier 2 as ordinary Suricata rule content, matched via Suricata's native `ja3.hash` / `ja4.hash` keywords. +3. **pcapng cannot be fed directly to Zeek at all**, and only partially to Suricata. Since pcapng is Wireshark's default output, an input normalization stage is mandatory rather than optional. + +**Trust model.** Trust is assigned **per source**, not per rule. Tier 1 is PANW VM-Series; Tier 2 is Suricata with a curated, admission-filtered ruleset set. Because trust is not modeled per rule, **ruleset curation is the entire false-positive defence** — which is why the admission filter and its per-run snapshot are product requirements, not implementation details. + +**Environment.** A GCP lab is a v1 requirement: a host running flabel, a PANW VM-Series in virtual-wire configuration, and a Suricata host, with millisecond clock synchronization across all three. + +## 6. Feature Description + +### 6.1 Capture Ingest & Normalization + +**Description:** Accepts a capture file, determines its true format, and normalizes it to a single artifact that every downstream consumer reads identically. + +**Key Business Rules / Logic:** + +- Accepted inputs: `pcap`, `pcapng`, and gzip-compressed variants of both. +- pcapng is normalized to pcap via `editcap -F pcap`. Multi-datalink captures are split before conversion. +- All three consumers (Zeek, Suricata, replay) receive the **same normalized file**, so they cannot disagree about the input. +- A capture whose header is unreadable is a **hard failure** — no labels are emitted. +- A capture that is **truncated** (readable prefix, incomplete tail) is processed, and the output is stamped `input_status: partial` with the packet count reached and the truncation offset. +- The normalization performed is recorded in run provenance, because a converted capture is not the original artifact. + +### 6.2 Zeek Processing & Fingerprint Enrichment + +**Description:** Runs Zeek over the normalized capture to produce flow logs and TLS fingerprints. Zeek's `uid` is the authoritative flow identity for the entire system. + +**Key Business Rules / Logic:** + +- All Zeek logs generated for the capture are retained in the output. +- JA4 (and JA4+, subject to the licence decision) is computed for every TLS connection via the `zeek/foxio/ja4` package. +- **Fingerprints are recorded as attributes, never as verdicts.** A fingerprint value alone never produces a label. +- Zeek `uid` is assigned to every flow and becomes the join key between `labels.json` and the Zeek logs. + +### 6.3 Tier 2 Detection — Suricata + +**Description:** Runs Suricata offline against the normalized capture using a curated, admission-filtered ruleset, producing Tier 2 detections. This path requires no lab and is fully deterministic. + +**Key Business Rules / Logic:** + +- **Per-source admission policy.** Signature rulesets (ET Open) are filtered on rule metadata: `confidence == High` **and** `signature_severity in (Major, Critical)`. IOC feeds (abuse.ch, malsilo, and similar) carry no such metadata and are admitted wholesale, with the feed snapshot date as their provenance. +- Rules lacking a `confidence` tag are **excluded** (fail-closed). +- Excluded sources: hunting/anomaly rulesets, self-described aggressive blacklists, and Positive Technologies. +- Encrypted-traffic detection is part of this tier, via ET Open's JA3 rules matched with Suricata's native fingerprint keywords. +- **The admitted rule set is snapshotted per run** — source, version, and date — because filter output changes as vendors revise metadata. Without this, two runs of "the same" flabel are not comparable. + +### 6.4 Tier 1 Detection — PANW VM-Series + +**Description:** Replays the normalized capture past a PANW VM-Series in virtual-wire configuration, then retrieves the resulting threat detections. + +**Key Business Rules / Logic:** + +- Replay runs at a **controlled rate**. `--topspeed` is prohibited: it trades timing accuracy for speed, and distorted timing can suppress stateful and rate-based detections. +- **Packets sent are reconciled against packets seen by the device.** A mismatch is a fidelity failure and is surfaced, because a dropped packet is a *missing* label — invisible unless reported. +- Threat logs are retrieved via the PAN-OS API, with the query **bounded by the replay window** and results **matched by flow tuple**. Time scopes the query; the tuple performs the match. +- Detections are stamped at replay time, not capture time. Labels reference the **capture's** timeline; replay time is never emitted as a label timestamp. +- Requires millisecond clock synchronization between flabel and the firewall. + +### 6.5 Consolidation & Correlation + +**Description:** Merges Tier 1 and Tier 2 detections into one label per flow, resolving each to a Zeek flow identity. + +**Key Business Rules / Logic:** + +- **One entry per flow.** A flow flagged by multiple tiers yields a single entry whose `sources[]` array retains every asserting detection. +- `max_tier` records the highest-trust source that asserted the flow, so consumers can filter by trust without walking the array. +- A detection that **cannot be correlated** to any flow is emitted in a separate `unmatched_detections[]` block with a reason and the raw device fields. It is never silently dropped and never guessed into a flow. +- The count of unmatched detections is surfaced in run metadata as a correlation-health signal. + +### 6.6 Output & Provenance + +**Description:** Writes Zeek logs and `labels.json` to a timestamped run directory, with complete provenance for the run. + +**Key Business Rules / Logic:** + +- Output layout — **each run gets its own directory**, so a re-run never destroys prior labels: + +``` +{input-pcap-name}/ +├── 2026-08-11T213045Z/ +│ ├── zeek/ # all Zeek logs +│ └── labels.json +└── latest -> 2026-08-11T213045Z/ +``` + +- `latest` is a stable pointer for scripted consumers. +- `labels.json` contains **malicious flows only.** An unlabeled flow is *unlabeled*, not verified benign — this distinction is stated in the output schema itself so it cannot be lost downstream. +- A capture with zero detections is a **successful run** producing an empty `labels` array, not an error. +- Every run records: input file identity and status, normalization applied, ruleset snapshots per source, tool versions, tiers attempted, and coverage actually achieved. +- `labels.json` carries a **schema version**, so consumers can detect shape changes rather than silently mis-parsing. + +**Label entry shape:** + +```json +{ + "flow": { + "uid": "CHhAvVGS1DHFjwGM9", + "src_ip": "10.0.0.5", "src_port": 49152, + "dst_ip": "203.0.113.10", "dst_port": 443, + "proto": "tcp", + "ts_first": "...", "ts_last": "..." + }, + "verdict": "malicious", + "max_tier": 1, + "sources": [ + { "tier": 1, "source": "panw", "threat": "...", "detected_at": "..." }, + { "tier": 2, "source": "suricata", "sid": 2028831, "rev": 1, + "ruleset": "et-open@2026-08-11", "confidence": "High", "threat": "..." } + ] +} +``` + +### 6.7 CLI & Run Modes + +**Description:** A single command processes a capture end to end. + +**Key Business Rules / Logic:** + +- `flabel ` is the primary invocation. Tier 1 (NGFW) is **required by default**; if the lab is unreachable, the run fails rather than silently producing partial coverage. +- `--offline` runs Tier 2 only, completing without any lab, and **stamps the output as lacking Tier 1 coverage.** +- Exit codes distinguish success, partial-input success, and failure, so the tool is usable in a pipeline. +- Progress and warnings go to stderr; machine-readable output goes to files, never stdout-mixed. + +## 7. User Stories + +Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labels). **OPS** = lab/platform operator (provisions and maintains the environment). + +| | | | | +| :-: | :-: | :-: | :-: | +| **ID** | **Priority** | **User Story** | **Notes** | +| US-01 | P0 | As a DME, I want to run one command against a capture and get Zeek logs plus a malicious-flow label file, so that I can turn raw captures into training data without manual analysis. | Core capability | +| US-02 | P0 | As a DME, I want every label to record the source, rule, and ruleset snapshot that produced it, so that I can defend or audit any individual label later. | Goal 1 | +| US-03 | P0 | As a DME, I want each label to join directly to the Zeek flow record, so that I can extract features for the labeled flow. | Zeek `uid` | +| US-04 | P0 | As a DME, I want a flow flagged by several sources to appear once with all sources listed, so that I don't have to deduplicate before training. | One entry per flow | +| US-05 | P0 | As a DME, I want to know when a run's coverage was incomplete — truncated input, dropped packets, uncorrelated detections — so that I don't train on a label set I believe to be complete when it isn't. | Goal 3 | +| US-06 | P0 | As a DME, I want re-running a capture to preserve the previous run's output, so that I can compare label sets across ruleset snapshots. | Timestamped run dirs | +| US-07 | P0 | As an OPS, I want the same capture and ruleset snapshot to yield identical labels, so that I can verify the pipeline is behaving deterministically. | Goal 2; `--offline` | +| US-08 | P1 | As an OPS, I want an `--offline` mode that runs without the NGFW, so that I can process captures and test the pipeline when the lab is unavailable. | Marked reduced coverage | +| US-09 | P1 | As a DME, I want pcapng and gzipped captures accepted directly, so that I can use files as they come off Wireshark or a sensor without pre-processing. | Normalization | +| US-10 | P1 | As a DME, I want JA4 fingerprints recorded on TLS flows, so that I can use them as model features and pivot on them during analysis. | Enrichment, not verdicts | +| US-11 | P1 | As an OPS, I want the ruleset admission filter and its results recorded per run, so that I can see exactly which rules were live and how many were excluded. | Ties to issue #11 | +| US-12 | P2 | As an OPS, I want a documented environment diagram in draw.io and mermaid form, so that the lab can be rebuilt or handed over. | From the brief | +| US-13 | P2 | As a DME, I want `labels.json` to carry a schema version, so that a shape change breaks loudly rather than silently. | Forward compatibility | + +## 8. UX Requirements + +**Key Workflows:** + +- **Label a capture (default).** `flabel capture.pcap` → normalize → Zeek + Suricata + PANW replay → consolidate → write run directory. Fails clearly if the lab is unreachable. +- **Label without the lab.** `flabel --offline capture.pcap` → Tier 2 only → output stamped as reduced coverage. +- **Compare across ruleset snapshots.** Re-run the same capture; each run lands in its own timestamped directory; `latest` points to the newest. +- **Inspect a label.** Read a label's `sources[]`, then join its `flow.uid` to `zeek/conn.log` for the full flow record. + +**Design Constraints / Guidelines:** + +- Command-line only; no GUI, no daemon, no service. +- One capture per invocation. Batch processing is the caller's job (shell loop, pipeline). +- Human-readable progress and warnings to **stderr**; all machine-consumable output to files. Never interleave the two. +- Failures state what went wrong, which stage it happened in, and what would fix it. +- Silence is never used to signal a problem. Absent coverage is always explicitly reported. + +**Accessibility Requirements:** + +- No information conveyed by colour alone; any colour is decorative and the text stands without it. +- Output remains fully legible when redirected to a file or read by a screen reader (no cursor-control or spinner-dependent rendering). +- Distinct, documented exit codes: success, success-with-partial-input, failure. +- All output UTF-8. + +**Prototype / Mockup Links:** + +- N/A — command-line tool, no visual design surface. + +## 9. Acceptance Criteria + +### US-01: Label a capture end to end + +- Given a valid pcap and a reachable lab, when `flabel capture.pcap` runs, then a timestamped run directory is created containing `zeek/` with the Zeek logs and `labels.json`. +- Given a capture containing no detectable threats, when the run completes, then `labels.json` exists with an empty `labels` array and the run is reported as successful. +- Given the NGFW is unreachable and `--offline` was not passed, when flabel runs, then it fails with a message naming the unreachable Tier 1 dependency, and no partial label file is written. + +### US-02: Label provenance + +- Given any emitted label, when its `sources[]` entries are inspected, then each records tier, source, the firing rule identity where applicable, and the ruleset snapshot identifier. +- Given a completed run, when the run metadata is inspected, then it lists every ruleset source with its version and snapshot date, plus the versions of Zeek, Suricata, and flabel. + +### US-03: Join to Zeek + +- Given a label with `flow.uid`, when that uid is looked up in `zeek/conn.log`, then exactly one matching flow record exists. +- Given a Tier 1 detection correlated to a flow, when its label is written, then the label carries the Zeek `uid` for that flow, not only the tuple. + +### US-04: One entry per flow + +- Given a flow flagged by both Tier 1 and Tier 2, when labels are written, then exactly one entry exists for that flow, its `sources[]` contains both detections, and `max_tier` is 1. +- Given the full `labels` array, when flow identities are compared, then no flow identity appears more than once. + +### US-05: Incomplete coverage is visible + +- Given a truncated capture, when the run completes, then `input_status` is `partial` and the packet count reached is recorded. +- Given a replay where packets seen by the device differ from packets sent, when the run completes, then the discrepancy is reported and the run is not presented as full-coverage. +- Given a detection that cannot be correlated to any flow, when labels are written, then it appears in `unmatched_detections[]` with a reason, and the unmatched count appears in run metadata. +- Given a capture whose header is unreadable, when flabel runs, then it fails and writes no labels. + +### US-06: Re-runs preserve history + +- Given a capture already processed, when flabel runs against it again, then a new timestamped directory is created, the previous run directory is unmodified, and `latest` points to the new run. + +### US-07: Reproducibility + +- Given the same capture, the same ruleset snapshot, and `--offline`, when flabel runs twice, then the two `labels.json` files are identical apart from run-metadata timestamps. + +### US-08: Offline mode + +- Given no reachable lab, when `flabel --offline capture.pcap` runs, then it completes successfully, emits Tier 2 labels, and the output records that Tier 1 was not attempted. +- Given an `--offline` run, when the output is inspected, then no label claims a Tier 1 source. + +### US-09: Format handling + +- Given a pcapng capture, when flabel runs, then it is normalized to pcap, processed successfully, and the conversion is recorded in provenance. +- Given a gzipped capture, when flabel runs, then it is decompressed and processed. +- Given a pcapng capture containing multiple link-layer types, when flabel runs, then it is split before conversion and processed without error. +- Given any accepted input, when the run completes, then Zeek, Suricata, and the replay all processed the identical normalized file. + +### US-10: Fingerprint enrichment + +- Given a capture containing TLS connections, when the run completes, then JA4 values are present on those flows in the Zeek output. +- Given a JA4 value matching no rule, when labels are written, then no label is produced from the fingerprint alone. + +## 10. Technical Considerations + +**Architecture / System Design Notes:** + +- **Hybrid ingest (Approach B).** Zeek and Suricata read the capture file directly; only PANW receives a replay. This confines replay-fidelity and clock-correlation risk to the Tier 1 path and makes Tier 2 deterministic — and makes `--offline` a natural consequence of the architecture rather than a special case. +- PANW deployed as a **virtual-wire pair**, per Palo Alto's replay guidance, not tap mode. +- Correlation is **tuple-driven**, with the replay window only scoping the API query. Replay-time stamps are never emitted as label timestamps. +- Fingerprint matching uses Suricata's native `ja3.hash` / `ja4.hash` keywords, so encrypted detection needs no separate engine and inherits Tier 2's filtering and provenance. +- Stack: Python 3.12, uv, pytest, ruff. Test-first development. + +**Dependencies (internal and external):** + +- **External tools:** Zeek 6+ (with `zeek/foxio/ja4`), Suricata 8+ (with `ja3`/`ja4` fingerprinting enabled), tcpreplay, Wireshark `editcap`. +- **External services:** PAN-OS API on the VM-Series; ruleset and IOC feed endpoints (ET Open, abuse.ch, and the other admitted sources). +- **Infrastructure:** GCP lab — flabel host, PANW VM-Series, Suricata host; NTP with millisecond accuracy across all three. **The lab is a v1 requirement.** +- **Licensing:** plain JA4 is BSD 3-Clause. The **JA4+ suite is FoxIO License 1.1 (non-commercial)** — approved for use pending Legal's assessment, since flabel output feeds product models. This is a blocking external dependency on Legal, not an engineering task. + +**Data & Privacy Considerations:** + +- **Captures contain real network traffic** and may include personal data, credentials in cleartext protocols, internal addressing, and business-sensitive content. Zeek logs derived from them include URLs, DNS queries, and certificate details. +- The repository is **public**. Captures, Zeek logs, `labels.json` outputs, device credentials, and internal identifiers must never be committed. `.gitignore` already excludes `*.pcap`, `*.pcapng`, `*.log`, `zeek/`, and `.env`. +- Test fixtures must not embed real capture data. A fixture strategy is required (synthetic or explicitly-licensed public captures) — tracked as part of scaffold. +- Device credentials and the GCP project identifier live in a gitignored `.env`; committed files reference them only as `${VAR}`. +- Labels and Zeek logs are derived data and inherit the sensitivity of their source capture. Retention and handling are the operator's responsibility; flabel does not transmit capture data anywhere outside the lab. + +**Performance / Scale Requirements:** + +- Tier 1 throughput is bounded by the controlled replay rate, deliberately: fidelity is preferred over speed. Wall-clock for a run is therefore at least the replay duration. +- Tier 2 and Zeek run at file-read speed and are the fast path; `--offline` runs are substantially quicker than full runs. +- One capture per invocation; no concurrency requirement within a run. +- **Concurrent runs against one device are unsafe** without coordination, since overlapping replay windows would make Tier 1 log queries ambiguous. Serialization is required — mechanism TBD at spec. +- No target maximum capture size is set for v1; behaviour on very large captures is an open question. + +## 11. Success Metrics + +| | | | | +| :-: | :-: | :-: | :-: | +| **Metric** | **Target** | **How Measured** | **Review Date** | +| Provenance completeness | 100% of labels carry tier, source, rule identity (where applicable), and ruleset snapshot | Automated check over `labels.json` in CI and per run | TBD | +| Reproducibility | 100% — identical `labels.json` across two `--offline` runs with a pinned snapshot | Regression test diffing two runs, ignoring run timestamps | TBD | +| Silent-loss paths | Zero | Code review plus tests asserting every loss condition surfaces in the `run` block | TBD | +| Replay integrity | Packets sent equal packets seen, or the discrepancy is reported | Per-run reconciliation; failure surfaced, never suppressed | TBD | +| Unmatched detection rate | Tracked and reported; no target set for v1 | Per-run `unmatched_count`; trend reviewed as capture volume grows | TBD | +| Supported-format success rate | 100% of pcap, pcapng, and gzipped inputs process or fail with a clear reason | Test matrix across format variants | TBD | +| Tier 2 admitted-rule count | Recorded per source; adequacy assessed once known | Measured at build time (issue #11) | TBD | + +**Note:** these metrics measure *pipeline integrity*, not *label accuracy*. Under trust-by-construction there is no measurement of false-positive rate — see Risks. + +## 12. Risks & Mitigations + +| | | | | +| :-: | :-: | :-: | :-: | +| **Risk** | **Likelihood** | **Impact** | **Mitigation** | +| Replay infidelity silently drops labels, and a missing label is invisible in the output | Med | High | `--topspeed` prohibited; controlled-rate replay; sent-vs-seen packet reconciliation surfaced per run; repeat-run detection diffing; Tier 2 kept off the replay path entirely | +| Trust-by-construction is unfalsifiable — no false-positive rate can be quoted if a label consumer asks | High | High | Ruleset snapshots recorded per run so labels are reproducible and auditable even if unmeasured; fail-closed admission filter; flagged for eng-review reconsideration | +| Tier 1 detections cannot be correlated to capture flows (port reuse, NAT, tunnelling) | Med | Med | Tuple-driven matching with time only scoping the query; uncorrelated detections emitted in `unmatched_detections[]` rather than dropped or guessed | +| Fingerprint verdicts drift as benign software adopts a fingerprint, silently invalidating a verdict | Med | Med | Fingerprint verdicts sourced through ET rules rather than raw feeds, so vendor revisions carry the aging burden; ruleset snapshot dates recorded | +| Admission filter proves too strict, leaving Tier 2 coverage too thin to be useful | Med | Med | Admitted-rule counts measured per source (issue #11); untagged-rule policy revisitable (issue #10); paid high-fidelity rulesets remain a costed fallback | +| JA4+ licensing (FoxIO 1.1, non-commercial) conflicts with output feeding product models | Med | High | Legal engaged as an approver; plain JA4 is BSD 3-Clause and available as an unrestricted fallback | +| Capture data or credentials leak into the public repository | Low | High | `.gitignore` coverage for captures, logs, and `.env`; no real capture data in fixtures; pre-commit secret checks | +| Lab environment build slips and blocks all Tier 1 work | Med | Med | `--offline` mode delivers Tier 2 labels with no lab; environment build tracked as explicit plan and build steps | +| Clock drift beyond millisecond breaks Tier 1 log correlation | Low | Med | NTP across all hosts; correlation designed to survive drift by matching on tuple rather than time | + +## 13. Open Questions + +| | | | | | +| :-: | :-: | :-: | :-: | :-: | +| **\#** | **Question** | **Owner** | **Target Date** | **Resolution** | +| 1 | Who are the PRD stakeholders — reviewers and approvers beyond Craig and Legal? | Craig | TBD | Open | +| 2 | What is the target release for v1? | Craig | TBD | Open | +| 3 | Does Legal approve JA4+ under FoxIO License 1.1 given output feeds product models, or do we restrict to plain JA4? | Legal | TBD | Open — blocking 6.2 scope | +| 4 | What is the admitted-rule count per source once the admission filter is applied, and is Tier 2 coverage adequate? | Craig | At build | Open — issue #11 | +| 5 | How are concurrent runs against a single PANW device serialized, given overlapping replay windows make log queries ambiguous? | TBD | At spec | Open | +| 6 | Is there a maximum supported capture size, and what is the behaviour beyond it? | Craig | At spec | Open | +| 7 | Does the PANW threat log need a settling delay before querying, and what is the exact bounded-`receive_time` filter syntax? | TBD | At spec | Open — on-device verification | +| 8 | What fixture strategy provides test captures without committing real traffic? | TBD | At scaffold | Open | +| 9 | Should `pawpatrules` remain admitted without an FP review, being the least-vetted admitted source? | Craig | At spec | Open | +| 10 | What are the review dates for the success metrics? | Craig | TBD | Open | + +15. Basic Test Cases + +| | | | | | +| :-: | :-: | :-: | :-: | :-: | +| **\#** | **Case** | **Expected Behavior** | **Observed Behavior** | **Pass/Fail** | +| 1 | Valid pcap, lab reachable | Run directory created with `zeek/` and `labels.json`; `latest` points to it | | | +| 2 | Capture with no detections | Success; `labels` is an empty array | | | +| 3 | pcapng input | Normalized to pcap, processed, conversion recorded in provenance | | | +| 4 | Gzipped pcap input | Decompressed and processed | | | +| 5 | pcapng with multiple link-layer types | Split before conversion; processed without error | | | +| 6 | Capture with unreadable header | Hard failure; no labels written | | | +| 7 | Truncated capture | Processed; `input_status: partial` with packet count reached | | | +| 8 | Flow flagged by both Tier 1 and Tier 2 | Single entry; both entries in `sources[]`; `max_tier` = 1 | | | +| 9 | Detection uncorrelatable to any flow | Appears in `unmatched_detections[]` with reason; counted in run metadata | | | +| 10 | Re-run of an already-processed capture | New timestamped directory; prior run untouched; `latest` updated | | | +| 11 | Two `--offline` runs, pinned snapshot | `labels.json` identical apart from run timestamps | | | +| 12 | `--offline` with no lab reachable | Succeeds; Tier 2 labels only; output records Tier 1 not attempted | | | +| 13 | Default run with lab unreachable | Fails naming the Tier 1 dependency; no partial label file | | | +| 14 | Replay with packet-count mismatch | Discrepancy reported; run not presented as full coverage | | | +| 15 | TLS capture | JA4 present on TLS flows in Zeek output; no label from fingerprint alone | | | +| 16 | Every emitted label | Complete provenance block; `flow.uid` resolves to exactly one `conn.log` record | | | + +## 14. References & Related Documents + +- [`docs/research.md`](research.md) — Stage 1 research findings and decisions +- [`docs/research-brief.md`](research-brief.md) — approved research brief (Stage 1 gate) +- [`docs/prep-n-research.md`](prep-n-research.md) — original design brief +- [`docs/status.yaml`](status.yaml) — pipeline state and stage issue mapping +- [OISF suricata-intel-index](https://github.com/OISF/suricata-intel-index/blob/master/index.yaml) — rule source licences and provenance +- [Signature Metadata — Emerging Threats wiki](https://community.emergingthreats.net/t/signature-metadata/96) — `confidence` / `signature_severity` definitions +- [JA3/JA4 Keywords — Suricata docs](https://docs.suricata.io/en/latest/rules/ja-keywords.html) +- [FoxIO License FAQ](https://github.com/FoxIO-LLC/ja4/blob/main/License%20FAQ.md) — JA4+ commercial-use terms +- [Retrieve Logs — PAN-OS XML API](https://docs.paloaltonetworks.com/ngfw/api/pan-os-xml-api-request-types-and-actions/retrieve-logs) +- GitHub issues [#10](https://github.com/DeepTempo/flabel/issues/10) (untagged ET rules) and [#11](https://github.com/DeepTempo/flabel/issues/11) (admission-filter measurement) diff --git a/docs/status.yaml b/docs/status.yaml index 96d6c1e..bc2e8c7 100644 --- a/docs/status.yaml +++ b/docs/status.yaml @@ -4,14 +4,14 @@ project: "flabel" created: "2026-08-11" github_repo: "DeepTempo/flabel" notion_url: "https://app.notion.com/p/3b92a84a5230812aa64aef638e389725" -prd_gdoc_url: "" # Google Doc PRD (set by /project:prd); "" = not created yet -current_stage: prd +prd_gdoc_url: "https://docs.google.com/document/d/1q_muS6AmCuvcNva4C1WtRE3_KXArJPxnapttsU6UQIE/edit" +current_stage: eng_review # NOTE: issue numbers are not in stage order. Two creates hit a transient TLS # error and were retried, and GitHub never reuses issue numbers. This mapping is # authoritative — trust it over the numeric order in the GitHub issue list. stages: research: { status: completed, issue: 1, artifact: docs/research.md, completed: "2026-08-11" } - prd: { status: pending, issue: 2, artifact: docs/prd.md, completed: null } + prd: { status: completed, issue: 2, artifact: docs/prd.md, completed: "2026-08-11" } eng_review: { status: pending, issue: 6, artifact: docs/eng-review.md, completed: null } plan: { status: pending, issue: 3, artifact: PLAN.md, completed: null } scaffold: { status: pending, issue: 4, artifact: .github/workflows/ci.yml, completed: null } @@ -21,3 +21,4 @@ log: - "2026-08-11 init — repo scaffolded, 7 stage issues filed, Notion row created" - "2026-08-11 init — repo recreated so main is the default branch; old repo renamed to DeepTempo/flabel-old and archived" - "2026-08-11 stage(research) completed — recommends Suricata over Snort, replay only for PANW, and JA4-as-enrichment instead of JA3/JA4 labelling" + - "2026-08-11 stage(prd) completed — labels.json schema settled (one entry per flow, sources[], Zeek uid join, malicious-only); timestamped run dirs; 13 user stories" From 24a379c1ec4fc604f092c95a344188e5bd2d8e04 Mon Sep 17 00:00:00 2001 From: Craig yourname Date: Tue, 11 Aug 2026 15:41:15 -0700 Subject: [PATCH 5/6] stage(prd): sibling run directories, {capture-name}_{datetime} MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Craig's change: put the datetime in the folder name so multiple runs are siblings rather than nested under a per-capture parent. my-capture_2026-08-11T213045Z/ my-capture_2026-08-12T091500Z/ Naming rules specified: extension stripped including a trailing .gz; datetime is UTC ISO-8601 with no colons, so names are filesystem-safe on every platform and a lexicographic sort is also chronological. Drops the 'latest' pointer. With siblings, a {capture-name}_latest symlink would be matched by the same {capture-name}_* glob used to enumerate runs and corrupt iteration. Newest run is the last entry of a sorted list. Updated §6.6, §8 workflows, US-06, US-01/US-06 acceptance criteria, and test cases 1 and 10. Added an acceptance criterion for .pcap.gz naming and one asserting sort order is chronological. Revision history bumped to 0.2. Google Doc regenerated (Drive MCP cannot edit in place): v0.2 at 1WwKukQ71L6... supersedes v0.1 at 1q_muS6Am..., which needs manual deletion. Refs #2 --- docs/prd.md | 37 +++++++++++++++++++++++-------------- docs/status.yaml | 2 +- 2 files changed, 24 insertions(+), 15 deletions(-) diff --git a/docs/prd.md b/docs/prd.md index 79d0875..73bc43c 100644 --- a/docs/prd.md +++ b/docs/prd.md @@ -15,6 +15,7 @@ | :-: | :-: | :-: | :-: | | **Date** | **Author** | **Version** | **Change Summary** | | 2026-08-11 | Craig | 0.1 | Initial draft | +| 2026-08-11 | Craig | 0.2 | Output layout changed to sibling run directories named `{capture-name}_{datetime}`; `latest` pointer removed | ## 1. Problem Statement @@ -146,21 +147,27 @@ The people affected are the engineers building DeepTempo's detection models, who ### 6.6 Output & Provenance -**Description:** Writes Zeek logs and `labels.json` to a timestamped run directory, with complete provenance for the run. +**Description:** Writes Zeek logs and `labels.json` to a per-run directory named after the capture and the run time, with complete provenance for the run. **Key Business Rules / Logic:** -- Output layout — **each run gets its own directory**, so a re-run never destroys prior labels: +- Output layout — **each run writes its own top-level directory named `{capture-name}_{datetime}`**, so runs of the same capture are siblings and a re-run never destroys prior labels: ``` -{input-pcap-name}/ -├── 2026-08-11T213045Z/ -│ ├── zeek/ # all Zeek logs -│ └── labels.json -└── latest -> 2026-08-11T213045Z/ +my-capture_2026-08-11T213045Z/ +├── zeek/ # all Zeek logs +└── labels.json + +my-capture_2026-08-12T091500Z/ +├── zeek/ +└── labels.json ``` -- `latest` is a stable pointer for scripted consumers. +- **Directory naming rules:** + - `{capture-name}` is the input filename with its extension stripped, including a trailing `.gz` (`my-capture.pcap.gz` → `my-capture`). Characters unsafe in a path are replaced. + - `{datetime}` is the run start in UTC, ISO-8601 with no colons — `2026-08-11T213045Z` — so the name is filesystem-safe on every platform. + - Because ISO-8601 sorts lexicographically, a plain sort of `{capture-name}_*` is also a chronological ordering, and the newest run is the last entry. +- **No `latest` pointer.** A `{capture-name}_latest` symlink would be matched by the same `{capture-name}_*` glob used to enumerate runs, so it would corrupt iteration. Consumers needing the newest run sort and take the last. - `labels.json` contains **malicious flows only.** An unlabeled flow is *unlabeled*, not verified benign — this distinction is stated in the output schema itself so it cannot be lost downstream. - A capture with zero detections is a **successful run** producing an empty `labels` array, not an error. - Every run records: input file identity and status, normalization applied, ruleset snapshots per source, tool versions, tiers attempted, and coverage actually achieved. @@ -210,7 +217,7 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | US-03 | P0 | As a DME, I want each label to join directly to the Zeek flow record, so that I can extract features for the labeled flow. | Zeek `uid` | | US-04 | P0 | As a DME, I want a flow flagged by several sources to appear once with all sources listed, so that I don't have to deduplicate before training. | One entry per flow | | US-05 | P0 | As a DME, I want to know when a run's coverage was incomplete — truncated input, dropped packets, uncorrelated detections — so that I don't train on a label set I believe to be complete when it isn't. | Goal 3 | -| US-06 | P0 | As a DME, I want re-running a capture to preserve the previous run's output, so that I can compare label sets across ruleset snapshots. | Timestamped run dirs | +| US-06 | P0 | As a DME, I want re-running a capture to preserve the previous run's output, so that I can compare label sets across ruleset snapshots. | Sibling run dirs, `{capture}_{datetime}` | | US-07 | P0 | As an OPS, I want the same capture and ruleset snapshot to yield identical labels, so that I can verify the pipeline is behaving deterministically. | Goal 2; `--offline` | | US-08 | P1 | As an OPS, I want an `--offline` mode that runs without the NGFW, so that I can process captures and test the pipeline when the lab is unavailable. | Marked reduced coverage | | US-09 | P1 | As a DME, I want pcapng and gzipped captures accepted directly, so that I can use files as they come off Wireshark or a sensor without pre-processing. | Normalization | @@ -225,7 +232,7 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe - **Label a capture (default).** `flabel capture.pcap` → normalize → Zeek + Suricata + PANW replay → consolidate → write run directory. Fails clearly if the lab is unreachable. - **Label without the lab.** `flabel --offline capture.pcap` → Tier 2 only → output stamped as reduced coverage. -- **Compare across ruleset snapshots.** Re-run the same capture; each run lands in its own timestamped directory; `latest` points to the newest. +- **Compare across ruleset snapshots.** Re-run the same capture; each run lands in its own sibling directory named `{capture-name}_{datetime}`; sorting the set gives them in chronological order. - **Inspect a label.** Read a label's `sources[]`, then join its `flow.uid` to `zeek/conn.log` for the full flow record. **Design Constraints / Guidelines:** @@ -251,7 +258,8 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe ### US-01: Label a capture end to end -- Given a valid pcap and a reachable lab, when `flabel capture.pcap` runs, then a timestamped run directory is created containing `zeek/` with the Zeek logs and `labels.json`. +- Given a valid pcap and a reachable lab, when `flabel my-capture.pcap` runs, then a directory named `my-capture_{datetime}` is created containing `zeek/` with the Zeek logs and `labels.json`. +- Given an input named `my-capture.pcap.gz`, when the run completes, then the output directory is named `my-capture_{datetime}` — both extensions stripped — and its `{datetime}` contains no colons. - Given a capture containing no detectable threats, when the run completes, then `labels.json` exists with an empty `labels` array and the run is reported as successful. - Given the NGFW is unreachable and `--offline` was not passed, when flabel runs, then it fails with a message naming the unreachable Tier 1 dependency, and no partial label file is written. @@ -279,7 +287,8 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe ### US-06: Re-runs preserve history -- Given a capture already processed, when flabel runs against it again, then a new timestamped directory is created, the previous run directory is unmodified, and `latest` points to the new run. +- Given a capture already processed, when flabel runs against it again, then a new sibling directory `{capture-name}_{datetime}` is created and the previous run directory is unmodified. +- Given two or more run directories for the same capture, when their names are sorted lexicographically, then they appear in chronological order and the last is the most recent. ### US-07: Reproducibility @@ -386,7 +395,7 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | | | | | | | :-: | :-: | :-: | :-: | :-: | | **\#** | **Case** | **Expected Behavior** | **Observed Behavior** | **Pass/Fail** | -| 1 | Valid pcap, lab reachable | Run directory created with `zeek/` and `labels.json`; `latest` points to it | | | +| 1 | Valid pcap, lab reachable | Directory `{capture-name}_{datetime}` created with `zeek/` and `labels.json` | | | | 2 | Capture with no detections | Success; `labels` is an empty array | | | | 3 | pcapng input | Normalized to pcap, processed, conversion recorded in provenance | | | | 4 | Gzipped pcap input | Decompressed and processed | | | @@ -395,7 +404,7 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | 7 | Truncated capture | Processed; `input_status: partial` with packet count reached | | | | 8 | Flow flagged by both Tier 1 and Tier 2 | Single entry; both entries in `sources[]`; `max_tier` = 1 | | | | 9 | Detection uncorrelatable to any flow | Appears in `unmatched_detections[]` with reason; counted in run metadata | | | -| 10 | Re-run of an already-processed capture | New timestamped directory; prior run untouched; `latest` updated | | | +| 10 | Re-run of an already-processed capture | New sibling directory; prior run untouched; sorted names are chronological | | | | 11 | Two `--offline` runs, pinned snapshot | `labels.json` identical apart from run timestamps | | | | 12 | `--offline` with no lab reachable | Succeeds; Tier 2 labels only; output records Tier 1 not attempted | | | | 13 | Default run with lab unreachable | Fails naming the Tier 1 dependency; no partial label file | | | diff --git a/docs/status.yaml b/docs/status.yaml index bc2e8c7..ca0a29c 100644 --- a/docs/status.yaml +++ b/docs/status.yaml @@ -4,7 +4,7 @@ project: "flabel" created: "2026-08-11" github_repo: "DeepTempo/flabel" notion_url: "https://app.notion.com/p/3b92a84a5230812aa64aef638e389725" -prd_gdoc_url: "https://docs.google.com/document/d/1q_muS6AmCuvcNva4C1WtRE3_KXArJPxnapttsU6UQIE/edit" +prd_gdoc_url: "https://docs.google.com/document/d/1WwKukQ71L6JQWkSpKvoR4lgwB96w3x1WHAHzZ30ul2E/edit" # v0.2; supersedes v0.1 doc 1q_muS6Am... (delete manually — Drive MCP has no delete) current_stage: eng_review # NOTE: issue numbers are not in stage order. Two creates hit a transient TLS # error and were retried, and GitHub never reuses issue numbers. This mapping is From 45b8d4895ab86a1eacd4d5e8f1eb7da922d237e0 Mon Sep 17 00:00:00 2001 From: Craig yourname Date: Tue, 11 Aug 2026 16:08:16 -0700 Subject: [PATCH 6/6] stage(prd): JA4 labeling moved into phase one (v0.3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Craig's review comment: JA4 as a labeling source needs to be phase one, not a future phase. Since I had written that it 'promotes into Tier 2 with no architectural change', deferring it was how it would end up never scheduled. Splits capability from content: - CAPABILITY is now in v1. Suricata ja3.hash AND ja4.hash matching are both first-class and enabled; JA4 rules pass through the identical per-source admission filter and snapshot provenance as any other rule. Built, enabled, and tested in v1. - CONTENT remains unavailable. No admitted source publishes ja4.hash rules, so JA4 label output is zero on release. Only this narrowed item stays in the Out of Scope table. Preserves the distinction that keeps the design coherent: a Zeek-computed fingerprint is an attribute, never a verdict. A label arises only where a fingerprint matches an admitted rule — the same machinery as every other detection, which is precisely why the promotion is cheap. Guards against the path silently rotting: the run records the admitted JA4 rule count, so zero is visibly 'no content published' rather than ambiguously 'path broken', and the capability is tested against a synthetic JA4 rule so it is proven independent of content availability. Changes: Out of Scope row narrowed to content only; §6.2 wording sharpened; §6.3 rewritten; US-14 (P0, labeling) and US-15 (P1, content visibility) added, US-10 note clarified as the enrichment half; acceptance criteria for US-10/14/15; success metric now tracks a separate JA4 rule count; new risk row; open question 11; test cases 17-18. Revision history at 0.3. research.md §C5 gets a superseding note — its finding (no free JA4 verdict source) is unchanged and still accurate; what changed is that the capability is no longer deferred behind it. Filed #13 to track JA4 content sourcing as a live issue, not a future phase. Refs #2 --- docs/prd.md | 32 ++++++++++++++++++++++++++------ docs/research.md | 2 ++ 2 files changed, 28 insertions(+), 6 deletions(-) diff --git a/docs/prd.md b/docs/prd.md index 73bc43c..fd3bf2a 100644 --- a/docs/prd.md +++ b/docs/prd.md @@ -16,6 +16,7 @@ | **Date** | **Author** | **Version** | **Change Summary** | | 2026-08-11 | Craig | 0.1 | Initial draft | | 2026-08-11 | Craig | 0.2 | Output layout changed to sibling run directories named `{capture-name}_{datetime}`; `latest` pointer removed | +| 2026-08-11 | Craig | 0.3 | JA4 labeling moved into phase one as a first-class Tier 2 capability (US-14, US-15); only JA4 *rule content* remains out of scope | ## 1. Problem Statement @@ -61,7 +62,7 @@ The people affected are the engineers building DeepTempo's detection models, who | :-: | :-: | :-: | | **Item** | **Reason** | **Future Phase?** | | Label validation against a ground-truth corpus | Trust-by-construction decided at Stage 1; trustworthiness argued from provenance | TBD — flagged for eng-review | -| JA4 as a labeling source | ET Open ships no `ja4.hash` rules yet; no free maintained JA4 verdict feed exists. JA4 is computed as enrichment only | Yes — promotes into Tier 2 with no architectural change once ET publishes JA4 rules | +| JA4 *rule content* — a source of malicious JA4 fingerprints | No admitted source publishes `ja4.hash` rules today, and no free maintained JA4 verdict feed exists. **The JA4 labeling capability itself is in phase one** (§6.3) — only the content is missing | Content only. The capability ships in v1 and begins producing labels the moment any admitted source publishes JA4 rules, with no code change | | Snort 3 as the Tier 2 engine | Suricata selected on free high-confidence ruleset volume and native fingerprint keywords | No | | FortiGate as the NGFW | PANW VM-Series selected as Tier 1 | No | | Free/OSS L7 equivalent to PANW App-ID | No free equivalent exists; line of inquiry closed | No | @@ -107,7 +108,7 @@ The people affected are the engineers building DeepTempo's detection models, who - All Zeek logs generated for the capture are retained in the output. - JA4 (and JA4+, subject to the licence decision) is computed for every TLS connection via the `zeek/foxio/ja4` package. -- **Fingerprints are recorded as attributes, never as verdicts.** A fingerprint value alone never produces a label. +- **A computed fingerprint is an attribute, not a verdict.** Zeek's JA4 output never produces a label by itself. Labels arise only where a fingerprint **matches an admitted rule**, which happens in the Tier 2 path (§6.3) — the same way every other detection is produced. - Zeek `uid` is assigned to every flow and becomes the join key between `labels.json` and the Zeek logs. ### 6.3 Tier 2 Detection — Suricata @@ -119,7 +120,9 @@ The people affected are the engineers building DeepTempo's detection models, who - **Per-source admission policy.** Signature rulesets (ET Open) are filtered on rule metadata: `confidence == High` **and** `signature_severity in (Major, Critical)`. IOC feeds (abuse.ch, malsilo, and similar) carry no such metadata and are admitted wholesale, with the feed snapshot date as their provenance. - Rules lacking a `confidence` tag are **excluded** (fail-closed). - Excluded sources: hunting/anomaly rulesets, self-described aggressive blacklists, and Positive Technologies. -- Encrypted-traffic detection is part of this tier, via ET Open's JA3 rules matched with Suricata's native fingerprint keywords. +- **Encrypted-traffic detection is part of this tier, and both fingerprint families are first-class in phase one.** Suricata's native `ja3.hash` **and** `ja4.hash` matching are both enabled, and JA4 rules pass through the identical per-source admission filter and snapshot provenance as any other rule. +- **JA4 has the capability but not yet the content.** No admitted source currently publishes `ja4.hash` rules, so JA4 label output will be zero on release. This is a *content* gap, not a capability gap: when any admitted source ships JA4 rules, they are picked up and produce labels with **no code change**. The path is therefore built, enabled, and tested in v1 rather than deferred. +- Because an inactive path is indistinguishable from a broken one, **the run records how many JA4 rules were admitted.** A zero count proves the path ran and found no content, rather than leaving silence to be misread as either working or failing. - **The admitted rule set is snapshotted per run** — source, version, and date — because filter output changes as vendors revise metadata. Without this, two runs of "the same" flabel are not comparable. ### 6.4 Tier 1 Detection — PANW VM-Series @@ -221,7 +224,9 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | US-07 | P0 | As an OPS, I want the same capture and ruleset snapshot to yield identical labels, so that I can verify the pipeline is behaving deterministically. | Goal 2; `--offline` | | US-08 | P1 | As an OPS, I want an `--offline` mode that runs without the NGFW, so that I can process captures and test the pipeline when the lab is unavailable. | Marked reduced coverage | | US-09 | P1 | As a DME, I want pcapng and gzipped captures accepted directly, so that I can use files as they come off Wireshark or a sensor without pre-processing. | Normalization | -| US-10 | P1 | As a DME, I want JA4 fingerprints recorded on TLS flows, so that I can use them as model features and pivot on them during analysis. | Enrichment, not verdicts | +| US-10 | P1 | As a DME, I want JA4 fingerprints recorded on TLS flows, so that I can use them as model features and pivot on them during analysis. | The enrichment half — Zeek-computed attributes, no verdict | +| US-14 | P0 | As a DME, I want a JA4 fingerprint matching an admitted rule to produce a Tier 2 label, so that encrypted-traffic detections are labelled by the same machinery as every other detection and need no rework when JA4 rule content becomes available. | The labeling half. Capability ships in v1; content arrives later | +| US-15 | P1 | As an OPS, I want the run to record how many JA4 rules were admitted, so that I can tell "no JA4 content published yet" apart from "the JA4 path is broken". | Guards against the capability silently rotting | | US-11 | P1 | As an OPS, I want the ruleset admission filter and its results recorded per run, so that I can see exactly which rules were live and how many were excluded. | Ties to issue #11 | | US-12 | P2 | As an OPS, I want a documented environment diagram in draw.io and mermaid form, so that the lab can be rebuilt or handed over. | From the brief | | US-13 | P2 | As a DME, I want `labels.json` to carry a schema version, so that a shape change breaks loudly rather than silently. | Forward compatibility | @@ -309,7 +314,18 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe ### US-10: Fingerprint enrichment - Given a capture containing TLS connections, when the run completes, then JA4 values are present on those flows in the Zeek output. -- Given a JA4 value matching no rule, when labels are written, then no label is produced from the fingerprint alone. +- Given a JA4 value that matches no admitted rule, when labels are written, then no label is produced from that fingerprint — a computed fingerprint alone is never a verdict. + +### US-14: JA4 labeling capability + +- Given an admitted ruleset containing a `ja4.hash` rule, when a capture contains a TLS flow whose JA4 matches it, then a Tier 2 label is produced carrying that rule's identity, ruleset snapshot, and confidence — structurally identical to any other Tier 2 label. +- Given JA4 rules are present in an admitted source, when the run executes, then Suricata JA4 fingerprinting is active rather than silently skipped. +- Given a JA4-matched label, when it is inspected, then it is indistinguishable in shape from a JA3- or content-matched Tier 2 label, requiring no special handling by consumers. + +### US-15: JA4 content visibility + +- Given no admitted source publishes JA4 rules, when the run completes, then run metadata records the JA4 rule path as active with an admitted JA4 rule count of zero. +- Given an admitted source begins publishing JA4 rules, when the next run executes, then those rules are admitted and counted with no change to flabel's code or configuration. ## 10. Technical Considerations @@ -355,7 +371,7 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | Replay integrity | Packets sent equal packets seen, or the discrepancy is reported | Per-run reconciliation; failure surfaced, never suppressed | TBD | | Unmatched detection rate | Tracked and reported; no target set for v1 | Per-run `unmatched_count`; trend reviewed as capture volume grows | TBD | | Supported-format success rate | 100% of pcap, pcapng, and gzipped inputs process or fail with a clear reason | Test matrix across format variants | TBD | -| Tier 2 admitted-rule count | Recorded per source; adequacy assessed once known | Measured at build time (issue #11) | TBD | +| Tier 2 admitted-rule count | Recorded per source, **including a separate JA4 rule count** | Measured at build time (issue #11); JA4 count surfaced in run metadata every run | TBD | **Note:** these metrics measure *pipeline integrity*, not *label accuracy*. Under trust-by-construction there is no measurement of false-positive rate — see Risks. @@ -368,6 +384,7 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | Trust-by-construction is unfalsifiable — no false-positive rate can be quoted if a label consumer asks | High | High | Ruleset snapshots recorded per run so labels are reproducible and auditable even if unmeasured; fail-closed admission filter; flagged for eng-review reconsideration | | Tier 1 detections cannot be correlated to capture flows (port reuse, NAT, tunnelling) | Med | Med | Tuple-driven matching with time only scoping the query; uncorrelated detections emitted in `unmatched_detections[]` rather than dropped or guessed | | Fingerprint verdicts drift as benign software adopts a fingerprint, silently invalidating a verdict | Med | Med | Fingerprint verdicts sourced through ET rules rather than raw feeds, so vendor revisions carry the aging burden; ruleset snapshot dates recorded | +| The JA4 labeling path ships with no rule content, so it appears functional while producing nothing — and may quietly rot untested | Med | Med | Admitted JA4 rule count surfaced in run metadata every run, so zero content is visible rather than assumed; the path is tested against a synthetic JA4 rule so the capability is proven independent of content availability; sourcing tracked as a live issue rather than a future phase | | Admission filter proves too strict, leaving Tier 2 coverage too thin to be useful | Med | Med | Admitted-rule counts measured per source (issue #11); untagged-rule policy revisitable (issue #10); paid high-fidelity rulesets remain a costed fallback | | JA4+ licensing (FoxIO 1.1, non-commercial) conflicts with output feeding product models | Med | High | Legal engaged as an approver; plain JA4 is BSD 3-Clause and available as an unrestricted fallback | | Capture data or credentials leak into the public repository | Low | High | `.gitignore` coverage for captures, logs, and `.env`; no real capture data in fixtures; pre-commit secret checks | @@ -389,6 +406,7 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | 8 | What fixture strategy provides test captures without committing real traffic? | TBD | At scaffold | Open | | 9 | Should `pawpatrules` remain admitted without an FP review, being the least-vetted admitted source? | Craig | At spec | Open | | 10 | What are the review dates for the success metrics? | Craig | TBD | Open | +| 11 | Where does JA4 rule content come from — wait for ET to publish, evaluate a commercial feed, or derive our own from malware captures? The capability ships in v1 regardless; this decides when it starts producing labels. | Craig | Post-v1 | Open — tracked as a live issue | 15. Basic Test Cases @@ -410,6 +428,8 @@ Personas: **DME** = DeepTempo detection-model engineer (primary consumer of labe | 13 | Default run with lab unreachable | Fails naming the Tier 1 dependency; no partial label file | | | | 14 | Replay with packet-count mismatch | Discrepancy reported; run not presented as full coverage | | | | 15 | TLS capture | JA4 present on TLS flows in Zeek output; no label from fingerprint alone | | | +| 17 | Synthetic `ja4.hash` rule matching a capture's TLS flow | Tier 2 label produced, structurally identical to a JA3 or content match, carrying rule identity and snapshot | | | +| 18 | No admitted source publishes JA4 rules | Run metadata records JA4 path active with admitted JA4 rule count of zero | | | | 16 | Every emitted label | Complete provenance block; `flow.uid` resolves to exactly one `conn.log` record | | | ## 14. References & Related Documents diff --git a/docs/research.md b/docs/research.md index 0662a10..e0b09f9 100644 --- a/docs/research.md +++ b/docs/research.md @@ -231,6 +231,8 @@ The `--offline` flag runs the Tier 2 path alone, with the output clearly marked | **2** | Suricata — ET Open (metadata-filtered, **including `emerging-ja3.rules`**) plus the CC0/MIT IOC feeds | Per-rule `confidence: High` is a vendor-declared low-FP assertion for signature rules; upstream curation plus snapshot provenance for IOC feeds | | *Enrichment* | JA4 / JA4+ via Zeek | Recorded as an attribute, not a verdict — no free verdict source yet. Promote into Tier 2 when ET ships `ja4.hash` rules. | +> **Superseded by the PRD (2026-08-11).** This section recommended deferring the JA4 *labeling* path until rule content existed. `docs/prd.md` v0.3 instead moves the **capability** into phase one — `ja4.hash` matching is built, enabled, and tested in v1, with only the rule *content* remaining unavailable. The finding above (no free JA4 verdict source exists) is unchanged and still accurate; what changed is that the capability is no longer deferred behind it. + --- ## Tools and maintenance status