Validator Design

[sehkone]

Validator Design

This document defines the Validator — the automated verification stage that checks Dataset Bundles before they enter the QFlow pipeline. For the overall platform architecture, see the overview document. For the bundle contents, see the Generator Design and Ground Truth Definition.

---

1. Purpose

The Validator automatically verifies that a Dataset Bundle produced by the Generator is complete, well-formed, internally consistent, and faithful to the Scenario Definition. It acts as a quality gate between generation and QFlow Replay.

Why it is needed:

• The Generator orchestrates multiple tools (nmap, Metasploit, tcpdump, Sysmon, Falco, etc.) across multiple VMs/containers. Any of these can fail silently.
• A missing PCAP file or a Ground Truth record that doesn't match the actual traffic would produce misleading evaluation results downstream.
• Automated validation catches these problems before they propagate.

Pipeline position:

Scenario Definition (YAML)
      ↓
Generator
      ↓
Dataset Bundle (+ Ground Truth)
      ↓
Validator  ← this component
      ↓
QFlow Replay

---

2. Input and Output

2.1 Input

Item	Description
Bundle directory	Path to the Dataset Bundle (e.g., dataset_bundle/)
Scenario Definition	Path to the .scenario.yaml file used to generate the bundle

2.2 Output

A Validation Report in JSON format containing:

• Per-check results (pass / fail / warn)
• Summary statistics (total, passed, failed, warnings)
• Human-readable messages for every finding

The Validator also returns an exit code:

Exit Code	Meaning
0	All checks passed
1	Warnings only (bundle is usable)
2	Errors found (bundle should not be used)

---

3. Validation Levels

Checks are organized into five levels. Levels 1–3 are fast (file-system and JSON parsing only) and always run. Levels 4–5 require deep artifact parsing (PCAP, log correlation) and are optional.

Level 1: Bundle Structure       — Do all expected files exist?
Level 2: Schema Compliance      — Are all files well-formed?
Level 3: Ground Truth Integrity — Is the Ground Truth internally consistent?
Level 4: Cross-Artifact Alignment — Does Ground Truth match the artifacts?
Level 5: Scenario Fulfillment   — Does the bundle satisfy the Scenario Definition?

---

4. Validation Checks

4.1 Level 1 — Bundle Structure

Verifies that the expected directory structure and files are present.

Check ID	Description	Severity
L1-001	meta.json exists	error
L1-002	ground_truth/manifest.jsonl exists	error
L1-003	ground_truth/manifest.jsonl is non-empty	error
L1-004	For each PCAP target in Scenario Definition collection.pcap: corresponding file exists in net/	error
L1-005	For each Sysmon target in collection.sysmon: host/<hostname>/sysmon.jsonl exists	error
L1-006	For each Falco target in collection.falco: host/<hostname>/falco.jsonl exists	error
L1-007	If Scenario declares workload includes server with DB: db/ directory contains at least one file	error
L1-008	If Scenario declares workload includes cloud: cloud/ directory contains at least one file	error

4.2 Level 2 — Schema Compliance

Verifies that each file is well-formed and parseable.

Check ID	Description	Severity
L2-001	meta.json is valid JSON	error
L2-002	meta.json contains required fields (name, environment, infrastructure, duration)	error
L2-003	Each line in manifest.jsonl is valid JSON	error
L2-004	Each Ground Truth record has required common fields: scope, label, start, end, source, target	error
L2-005	label values are normal or anomaly only	error
L2-006	scope values are session or event only	error
L2-007	PCAP files begin with valid pcap/pcapng magic bytes	error
L2-008	Each line in Sysmon JSONL files is valid JSON	warn
L2-009	Each line in Falco JSONL files is valid JSON	warn

4.3 Level 3 — Ground Truth Integrity

Verifies internal consistency of the Ground Truth data.

Check ID	Description	Severity
L3-001	start < end for every record	error
L3-002	Records are sorted by start timestamp	warn
L3-003	Session-scope network records have: session_type, protocol, src_ip, src_port, dst_ip, dst_port	error
L3-004	Session-scope process records have: session_type, host, pid	error
L3-005	Anomaly records have category field	error
L3-006	Attack records (category: attack) have technique, phase, tool	error
L3-007	technique values match MITRE ATT&CK format (T[0-9]{4} or T[0-9]{4}\.[0-9]{3})	warn
L3-008	phase values are valid kill chain phases	error
L3-009	Campaign steps: within each campaign_id, steps are sequential starting from 1, no gaps	error
L3-010	Campaign steps: within each campaign_id, records are in temporal order (step N starts before step N+1)	warn
L3-011	source and target host names appear in meta.json host list	warn
L3-012	IP addresses in network records match IPs assigned to declared hosts	warn

Valid kill chain phases (L3-008):

reconnaissance, initial_access, credential_access, lateral_movement, c2, exfiltration

4.4 Level 4 — Cross-Artifact Alignment

Verifies that Ground Truth records correspond to actual data in the artifacts. These checks require parsing PCAP and log files.

Check ID	Description	Severity
L4-001	For each network session in Ground Truth: at least one matching flow (5-tuple) exists in PCAP within the declared time window	error
L4-002	For each process session in Ground Truth: at least one matching process event exists in the corresponding host log (Sysmon or Falco)	warn
L4-003	No PCAP flows exist that are completely absent from Ground Truth (coverage check)	warn
L4-004	Attack sessions: traffic exists in PCAP during the declared [start, end] window	error

Matching criteria for L4-001:

A Ground Truth network session matches a PCAP flow if:

• Protocol matches
• Source IP and destination IP match (or reverse for bidirectional)
• Destination port matches
• At least one packet in the flow falls within [start - tolerance, end + tolerance]

The tolerance accounts for timestamp granularity differences between the Generator's clock and the capture clock. Default: 5 seconds.

Note on L4-003: This check produces warnings, not errors. The Generator may produce background traffic (e.g., ARP, DNS, DHCP) that is not explicitly recorded in Ground Truth. The Validator reports unmatched flows for manual review but does not fail the bundle.

4.5 Level 5 — Scenario Fulfillment

Verifies that the bundle satisfies what the Scenario Definition declared.

Check ID	Description	Severity
L5-001	For each declared normal activity: at least one label: normal record exists with matching source host and approximate timing	warn
L5-002	For each declared attack activity: at least one label: anomaly record exists with matching technique and tool	error
L5-003	Ground Truth time span approximately covers the declared duration.total (within ±10%)	warn
L5-004	All declared hosts appear in at least one Ground Truth record (as source or target)	warn
L5-005	All declared collection targets produced non-empty artifact files	error

---

5. Validation Report Format

The Validator outputs a JSON report:

{
  "bundle": "dataset_bundle/",
  "scenario": "ac-1.scenario.yaml",
  "timestamp": "2025-01-15T12:00:00Z",
  "levels_run": [1, 2, 3, 4, 5],
  "summary": {
    "total": 28,
    "passed": 25,
    "failed": 1,
    "warnings": 2
  },
  "results": [
    {
      "check_id": "L1-001",
      "level": 1,
      "status": "pass",
      "message": "meta.json exists"
    },
    {
      "check_id": "L4-001",
      "level": 4,
      "status": "fail",
      "message": "Ground Truth network session has no matching PCAP flow",
      "details": {
        "record_index": 42,
        "src_ip": "10.0.2.5",
        "dst_ip": "10.0.1.10",
        "dst_port": 445,
        "start": "2025-01-15T06:02:00Z",
        "end": "2025-01-15T06:02:05Z"
      }
    },
    {
      "check_id": "L3-002",
      "level": 3,
      "status": "warn",
      "message": "Records not sorted by start timestamp",
      "details": {
        "record_index": 15,
        "expected_after": "2025-01-15T03:00:00Z",
        "actual": "2025-01-15T02:59:58Z"
      }
    }
  ]
}

---

6. Severity Definitions

Severity	Meaning	Effect on Exit Code
error	Bundle is unusable for this aspect. The artifact is missing, corrupted, or fundamentally inconsistent.	Exit code 2
warn	Bundle is usable but has a concern. Minor timing drift, non-critical gap, or informational mismatch.	Exit code 1 (if no errors)
pass	Check succeeded.	No effect

If any check produces error, the overall result is fail (exit code 2) regardless of other results. If only warn findings exist, the overall result is warn (exit code 1).

---

7. Usage

validator <bundle_path> <scenario_path> [options]

Options:
  --level <N>       Maximum validation level to run (1-5, default: 3)
  --tolerance <sec> Time tolerance for cross-artifact matching (default: 5)
  --output <path>   Write report to file (default: stdout)
  --format <fmt>    Output format: json, text (default: json)

Examples:

# Quick structural check (Levels 1-3)
validator ./dataset_bundle/ ./ac-1.scenario.yaml

# Full validation including PCAP parsing (Levels 1-5)
validator ./dataset_bundle/ ./ac-1.scenario.yaml --level 5

# With custom tolerance and text output
validator ./dataset_bundle/ ./ac-1.scenario.yaml --level 5 --tolerance 10 --format text

---

8. Implementation Notes

8.1 Language

Rust, consistent with the Generator. Shares common crate for:

• Scenario Definition parsing (scenario crate)
• Ground Truth record types (ground_truth crate)
• Dataset Bundle path conventions (bundle crate)

8.2 PCAP Parsing (Level 4)

For cross-artifact alignment, the Validator needs to extract network flows from PCAP files. Options:

Approach	Pros	Cons
pcap crate	Native Rust, no external dependency	Low-level, must build flow extraction
rtshark wrapper	Leverages tshark's protocol dissection	Requires tshark installed

The Validator does not need deep protocol dissection — only 5-tuple extraction and timestamp matching. The pcap crate is sufficient for Phase 1.

8.3 Performance Considerations

• Levels 1–3 operate on file metadata and JSON parsing only. Expected runtime: < 1 second for any bundle.
• Level 4 requires reading entire PCAP files. For large bundles, this may take minutes. The Validator streams PCAP packets and builds an in-memory flow table indexed by 5-tuple.
• Level 5 re-uses data already loaded by Level 4 when available.

---

9. Extension Path

Extension	Trigger	Description
Event-level validation	Ground Truth adds scope: event records	Validate that event records reference valid artifact offsets (packet numbers, line numbers)
Encryption verification	Encryption-focused anchor cases	Check that PCAP contains TLS handshakes when encryption: tls is declared; check for plaintext when tls_termination is used on inner segment
Statistical validation	Large-scale bundles	Verify that traffic volume, session counts, and timing distributions are reasonable for the declared scenario
CI integration	Automated testing pipeline	Run Validator as part of CI after every Generator change to catch regressions