Ground Truth Definition

[sehkone]

Ground Truth Definition

This document defines the Ground Truth format — the answer data that records what activity (normal or anomalous) occurred during a scenario. For the overall platform architecture, see the overview document. For the scenario input format, see the Scenario Definition Schema.

---

1. Purpose

Ground Truth is the authoritative record of what happened during a dataset generation run. It enables:

• Validation: The Validator checks that collected artifacts are consistent with intended activities
• Evaluation: QFlow replay results can be compared against Ground Truth to measure detection accuracy
• Reproducibility: Ground Truth documents exactly what was generated, enabling meaningful comparisons across runs

The Generator produces Ground Truth automatically as it executes each activity in the Scenario Definition.

---

2. Label System

2.1 Binary Classification

Every Ground Truth record has a label field with one of two values:

Label	Meaning
normal	Legitimate, expected behavior
anomaly	Any deviation from normal — attacks, policy violations, runaway behavior, etc.

This binary classification is the primary label. All downstream analysis starts from this distinction.

2.2 Anomaly Categories

Anomaly records include a category field for finer classification:

Category	Description	Example
attack	Malicious activity with adversarial intent	nmap scan, credential dump, data exfiltration
policy_violation	Unauthorized action without adversarial intent	AI Agent accessing tables outside its allowed scope
runaway	Excessive or uncontrolled behavior	AI Agent sending 1000 req/min to external LLM

New categories can be added without changing the binary label system.

2.3 Attack Metadata

When category is attack, additional fields provide context:

Field	Type	Required	Description
technique	string	yes	MITRE ATT&CK technique ID (e.g., T1046)
phase	string	yes	Kill chain phase: reconnaissance, initial_access, credential_access, lateral_movement, c2, exfiltration
tool	string	yes	Tool used (e.g., nmap, metasploit, caldera)
cve	string	no	CVE identifier if applicable (e.g., CVE-2017-0144)

---

3. Granularity

3.1 Scope Levels

Ground Truth supports multiple granularity levels, indicated by the scope field:

Scope	Unit	Description	Phase
session	Network session (5-tuple) or process execution	One record per TCP/UDP session or per host-side process	Phase 1 (required)
event	Individual packet or log line	One record per PCAP packet or per Sysmon/Falco event	Future extension

3.2 Session Scope (Phase 1)

Each record describes one network session or one process execution:

• Network session: Identified by 5-tuple (protocol, src_ip, src_port, dst_ip, dst_port)
• Process execution: Identified by host + process ID + start timestamp

The Generator records session information as it executes each activity. Attack tools like nmap produce multiple sessions per activity; each session gets its own record.

3.3 Event Scope (Future Extension)

Event-level records reference specific artifacts:

• PCAP: Artifact filename + packet timestamp or packet number
• Sysmon/Falco JSONL: Artifact filename + line offset or event timestamp

Event-level records use parent to reference their containing session. This allows consumers to read only session-level records for coarse analysis, or drill into event-level records for fine-grained analysis.

---

4. Campaign Relationship

Attack activities that form a multi-step kill chain share a campaign_id and are ordered by step:

{"scope": "session", "campaign_id": "exfil-chain-1", "step": 1, "label": "anomaly", "category": "attack", "technique": "T1046", "phase": "reconnaissance", ...}
{"scope": "session", "campaign_id": "exfil-chain-1", "step": 2, "label": "anomaly", "category": "attack", "technique": "T1210", "phase": "initial_access", ...}
{"scope": "session", "campaign_id": "exfil-chain-1", "step": 3, "label": "anomaly", "category": "attack", "technique": "T1003", "phase": "credential_access", ...}
{"scope": "session", "campaign_id": "exfil-chain-1", "step": 4, "label": "anomaly", "category": "attack", "technique": "T1048", "phase": "exfiltration", ...}

• campaign_id: Groups related attack steps into a single campaign
• step: Ordering within the campaign (1-based)
• Records without campaign_id are standalone activities (normal traffic or isolated anomalies)

---

5. Record Schema

5.1 Common Fields (all records)

Field	Type	Required	Description
scope	string	yes	session or event
label	string	yes	normal or anomaly
start	string	yes	ISO 8601 timestamp (e.g., 2025-01-15T06:02:00Z)
end	string	yes	ISO 8601 timestamp
source	string	yes	Source host name (e.g., attacker-001)
target	string	yes	Target host name (e.g., win-001)

5.2 Session-Scope Fields

Field	Type	Required	Description
session_type	string	yes	network or process
protocol	string	conditional	tcp, udp, icmp (required when session_type: network)
src_ip	string	conditional	Source IP (required when session_type: network)
src_port	number	conditional	Source port
dst_ip	string	conditional	Destination IP (required when session_type: network)
dst_port	number	conditional	Destination port
host	string	conditional	Host name (required when session_type: process)
pid	number	conditional	Process ID (required when session_type: process)
process_name	string	no	Process name (e.g., nmap, svchost.exe)

5.3 Anomaly Fields

Field	Type	Required	Description
category	string	yes (when anomaly)	attack, policy_violation, runaway
technique	string	conditional	MITRE ATT&CK ID (required when category: attack)
phase	string	conditional	Kill chain phase (required when category: attack)
tool	string	conditional	Tool name (required when category: attack)
cve	string	no	CVE identifier (e.g., CVE-2017-0144)
description	string	no	Human-readable description of the anomaly

5.4 Campaign Fields

Field	Type	Required	Description
campaign_id	string	no	Campaign identifier for multi-step attacks
step	number	conditional	Step number within campaign (required when campaign_id is present)

5.5 Event-Scope Fields (Future Extension)

Field	Type	Required	Description
parent	string	yes (when event)	Reference to parent session (session identifier)
artifact	string	yes (when event)	Artifact filename (e.g., edge.pcap, win-001/sysmon.jsonl)
offset	string	yes (when event)	Location within artifact (packet number, line number, or timestamp)

---

6. Format

Ground Truth is stored as ground_truth/manifest.jsonl within the Dataset Bundle:

dataset_bundle/
  meta.json
  net/
  host/
  db/
  cloud/
  ground_truth/
    manifest.jsonl        ← Ground Truth file

Each line in manifest.jsonl is a single JSON record. Records are ordered by start timestamp.

---

7. Examples

7.1 Normal Web Browsing Session

{"scope": "session", "label": "normal", "start": "2025-01-15T00:05:12Z", "end": "2025-01-15T00:05:15Z", "source": "win-001", "target": "linux-srv-001", "session_type": "network", "protocol": "tcp", "src_ip": "10.0.1.10", "src_port": 52341, "dst_ip": "10.0.1.50", "dst_port": 443}

7.2 Attack Session with Campaign

{"scope": "session", "label": "anomaly", "category": "attack", "campaign_id": "exfil-chain-1", "step": 1, "start": "2025-01-15T06:02:00Z", "end": "2025-01-15T06:02:05Z", "source": "attacker-001", "target": "win-001", "session_type": "network", "protocol": "tcp", "src_ip": "10.0.2.5", "src_port": 43210, "dst_ip": "10.0.1.10", "dst_port": 445, "technique": "T1046", "phase": "reconnaissance", "tool": "nmap"}

7.3 Attack with CVE

{"scope": "session", "label": "anomaly", "category": "attack", "campaign_id": "exfil-chain-1", "step": 2, "start": "2025-01-15T07:00:00Z", "end": "2025-01-15T07:30:00Z", "source": "attacker-001", "target": "win-002", "session_type": "network", "protocol": "tcp", "src_ip": "10.0.2.5", "src_port": 44100, "dst_ip": "10.0.1.11", "dst_port": 445, "technique": "T1210", "phase": "initial_access", "tool": "metasploit", "cve": "CVE-2017-0144"}

7.4 AI Agent Policy Violation

{"scope": "session", "label": "anomaly", "category": "policy_violation", "start": "2025-01-15T06:00:00Z", "end": "2025-01-15T06:30:00Z", "source": "ai-agent-001", "target": "db-001", "session_type": "network", "protocol": "tcp", "src_ip": "10.0.1.20", "src_port": 55100, "dst_ip": "10.0.2.10", "dst_port": 5432, "description": "AI agent accessed hr_salary table outside its allowed scope"}

7.5 AI Agent Runaway

{"scope": "session", "label": "anomaly", "category": "runaway", "start": "2025-01-15T08:00:00Z", "end": "2025-01-15T09:00:00Z", "source": "ai-agent-002", "target": "ai-proxy-001", "session_type": "network", "protocol": "tcp", "src_ip": "10.0.1.21", "src_port": 55200, "dst_ip": "10.0.1.30", "dst_port": 8443, "description": "AI agent sent 60000 requests in 1 hour to external LLM endpoint"}

---

8. Generation Process

The Generator produces Ground Truth as a byproduct of scenario execution:

1. Before execution: The Orchestrator reads the Scenario Definition and prepares a Ground Truth writer
2. During normal activities: Each normal activity generator reports its sessions (source, target, protocol, ports, timestamps)
3. During attack activities: Each attack tool wrapper reports its sessions with attack metadata (technique, phase, tool, CVE). If the attack is part of a campaign, the Orchestrator assigns campaign_id and step
4. After execution: The writer flushes all records to manifest.jsonl, sorted by start timestamp

Ground Truth records are written with actual timestamps observed during execution, not the planned times from the Scenario Definition. This ensures accuracy even when tools take longer or shorter than expected.

---

9. Extension Path

9.1 Event-Level Records

When the Generator supports event-level tracking (e.g., per-packet or per-log-line), event records are appended to the same manifest.jsonl:

{"scope": "session", "label": "anomaly", "category": "attack", "technique": "T1046", ...}
{"scope": "event", "parent": "sess-001", "artifact": "edge.pcap", "offset": "pkt:12345", "label": "anomaly", ...}
{"scope": "event", "parent": "sess-001", "artifact": "edge.pcap", "offset": "pkt:12346", "label": "anomaly", ...}

Consumers that only need session-level analysis filter by scope: session. Consumers that need event-level precision read all records.

9.2 New Anomaly Categories

New categories (e.g., misconfiguration, data_leak) can be added to the category field without changing the binary normal/anomaly label system or the JSONL format.