feat(evals): add six-phase synthetic SSDLC benchmark

docs/07-evaluation-plan.md

@@ -21,6 +21,12 @@ supported precision/recall/F1, contradiction recall, abstention, citation
 validity, and confusion reporting. This vertical slice supports the public demo
 but does not mark the broader M2 or M5 milestones complete.
 
+A separate six-case `ssdlc_synthetic_v1` starter set now covers all six phases
+with deterministic risk/compliance-gap detection, severity, policy-control,
+evidence-locator, and schema metrics. Its manifest explicitly records
+`not_expert_reviewed`; it validates the data and scoring contracts but does not
+satisfy M4's 20–30 case or two-senior-reviewer requirements.
+
 ---
 
 ## 1. Goals and Non-Goals

evals/README.md

@@ -13,9 +13,14 @@ Current scope:
   precision/recall/F1, contradiction recall, abstention rate, citation validity,
   and a verdict confusion matrix. This is a focused Design-phase addition; it
   does not claim completion of the broader M2 or M5 milestones.
+- **Six-phase synthetic starter set**: one offline, license-clean case for each
+  SSDLC phase, with 24 golden risks and compliance gaps total, stable evidence
+  locators, project-local policy mappings, deterministic one-to-one matching,
+  and set/severity/policy/evidence scorecards.
 
-Raw datasets are not committed. Use the dataset-specific fetch script to download
-public data into ignored local paths.
+External raw datasets are not committed. Use each dataset-specific fetch script
+to download public data into ignored local paths. Fully synthetic fixtures may
+be committed when their manifest and license metadata explicitly permit it.
 
 ```bash
 python evals/datasets/owasp_benchmark/fetch.py
@@ -32,3 +37,19 @@ The runner writes:
 - `evals/reports/<run_id>/scorecard.md`
 
 M1 uses deterministic hard-key CWE triage scoring and does not use an LLM judge.
+
+## Six-phase synthetic starter set
+
+```bash
+python -m evals.runner.run_eval \
+  --dataset-id ssdlc_synthetic_v1 \
+  --raw-dir evals/datasets/ssdlc_synthetic_v1 \
+  --run-id local-six-phase \
+  --repeats 1
+```
+
+The dataset manifest pins every committed fixture and ground-truth file. Its
+`review_status` is deliberately `not_expert_reviewed`: this is a small,
+AI-assisted starter set and scorer contract, not completion of M4's 20–30 case,
+two-expert annotation protocol. The checked-in deterministic oracle baseline
+tests the harness itself and must not be reported as model performance.

evals/adapters/ssdlc_synthetic.py

@@ -0,0 +1,123 @@
+"""Adapter for DocSentinel's six-phase, synthetic SSDLC golden set."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections.abc import Iterable
+from pathlib import Path
+from typing import Any
+
+from evals.models import EvalCase
+
+DATASET_ID = "ssdlc_synthetic_v1"
+CASES_FILE = "cases.jsonl"
+MANIFEST_FILE = "manifest.json"
+PHASES = {
+    "requirements",
+    "design",
+    "development",
+    "testing",
+    "deployment",
+    "operations",
+}
+
+
+def to_cases(raw_dir: Path) -> Iterable[EvalCase]:
+    """Validate dataset integrity and yield normalized six-phase cases."""
+    root = Path(raw_dir).resolve()
+    manifest = _load_json(root / MANIFEST_FILE)
+    _validate_manifest(root, manifest)
+
+    seen_ids: set[str] = set()
+    seen_phases: set[str] = set()
+    with (root / CASES_FILE).open(encoding="utf-8") as handle:
+        for line_number, line in enumerate(handle, 1):
+            if not line.strip():
+                continue
+            try:
+                case = EvalCase.model_validate_json(line)
+            except ValueError as exc:
+                raise ValueError(
+                    f"Invalid {CASES_FILE} line {line_number}: {exc}"
+                ) from exc
+            _validate_case(root, case, seen_ids)
+            seen_ids.add(case.case_id)
+            seen_phases.add(case.phase)
+            yield case
+
+    if seen_phases != PHASES:
+        missing = sorted(PHASES - seen_phases)
+        extra = sorted(seen_phases - PHASES)
+        raise ValueError(
+            "Synthetic dataset must cover all six phases; "
+            f"missing={missing}, extra={extra}"
+        )
+    expected_cases = int(manifest.get("n_cases") or 0)
+    if len(seen_ids) != expected_cases:
+        raise ValueError(
+            f"Manifest declares {expected_cases} cases but {len(seen_ids)} were loaded"
+        )
+
+
+def _validate_case(root: Path, case: EvalCase, seen_ids: set[str]) -> None:
+    if case.dataset_id != DATASET_ID:
+        raise ValueError(f"Unexpected dataset_id for {case.case_id}: {case.dataset_id}")
+    if case.case_id in seen_ids:
+        raise ValueError(f"Duplicate case_id: {case.case_id}")
+    if case.phase not in PHASES:
+        raise ValueError(f"Unsupported phase for {case.case_id}: {case.phase}")
+    if case.skill_id != f"ssdlc-{case.phase}":
+        raise ValueError(
+            f"Skill/phase mismatch for {case.case_id}: {case.skill_id} / {case.phase}"
+        )
+    if not case.inputs:
+        raise ValueError(f"Case has no input documents: {case.case_id}")
+    if not case.ground_truth.risk_items or not case.ground_truth.compliance_gaps:
+        raise ValueError(f"Case has incomplete finding ground truth: {case.case_id}")
+    for item in [
+        *case.ground_truth.risk_items,
+        *case.ground_truth.compliance_gaps,
+    ]:
+        if not item.id or not item.match_terms or not item.evidence_locators:
+            raise ValueError(
+                "Ground truth needs id, match_terms, and evidence_locators: "
+                f"{case.case_id}"
+            )
+    for item in case.inputs:
+        path = (root / item.path).resolve()
+        if root not in path.parents or not path.is_file():
+            raise ValueError(
+                f"Missing or escaping input for {case.case_id}: {item.path}"
+            )
+
+
+def _validate_manifest(root: Path, manifest: dict[str, Any]) -> None:
+    if manifest.get("dataset_id") != DATASET_ID:
+        raise ValueError(
+            f"Unexpected manifest dataset_id: {manifest.get('dataset_id')}"
+        )
+    if manifest.get("provenance") != "fully_synthetic":
+        raise ValueError(
+            "Synthetic dataset manifest must declare fully_synthetic provenance"
+        )
+    if manifest.get("review_status") != "not_expert_reviewed":
+        raise ValueError("Manifest must preserve the not_expert_reviewed boundary")
+    files = manifest.get("files") or {}
+    if not files:
+        raise ValueError("Manifest has no file checksums")
+    for relative, expected in sorted(files.items()):
+        path = (root / relative).resolve()
+        if root not in path.parents or not path.is_file():
+            raise ValueError(
+                f"Manifest path missing or escaping dataset root: {relative}"
+            )
+        actual = hashlib.sha256(path.read_bytes()).hexdigest()
+        if actual != expected:
+            raise ValueError(
+                f"Checksum mismatch for {relative}: {actual} != {expected}"
+            )
+
+
+def _load_json(path: Path) -> dict[str, Any]:
+    return json.loads(path.read_text(encoding="utf-8"))

evals/configs/matrix.yaml

@@ -8,10 +8,13 @@ datasets:
     repeats: 1
     phase: testing
     skill_id: ssdlc-testing
+  ssdlc_synthetic_v1:
+    repeats: 1
+    phase: full_ssdlc
+    skill_id: phase-specific
 
 models:
   - id: ollama-local
     provider: ollama
     model_id: llama2
     temperature: 0.0
-

evals/datasets/ssdlc_synthetic_v1/README.md

@@ -0,0 +1,34 @@
+# Six-Phase Synthetic SSDLC Golden Set v1
+
+This small offline dataset exercises DocSentinel's requirements, design,
+development, testing, deployment, and operations skills. Every organization,
+system, artifact, and finding is fictional. The files contain no copied client
+material, production secrets, or third-party benchmark records and are covered
+by the repository's MIT license.
+
+Each source statement has a stable locator such as `REQ-02` or `DEP-03`.
+`cases.jsonl` records expected risks, severity, project-local policy controls,
+and supporting locators. `manifest.json` pins every committed data file by
+SHA-256 so an evaluation cannot silently run against modified ground truth.
+
+Important review boundary: these AI-assisted fixtures are intentionally marked
+`not_expert_reviewed`. They are a contestable starter set and a deterministic
+scorer/CI contract, not an expert-approved security baseline. The committed
+oracle scorecard proves the harness can recover known truth; it is not model
+performance. Maintainers should review and revise findings before promoting a
+future baseline to `approved`.
+
+Run an actual pipeline evaluation with:
+
+```bash
+python -m evals.runner.run_eval \
+  --dataset-id ssdlc_synthetic_v1 \
+  --raw-dir evals/datasets/ssdlc_synthetic_v1 \
+  --run-id local-six-phase \
+  --repeats 1
+```
+
+The scorer is deterministic: exact IDs match first; otherwise at least 60% of
+an expected finding's declared terms must appear in the prediction. It then
+reports risk and compliance-gap precision/recall/F1 plus severity, policy
+mapping, evidence-locator, and schema fidelity on matched findings.

evals/datasets/ssdlc_synthetic_v1/cases.jsonl

@@ -0,0 +1,6 @@
+{"case_id":"synthetic-requirements-001","dataset_id":"ssdlc_synthetic_v1","phase":"requirements","skill_id":"ssdlc-requirements","inputs":[{"path":"inputs/requirements.md","type":"markdown"}],"ground_truth":{"risk_items":[{"id":"REQ-RISK-MFA","title":"Privileged accounts lack MFA","severity":"high","description":"Password-only administrator authentication increases account-takeover risk.","match_terms":["administrator","password","multi-factor authentication"],"evidence_locators":["REQ-02"]},{"id":"REQ-RISK-RETENTION","title":"Personal data has indefinite retention","severity":"medium","description":"No deletion workflow or retention schedule is defined for tickets and logs.","match_terms":["retained indefinitely","deletion","retention schedule"],"evidence_locators":["REQ-03"]}],"compliance_gaps":[{"id":"REQ-GAP-MFA","framework":"generic-ssdlc","control_or_clause":"GEN-IAM-01","gap_description":"Strong authentication is not required for administrator access.","match_terms":["strong authentication","administrator","multi-factor"],"evidence_locators":["REQ-02"]},{"id":"REQ-GAP-RETENTION","framework":"generic-ssdlc","control_or_clause":"GEN-PRV-02","gap_description":"The requirements omit a data-retention and deletion policy.","match_terms":["data retention","deletion","schedule"],"evidence_locators":["REQ-03"]}]},"meta":{"license":"MIT","provenance":"fully_synthetic","review_status":"not_expert_reviewed","annotators":0,"iaa":null}}
+{"case_id":"synthetic-design-001","dataset_id":"ssdlc_synthetic_v1","phase":"design","skill_id":"ssdlc-design","inputs":[{"path":"inputs/design.md","type":"markdown"}],"ground_truth":{"risk_items":[{"id":"DES-RISK-IDENTITY","title":"Client-controlled identity header enables impersonation","severity":"high","description":"The API trusts an unsigned browser-supplied user identifier as account identity.","match_terms":["X-User-Id","account identity","without a signed session"],"evidence_locators":["DES-01"]},{"id":"DES-RISK-TRANSPORT","title":"Sensitive metadata crosses an unencrypted service hop","severity":"high","description":"Customer document metadata is sent to the indexer over plaintext HTTP.","match_terms":["unencrypted HTTP","customer names","indexing service"],"evidence_locators":["DES-02"]}],"compliance_gaps":[{"id":"DES-GAP-IDENTITY","framework":"generic-ssdlc","control_or_clause":"GEN-IAM-01","gap_description":"The design lacks a trusted authentication mechanism for the document API.","match_terms":["authentication","trusted","document API"],"evidence_locators":["DES-01"]},{"id":"DES-GAP-TRANSPORT","framework":"generic-ssdlc","control_or_clause":"GEN-ENC-01","gap_description":"Encryption in transit is missing between the API and indexing service.","match_terms":["encryption in transit","API","indexing service"],"evidence_locators":["DES-02"]}]},"meta":{"license":"MIT","provenance":"fully_synthetic","review_status":"not_expert_reviewed","annotators":0,"iaa":null}}
+{"case_id":"synthetic-development-001","dataset_id":"ssdlc_synthetic_v1","phase":"development","skill_id":"ssdlc-development","inputs":[{"path":"inputs/development.md","type":"markdown"}],"ground_truth":{"risk_items":[{"id":"DEV-RISK-SQLI","title":"Raw search input is interpolated into SQL","severity":"critical","description":"Direct query-string interpolation creates SQL injection risk.","match_terms":["SQL","interpolating","query parameter"],"evidence_locators":["DEV-01"]},{"id":"DEV-RISK-CVE","title":"Deployed dependencies lack continuous vulnerability monitoring","severity":"high","description":"Quarterly manual review leaves newly disclosed dependency CVEs undetected.","match_terms":["quarterly","continuous alerting","CVE"],"evidence_locators":["DEV-02"]}],"compliance_gaps":[{"id":"DEV-GAP-SQLI","framework":"generic-ssdlc","control_or_clause":"GEN-SEC-01","gap_description":"The search handler does not validate input or use a parameterized query.","match_terms":["search handler","input","parameterized"],"evidence_locators":["DEV-01"]},{"id":"DEV-GAP-CVE","framework":"generic-ssdlc","control_or_clause":"GEN-SC-03","gap_description":"Continuous dependency vulnerability monitoring is absent.","match_terms":["dependency vulnerability","continuous","monitoring"],"evidence_locators":["DEV-02"]}]},"meta":{"license":"MIT","provenance":"fully_synthetic","review_status":"not_expert_reviewed","annotators":0,"iaa":null}}
+{"case_id":"synthetic-testing-001","dataset_id":"ssdlc_synthetic_v1","phase":"testing","skill_id":"ssdlc-testing","inputs":[{"path":"inputs/testing.md","type":"markdown"}],"ground_truth":{"risk_items":[{"id":"TST-RISK-COVERAGE","title":"DAST excludes the privileged administration surface","severity":"high","description":"Unauthenticated scanning leaves the entire admin API outside dynamic testing.","match_terms":["DAST","admin API","excludes"],"evidence_locators":["TST-01"]},{"id":"TST-RISK-BYPASS","title":"Critical authentication bypass is unretested before release","severity":"critical","description":"A known administrator bypass remains open and verification is deferred until after launch.","match_terms":["authentication bypass","critical","retest"],"evidence_locators":["TST-02"]}],"compliance_gaps":[{"id":"TST-GAP-COVERAGE","framework":"generic-ssdlc","control_or_clause":"GEN-VUL-02","gap_description":"Dynamic testing does not cover authenticated administration endpoints.","match_terms":["dynamic testing","administration","authenticated"],"evidence_locators":["TST-01"]},{"id":"TST-GAP-BYPASS","framework":"generic-ssdlc","control_or_clause":"GEN-VUL-01","gap_description":"The critical authentication finding lacks pre-release remediation verification.","match_terms":["critical","authentication finding","verification"],"evidence_locators":["TST-02"]}]},"meta":{"license":"MIT","provenance":"fully_synthetic","review_status":"not_expert_reviewed","annotators":0,"iaa":null}}
+{"case_id":"synthetic-deployment-001","dataset_id":"ssdlc_synthetic_v1","phase":"deployment","skill_id":"ssdlc-deployment","inputs":[{"path":"inputs/deployment.md","type":"markdown"}],"ground_truth":{"risk_items":[{"id":"DEP-RISK-IMAGE","title":"Production image is released without vulnerability scanning","severity":"high","description":"Unknown container and infrastructure-image vulnerabilities can reach production.","match_terms":["release image","without","vulnerability scan"],"evidence_locators":["DEP-01"]},{"id":"DEP-RISK-RESTORE","title":"Backups have no demonstrated restore capability","severity":"high","description":"Untested restores and undefined recovery objectives make recovery uncertain.","match_terms":["backups","never restored","recovery"],"evidence_locators":["DEP-02"]}],"compliance_gaps":[{"id":"DEP-GAP-IMAGE","framework":"generic-ssdlc","control_or_clause":"GEN-VUL-03","gap_description":"Container and infrastructure images are not scanned before promotion.","match_terms":["container","images","scanned"],"evidence_locators":["DEP-01"]},{"id":"DEP-GAP-RESTORE","framework":"generic-ssdlc","control_or_clause":"GEN-BCP-01","gap_description":"Backup recovery is not tested and recovery objectives are undefined.","match_terms":["backup","recovery","tested"],"evidence_locators":["DEP-02"]}]},"meta":{"license":"MIT","provenance":"fully_synthetic","review_status":"not_expert_reviewed","annotators":0,"iaa":null}}
+{"case_id":"synthetic-operations-001","dataset_id":"ssdlc_synthetic_v1","phase":"operations","skill_id":"ssdlc-operations","inputs":[{"path":"inputs/operations.md","type":"markdown"}],"ground_truth":{"risk_items":[{"id":"OPS-RISK-LOGS","title":"Short-lived local logs cannot support durable incident investigation","severity":"high","description":"Node-local security logs are overwritten after one day and lack tamper resistance.","match_terms":["security logs","overwritten","tamper-resistant"],"evidence_locators":["OPS-01"]},{"id":"OPS-RISK-PATCH","title":"Critical dependency advisories have no remediation SLA","severity":"high","description":"Quarterly backlog review allows critical dependency exposure to persist.","match_terms":["critical dependency","quarter","remediation deadline"],"evidence_locators":["OPS-02"]}],"compliance_gaps":[{"id":"OPS-GAP-LOGS","framework":"generic-ssdlc","control_or_clause":"GEN-LOG-02","gap_description":"Log retention and tamper-resistant centralized storage are insufficient.","match_terms":["log retention","tamper-resistant","centralized"],"evidence_locators":["OPS-01"]},{"id":"OPS-GAP-PATCH","framework":"generic-ssdlc","control_or_clause":"GEN-SC-03","gap_description":"Critical dependency vulnerabilities have no defined remediation deadline.","match_terms":["critical dependency","remediation","deadline"],"evidence_locators":["OPS-02"]}]},"meta":{"license":"MIT","provenance":"fully_synthetic","review_status":"not_expert_reviewed","annotators":0,"iaa":null}}

evals/datasets/ssdlc_synthetic_v1/inputs/deployment.md

@@ -0,0 +1,13 @@
+# Deployment Review: ExampleCo Reporting Service
+
+[DEP-01] The release image is promoted to production without a container or
+infrastructure image vulnerability scan.
+
+[DEP-02] Database backups are created nightly, but the team has never restored
+one and has not measured recovery time or recovery-point objectives.
+
+[DEP-03] Production credentials are injected from a managed secret store and
+are absent from the image and deployment manifest.
+
+[DEP-04] The service runs as a non-root user with a read-only root filesystem
+and drops all Linux capabilities not required at runtime.

evals/datasets/ssdlc_synthetic_v1/inputs/design.md

@@ -0,0 +1,14 @@
+# Design Review: ExampleCo Document Service
+
+[DES-01] A browser sends `X-User-Id` directly to the document API. The API uses
+that value as the account identity without a signed session or trusted gateway.
+
+[DES-02] The API sends document metadata to an internal indexing service over
+unencrypted HTTP on the cluster network. Metadata includes customer names and
+document titles.
+
+[DES-03] Object storage denies public access, and each document is encrypted at
+rest with a managed key whose rotation is enabled.
+
+[DES-04] Authorization checks bind each document ID to the authenticated tenant
+before object storage is queried.

evals/datasets/ssdlc_synthetic_v1/inputs/development.md

@@ -0,0 +1,14 @@
+# Development Review: ExampleCo Search API
+
+[DEV-01] The search handler builds a SQL statement by interpolating the raw
+`query` request parameter into the statement text before execution.
+
+[DEV-02] Dependency versions are pinned, but vulnerability review is a manual
+quarterly spreadsheet exercise. There is no continuous alerting for newly
+disclosed CVEs affecting versions already deployed.
+
+[DEV-03] The repository blocks commits containing known credential formats and
+uses short-lived workload identity for CI publishing.
+
+[DEV-04] Unit tests cover authorization failures and output encoding for search
+results.

evals/datasets/ssdlc_synthetic_v1/inputs/operations.md

@@ -0,0 +1,13 @@
+# Operations Review: ExampleCo Customer Platform
+
+[OPS-01] Security logs remain only on each application node and are overwritten
+after 24 hours. They are not forwarded to tamper-resistant centralized storage.
+
+[OPS-02] Critical dependency advisories are collected, but the backlog is
+reviewed only once per quarter and no remediation deadline is defined.
+
+[OPS-03] The on-call rotation is staffed continuously and the incident runbook
+contains tested escalation contacts and customer-notification templates.
+
+[OPS-04] Administrative sessions are revoked when an employee leaves the
+organization.