Build an open SSDLC assessment benchmark

[arthurpanhku]

Why this matters

DocSentinel can only be trusted if its security review output can be measured against repeatable ground truth. Today, SSDLC document review is hard to compare across LLM providers, prompts, retrieval settings, and policy packs because there is no shared benchmark.

From first principles: if the project cannot tell whether a change improves recall, precision, citation quality, and false-positive rate, the community cannot safely optimize the system.

Community help wanted

We need a public, synthetic, license-clean benchmark for SSDLC assessments across requirements, design, development, testing, deployment, and operations.

Suggested scope

• Create synthetic but realistic input documents for each SSDLC phase.
• Define expected findings, severity, policy mappings, and evidence locations.
• Add an evaluation runner that can compare structured assessment output against expected results.
• Track metrics such as finding recall, false positives, evidence citation accuracy, schema validity, and runtime/cost.
• Include provider-neutral fixtures that can run without proprietary or confidential data.

Acceptance criteria

• A documented benchmark dataset exists under a clearly licensed path.
• The benchmark can run in CI or locally with deterministic fixtures.
• Results are emitted as machine-readable JSON and a human-readable summary.
• At least one baseline report is checked in for regression comparison.

Progress (2026-08-18)

• Add a reproducible evaluation harness with normalized cases and a service-level runner.
• Add an OWASP Benchmark adapter with deterministic, provider-neutral fixtures and hard-key CWE triage scoring.
• Emit machine-readable scorecard.json and human-readable scorecard.md output.
• Add tested synthetic indirect prompt-injection documents for offline guardrail regression.
• Cover all six SSDLC phases with license-clean synthetic documents and golden findings.
• Define expert-reviewed expected evidence locations, severity, and policy mappings.
• Check in an approved baseline report and enforce regression thresholds in CI.
• Expand metrics to citation fidelity, schema validity, runtime/cost, and false-positive rate across the full benchmark.

[arthurpanhku]

Roadmap update: benchmark is now a core citation asset

The six-phase synthetic starter set has landed, so the remaining milestone is no longer simply “cover all phases.” The next benchmark release should be coordinated by maintainers while inviting expert annotation and methodology review from the community.

Remaining target:

• 20–30 enterprise-style cases across all six phases;
• at least two senior security annotators with inter-annotator agreement;
• public license-clean set plus a private hold-out;
• model matrix with repeated runs and variance;
• critical-finding recall, false-positive rate, citation precision, calibration, consistency, latency, and cost;
• approved baselines and CI regression thresholds;
• one-command reproduction of the published scorecard.

Related roadmap issues: #43 Evidence Envelope, #44 SARIF, #45 SBOM, #46 SLSA verification, and #49 cross-phase traceability.