Skip to content

feat(evals): add six-phase synthetic SSDLC benchmark - #40

Merged
arthurpanhku merged 1 commit into
arthurpanhku:mainfrom
reacher-z:feat/ssdlc-six-phase-benchmark
Aug 19, 2026
Merged

feat(evals): add six-phase synthetic SSDLC benchmark#40
arthurpanhku merged 1 commit into
arthurpanhku:mainfrom
reacher-z:feat/ssdlc-six-phase-benchmark

Conversation

@reacher-z

Copy link
Copy Markdown
Contributor

Summary

Progresses #19 with an executable starter benchmark spanning all six SSDLC phases:

  • add six fully synthetic Markdown assessment cases with 12 risk truths and 12 compliance-gap truths
  • validate manifest hashes, phase coverage, source boundaries, truth metadata, and project-local policy mappings
  • add a deterministic one-to-one set scorer with risk/gap precision, recall, F1, severity, policy mapping, evidence locator, and schema-validity metrics
  • route benchmark scorers through the registry and emit a versioned JSON/Markdown scorecard
  • include an explicitly unapproved deterministic oracle baseline for harness validation only

Review boundary

  • All cases are synthetic and contain no real PII or third-party source material.
  • The manifest is marked not_expert_reviewed.
  • The oracle baseline is not a model-performance result.
  • This PR does not claim to complete the M4 requirement for 20–30 cases approved by two security experts; it supplies the validated six-phase foundation for that later review and expansion.

Validation

  • .venv/bin/pytest -q evals/tests — 12 passed
  • .venv/bin/pytest -q — 124 passed
  • .venv/bin/ruff check . — passed
  • .venv/bin/python scripts/export_contracts.py --check — generated contracts current
  • pre-commit hooks on all files included in this PR — passed
  • manifest load — 6 phases, 12 risk truths, 12 compliance-gap truths

AI assistance disclosure

Implementation and initial synthetic fixture drafting were assisted by OpenAI Codex. The contribution includes deterministic validation, mutation-style regression tests, explicit unreviewed boundaries, and no claim of human security-expert review.

Checklist

  • Tests added and passing
  • Documentation added
  • No secrets, real personal data, or third-party benchmark content included
  • AI assistance disclosed per CONTRIBUTING_WITH_AI.md

@arthurpanhku
arthurpanhku merged commit e14bb4c into arthurpanhku:main Aug 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants