Skip to content

feat: complete benchmark MVP infrastructure - #1

Merged
bjcorder merged 1 commit into
mainfrom
feat/complete-benchmark-mvp
May 25, 2026
Merged

feat: complete benchmark MVP infrastructure#1
bjcorder merged 1 commit into
mainfrom
feat/complete-benchmark-mvp

Conversation

@bjcorder

Copy link
Copy Markdown
Contributor

Summary

  • Add provider runners and suite run harness for echo, OpenAI-compatible, OpenRouter, and Anthropic APIs
  • Add public-dev task pack across authz, supply-chain, threat-modeling, and LLM appsec
  • Add deterministic task generators with canaries and suite validation commands
  • Add fixture semantic judge calibration flow and public calibration fixtures
  • Add sealed suite pack/verify/unpack infrastructure with path-safety protections
  • Expand documentation for providers, generators, semantic judging, and sealed suites

Validation

  • ruff check .
  • PYTHONPATH=src pytest -q
  • prodsecbench validate-suite tasks/public-dev
  • prodsecbench run-suite tasks/public-dev --provider echo --output /tmp/prodsecbench-public-dev-run.json
  • prodsecbench calibrate-judge calibration/public/authz-idor-001 --output /tmp/prodsecbench-calibration.json
  • prodsecbench seal-suite tasks/public-dev /tmp/prodsecbench-public-dev.psbpack --suite-id public-dev-smoke
  • prodsecbench verify-sealed-pack /tmp/prodsecbench-public-dev.psbpack /tmp/prodsecbench-public-dev.manifest.json

Security notes

  • Task artifact and prompt paths now reject absolute paths and parent traversal before file reads
  • Sealed pack unpacking uses resolved path containment checks and rejects archive links

@bjcorder
bjcorder merged commit b23e3fd into main May 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant