CYBRA runs eight independent LLM security benchmarks against any OpenAI-compatible model and consolidates the results into one report with an explanation behind every number — no CLI flags, no hand-editing JSONL.
Built for the security engineer who owns the model, not for the researcher who writes the harness: add an endpoint, pick the models, press Run, get a report you can hand to leadership.
| Layer | Responsibility |
|---|---|
security_suite/ |
Evaluation core: 8 adapters, judge, relay with a request budget. Pre-existing, already-validated code. |
scan/ |
Supervisor: spawns a run, tracks progress, cancels, detects available tools. |
scan/ surface |
Exposes capabilities so the UI can show an honest "not installed" state instead of failing mid-run. |
backend/ |
FastAPI + SQLite: auth, CRUD, live progress, dataset versioning, 11 metrics. |
frontend/ |
React SPA: dark console and a light printable report. |
Benchmarks: garak, Promptfoo, PyRIT, HarmBench, JailbreakBench, BIPIA, AgentDojo, Security Regression.
Metrics: 11 — see Metrics below.
docker build -f docker/Dockerfile -t cybra/cybra .
docker run -p 8000:8000 -v cybra-data:/data cybra/cybraOpen http://localhost:8000. On first start the admin login and password are printed to the log — change them in Settings.
The image is two-stage: Node builds the SPA, then it is copied into the Python
stage and served by the same process. One image, one port — no CORS, no second
container, no reverse proxy. Dependencies are installed from pyproject.toml
before the source is copied, so editing backend/cybra/*.py does not trigger
a re-download of fastapi/uvicorn. The package is not installed with -e:
PYTHONPATH=/app/backend points at the same place the editable install would,
and config.root (parents[2] from cybra/config.py, i.e. /app) keeps
finding datasets/ and var/.
docker compose -f docker/compose.prod.yaml up -d # pulls, never builds; needs docker/.envpython -m venv .venv && .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e backend
cd frontend && npm install && npm run build && cd ..
python -m cybra --host 127.0.0.1 --port 8000UI development with hot reload: cd frontend && npm run dev (proxies /api
to port 8000).
No real API and no heavy tooling required:
python -m cybra --engine mockRuns against a built-in fixture. Verdicts are synthetic and the UI and reports are marked accordingly.
garak, PyRIT, AgentDojo and Promptfoo are not bundled into the image or
the exe: they pull torch or a Node binary measured in gigabytes. CYBRA detects
what is present and shows an honest status in the UI instead of crashing
mid-run.
python -m cybra setup --tools garak pyrit agentdojo promptfooEach benchmark installs into an isolated .tools/<name> environment. Install
logs are kept and re-running is idempotent.
POST /api/v1/runswith a profile, models and tool selection.scan/spawnsrun.pyas a child process and reads its file contract.- Progress is published at
GET /runs/{id}/progress(polled by the UI). - On completion
summary.jsonandrecords.jsonlare imported into the database. - The UI renders the model × benchmark matrix and a printable report.
The evaluation core is deliberately not rewritten: the judge and PASS/FAIL criteria have already been validated against real models, and reimplementing them would change what the verdicts mean.
Two editing paths, as required:
- through the UI — edit a case in the dataset workspace;
- by configuration — upload
jsonl/json/yaml/csv; keep the file in git next to the repository.
Versions are immutable. Any edit creates a new version pointing at its parent, so an old run is always reproducible. Version diff and export are included.
Eleven metrics, each with a formula, a meaning and an explicit caveat:
| What it shows | Metric |
|---|---|
| Attack success | attack_success_rate |
| Share of failed checks | failed_check_rate |
| Over-refusals | refusal_rate |
| Utility on benign input | benign_utility_rate |
| Following injected instructions | injection_follow_rate |
| Secret leakage | leak_rate |
| Not scored (report confidence) | unscored_rate |
| Completeness | coverage |
| Safety drop under attack wrappers | resilience_gap |
| Utility under injection | utility_under_attack |
| Blocked legitimate agent actions | agent_false_block_rate |
The full vocabulary is in docs/METHODOLOGY.md and docs/metrics.md.
No single "security score" is computed. For garak the unit is a detector judgement; for the others it is an individual case. Combining them into one number would present incomparable quantities as comparable.
These are two different counts and both are correct:
- 8 benchmarks are the tools that generate test cases and verdicts
(
security_suite/common.py, constantTOOLS); - 11 metrics are the formulas computed over those verdicts
(
backend/cybra/metrics.py, constantCATALOG).
Six of the eleven are different filters over the same PASS/FAIL stream; only
resilience_gap, utility_under_attack and agent_false_block_rate add
independent signal.
| Role | Permissions |
|---|---|
viewer |
reads runs and reports |
operator |
starts runs, edits endpoints and datasets |
admin |
manages users, deletes endpoints |
Passwords use argon2, sessions live in HttpOnly cookies, mutating requests carry a CSRF token, endpoint keys are Fernet-encrypted and never returned by the API.
pip install pytest httpx ruff
pytest tests/ -q
ruff check backend scan40 tests: API (test_api.py), report core (test_core.py), scan layer
(test_scan.py), metrics (test_metrics_new.py). test_api.py needs no API
key: it uses the mock engine and covers auth, CSRF, dataset versioning, a full
run and metrics.
CI runs tests and the linter on every push to main and on pull requests:
| Config | Where | What it does |
|---|---|---|
.github/workflows/ci.yml |
GitHub Actions | lint, tests, image build + smoke |
.gitlab-ci.yml |
GitLab CI | lint, tests, image build, push to registry, smoke |
The GitHub pipeline is the public mirror: it proves the code is sound but
throws the image away. The GitLab one is the delivery path — it publishes
ap3x0/cybra:<commit-sha> and :latest, and its verify
stage pulls that pushed image back and waits for /api/v1/health. An image
counts as delivered only if it actually starts.
Deploying that artefact on any machine:
docker pull ap3x0/cybra:latest
docker run -p 8000:8000 -v cybra-data:/data \
-e CYBRA_SECRET_KEY="<generate: 32 random bytes, urlsafe base64>" \
ap3x0/cybra:latestGenerate the key with:
python -c "import base64,os;print(base64.urlsafe_b64encode(os.urandom(32)).decode())"| File | Contents |
|---|---|
| docs/METHODOLOGY.md | Full methodology: properties, benchmarks, 11 metrics, verdicts, limits |
| docs/SUITE.md | Running the suite without the console: CLI, profiles, artifacts |
| docs/metrics.md | Metric dictionary with formulas |
| docs/metrics-research.md | Research behind the safety and availability metrics |
| docs/DATA_SOURCES.md | Data sources and their licences |
| docs/VALIDATION-CYBRA.md | What is verified and what is not |
| docs/DEPLOYMENT.md | Delivery and deployment: CI/CD, image, environment, state |
| examples/README.md | Endpoint configuration examples |
Russian documentation: docs/ru/METHODOLOGY.md and friends in the same folder.
The report does not cover ACLs, tenant isolation, authorisation bypass, real RAG
indexing, browsing or external side effects. COMPLETE refers to the selected
profile, not to every attack a benchmark offers, and is not a security
certificate.
Copyright 2026 Ap3x0s. Licensed under Apache-2.0.
Third-party data and tools — see licenses/ and
docs/DATA_SOURCES.md.








