Repository navigation
test(cyberbattlesim): compose RAES conformance with simulator-specific probes #28
Description
Activity
🛠️ Picked up by /implement - driver codex, branch
28-cyberbattlesim-conformance-probes, 2026-08-01T05:05:00.733Z.gc workflow phase recorded:
preflight(issue #28). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.gc workflow phase recorded:
plan(issue #28). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Plan
Implement issue #28 as adapter-local conformance composition around the published RAES runner, not as a second conformance authority.
- Add focused failing tests first in
tests/test_cyberbattlesim_conformance.pythat prove:run_cyberbattlesim_conformance()returns the exact publishedBackendConformanceReportfrom the existing base wrapper andcyberbattlesim_backend_conformance_payload()serializes it throughbackend_conformance_report_payload().- CyberBattleSim source-protocol diagnostics validate through
diagnostic_model()and use JSON-pointer addresses. - Manifest capability evidence is derived from
backend_manifest_payload()JSON-pointer surfaces and fails closed when a new affirmative surface lacks evidence. - Declared weaknesses come from the existing qualification/loss-disclosure artifacts, not a copied table.
- Portable projections for hostile failure paths on provisioner, orchestrator, participant runtime, evaluator, and cleanup do not leak native sentinels through actual model payloads.
- Add
src/raes_adapters/cyberbattlesim/backend/conformance.pyas a small adapter-local composition module. It will reusecreate_cyberbattlesim_target(),run_conformance_probe(),backend_conformance_report_payload(),backend_manifest_payload(),diagnostic_model(),load_qualification(),load_loss_disclosures(), andvalidate_all(). It will not define a report DTO, copy fixtures/profiles, append report cases, or restate manifest capability values. - Fix CyberBattleSim backend diagnostics whose addresses are currently dotted identifiers so every emitted RAES diagnostic can pass the published
DiagnosticModelJSON-pointer shape. Keep messages bounded and input-free; do not include native exception text, paths, arguments, or tracebacks. - Export only the new conformance helpers from
raes_adapters.cyberbattlesim.backend.__init__and update README docs with the canonical usage and claim separation. - Extend the existing
_distributions()clean-install boundary innoxfile.pyto install the built wheel with thecyberbattlesimextra in a throwaway environment, run from isolated Python withPYTHONPATHcleared and safe-path behavior enabled, construct the target with an injected deterministic driver, run the canonical conformance payload projection, and prove packaged CyberBattleSim evidence resources are present. - Run targeted verification first (
pytestfor the new/constrained tests andtool-testsif nox wiring is touched), then the workflow’s mandatory completion gates.
Design Checks
Security: no authentication, HTTP, CLI, subprocess simulator, network fetch, secret-store input, or environment override is added. Failure probes will validate actual portable payloads and RAES diagnostics, not
str()of native objects. Native sentinels stay inside test drivers and must be absent fromApplyResult, snapshots, observations, evidence, cleanup receipts, diagnostics, and conformance payloads.Maintainability: the change builds on the existing CyberBattleSim backend,
scenario_ledgervalidators, base conformance helper, RAES serializers, nox distribution helper shape, and preflight guardrails. No new base authority, profile registry, fixture corpus, exception hierarchy, schema, report DTO, or workflow is introduced.Extensibility: the seam remains explicit
EvidenceSelectionplus injected driver/suite tier. A future CyberBattleSim selection can reuse the conformance composition without a cross-simulator registry or RAES corpus change.Whole-repo view: affected surfaces are
src/raes_adapters/cyberbattlesim/backend/, tests, README,noxfile.py, and the preflight docs already added underdocs/decisions/,docs/index.md, andmkdocs.yml. Packaging remains one distribution, oneuv.lock, isolated extras, and Release Please-owned changelog/versioning.Documentation coverage:
gc_documentation_coveragereturnedoutcome_required=false; README will still be updated because the issue adds user-facing conformance usage.Conventional Commit PR title target:
test(cyberbattlesim): compose conformance probes.- Add focused failing tests first in
gc_codex_review — cycle 1 of 1 (pre-push) on issue #28 (branch
28-cyberbattlesim-conformance-probes)
Diff mode: inline — the complete diff was supplied in one promptCore review
Verdict:
don't-shipThe change chooses the correct high-level seam: canonical backend conformance remains a RAES-owned report and projection, while adapter-local source evidence stays separate and diagnostics are normalized at the backend boundary. The clean-install lane also exercises the packaged composition appropriately. However, the capability-to-evidence closure—the central new adapter-owned concern—is represented by static reference strings rather than results from executable probes. That breaks the promised fail-closed boundary and makes the production API claim evidence that the code has not established.
Blocking findings (1):
- [class] Capability closure is disconnected from executable evidence —
src/raes_adapters/cyberbattlesim/backend/conformance.py:40
Each capability surface is assigned hard-coded evidence-reference strings, andcyberbattlesim_manifest_capability_evidence_gaps()checks only whether the surface name exists in this table. It therefore reports complete coverage even when the referenced probes never ran or failed, and a new affirmative capability nested beneath an already-known surface also remains invisible. No producer or result join for these references appears in the diff; the tests merely assert that the tuples are non-empty. Derive affirmative capability addresses at the actual manifest capability granularity and compute coverage from passing executable probe results, so absent or failed evidence produces a gap.
Security review
Verdict:
shipThis change is shaped correctly: it composes the published RAES conformance runner and serializer with adapter-local evidence, normalizes diagnostics at the existing portable boundary, and verifies the packaged wheel using a static isolated probe. The security-relevant seams—native exception containment, portable serialization, source-ledger validation, and clean-install process isolation—remain with their established owners. No new network, authentication, persistence, dynamic-import, shell-interpolation, secret-handling, or user-controlled filesystem boundary is introduced, and the design leaves the next qualified selection injectable without widening authority.
No blocking findings.
- [class] Capability closure is disconnected from executable evidence —
gc_codex_review pre-push cycle 1 of 1 complete for issue #28 on branch '28-cyberbattlesim-conformance-probes'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Review decision record — codex cycle 1 (issue #28)
Reviewer: codex
Cycle: 1Architectural read:
Core reviewer: The change chooses the correct high-level seam: canonical backend conformance remains a RAES-owned report and projection, while adapter-local source evidence stays separate and diagnostics are normalized at the backend boundary. The clean-install lane also exercises the packaged composition appropriately. However, the capability-to-evidence closure—the central new adapter-owned concern—is represented by static reference strings rather than results from executable probes. That breaks the promised fail-closed boundary and makes the production API claim evidence that the code has not established.
Security reviewer: This change is shaped correctly: it composes the published RAES conformance runner and serializer with adapter-local evidence, normalizes diagnostics at the existing portable boundary, and verifies the packaged wheel using a static isolated probe. The security-relevant seams—native exception containment, portable serialization, source-ledger validation, and clean-install process isolation—remain with their established owners. No new network, authentication, persistence, dynamic-import, shell-interpolation, secret-handling, or user-controlled filesystem boundary is introduced, and the design leaves the next qualified selection injectable without widening authority.
Blocking findings: 1
Finding 1 —
class(5 instances)- ID:
F1 - Title: [core] Capability closure is disconnected from executable evidence
- Location:
src/raes_adapters/cyberbattlesim/backend/conformance.py:40 - Decision: fix
- Rationale: Each capability surface is assigned hard-coded evidence-reference strings, and
cyberbattlesim_manifest_capability_evidence_gaps()checks only whether the surface name exists in this table. It therefore reports complete coverage even when… - Instances:
src/raes_adapters/cyberbattlesim/backend/conformance.py:40src/raes_adapters/cyberbattlesim/backend/conformance.py:45src/raes_adapters/cyberbattlesim/backend/conformance.py:50src/raes_adapters/cyberbattlesim/backend/conformance.py:55src/raes_adapters/cyberbattlesim/backend/conformance.py:61
- ID:
gc_codex_review — cycle 2 of 1 (pre-push) on issue #28 (branch
28-cyberbattlesim-conformance-probes)
Diff mode: inline — the complete diff was supplied in one promptCore review
Verdict:
ship-with-fixesThe change is mostly shaped along the right seam: canonical RAES backend conformance stays delegated to run_conformance_probe/backend_conformance_report_payload, while CyberBattleSim-local source protocol evidence remains adapter-owned diagnostics and README/docs/nox wiring stays inside the backend-specific boundary. The cross-cutting concern it touches is capability-to-evidence closure, and that is the one place where the implementation weakens the intended design: the evidence join is derived from manifest pointers, but not from a stable probe inventory, so it can bless new affirmative capability claims with old generic evidence. That forecloses the obvious next variation, adding a new manifest capability and requiring a matching executable probe.
Blocking findings (1):
- [one-off] Capability evidence join blesses every affirmative capability with generic evidence —
src/raes_adapters/cyberbattlesim/backend/conformance.py:81
cyberbattlesim_manifest_capability_evidence assigns the same passed_evidence_refs tuple to every affirmative capability pointer. If the manifest later adds payload["capabilities"]["provisioner"]["supports_new_mode"] = True and the existing canonical report/source diagnostics pass, cyberbattlesim_manifest_capability_evidence_gaps will report no gap even though no probe is mapped to that new capability. That violates the guardrail's fail-closed capability-to-evidence closure and turns the local inventory into a blanket pass. The fix should join manifest-derived pointers against an explicit adapter-local probe/evidence inventory and only mark capabilities covered by matching passing evidence.
Security review
Verdict:
shipThis change is shaped as adapter-local conformance composition around the published RAES report/projector boundary, plus documentation and clean-install verification. The cross-cutting security concern is portable-output hygiene: diagnostics and report serialization stay on the RAES model/projector path, and the touched failure surfaces move diagnostic addresses toward JSON Pointer validation without adding HTTP, subprocess, credential, filesystem-input, or raw-query surfaces. I do not see a concrete exploitable security regression in the provided diff.
No blocking findings.
- [one-off] Capability evidence join blesses every affirmative capability with generic evidence —
gc_codex_review pre-push cycle 2 (USER-AUTHORIZED OVERRIDE past cap 1) complete for issue #28 on branch '28-cyberbattlesim-conformance-probes'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.
Override reason: user said: overcap authorized.Review decision record — codex cycle 2 (issue #28)
Reviewer: codex
Cycle: 2Architectural read:
Core reviewer: The change is mostly shaped along the right seam: canonical RAES backend conformance stays delegated to run_conformance_probe/backend_conformance_report_payload, while CyberBattleSim-local source protocol evidence remains adapter-owned diagnostics and README/docs/nox wiring stays inside the backend-specific boundary. The cross-cutting concern it touches is capability-to-evidence closure, and that is the one place where the implementation weakens the intended design: the evidence join is derived from manifest pointers, but not from a stable probe inventory, so it can bless new affirmative capability claims with old generic evidence. That forecloses the obvious next variation, adding a new manifest capability and requiring a matching executable probe.
Security reviewer: This change is shaped as adapter-local conformance composition around the published RAES report/projector boundary, plus documentation and clean-install verification. The cross-cutting security concern is portable-output hygiene: diagnostics and report serialization stay on the RAES model/projector path, and the touched failure surfaces move diagnostic addresses toward JSON Pointer validation without adding HTTP, subprocess, credential, filesystem-input, or raw-query surfaces. I do not see a concrete exploitable security regression in the provided diff.
Blocking findings: 1
Finding 1 —
one-off- ID:
F1 - Title: [core] Capability evidence join blesses every affirmative capability with generic evidence
- Location:
src/raes_adapters/cyberbattlesim/backend/conformance.py:81 - Decision: fix
- Rationale: cyberbattlesim_manifest_capability_evidence assigns the same passed_evidence_refs tuple to every affirmative capability pointer. If the manifest later adds payload["capabilities"]["provisioner"]["supports_new_mode"] = True and the existing…
- ID:
gc_test_quality_review cycle 1 of 1 — issue #28
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:28-cyberbattlesim-conformance-probes
Cycle: 1 / 1
Findings: 1Finding 1 — [critical]
tests/test_cyberbattlesim_conformance.py::test_failure_surface_diagnostics_validate_and_do_not_leak_native_sentinelsProblem: The test drives five different failure paths (provisioner construct-failure, orchestrator unsupported-resource, participant-runtime step-failure, evaluator projection-failure, and cleanup close-failure) but only asserts
SENTINEL not in _json_text(payloads). It never asserts that any of the underlying operations actually failed — noassert not provisioning.success,assert not evaluation.success,assert participant.action_result.status in {"rejected", "failed"}, orassert cleanup.cleanup_status == "failed". Since none of the well-behaved (non-failure) code paths ever emit SENTINEL either, a regression that silently swallows the driver exception and reports fabricated success (fail-open instead of fail-closed) would leave the assembled payload free of SENTINEL and this test would still pass.
Why it matters: This is exactly the security/audit-integrity property the test name claims to cover: native driver failures must be bounded into a diagnosed failure, not swallowed into a false success. A regression that catches the exception but forgets to set success=False (or forgets to add the diagnostic) is a fail-open bug in production adapter code, and this test provides no signal for it — it only detects the orthogonal failure mode (leaking exception text).
Fix: Add the same pairing of assertions used in the sibling suitetests/test_cyberbattlesim_backend.py(e.g. lines 625/675/696/889-890:assert not result.success,assert failed.action_result.status == "failed",assert failed.obligation_results[...].status == "failed") for each of the five failure calls in this test: assertprovisioning.success is False,orchestration.success is False,participant.apply_result.success is Falseandparticipant.action_result.statusis a failure status,evaluation.success is False, andcleanup.cleanup_status == "failed"(or the equivalent obligation-result status), in addition to the existing no-leak check.gc_test_quality_review cycle 1 of 1 complete for issue #28 on branch '28-cyberbattlesim-conformance-probes'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Review decision record — test-quality cycle 1 (issue #28)
Reviewer: test-quality
Cycle: 1Architectural read:
This PR adds a conformance-composition module (
conformance.py) that is a thin, well-scoped projector over already-published RAES report/manifest shapes plus a capability-evidence "fails closed" derivation — a sensible seam that keeps this backend's declared capabilities honestly tied to probe evidence rather than to unverified self-declaration. The test file mirrors the existingtest_cyberbattlesim_backend.pyfixture shape (a fake driver withfail_*flags) reasonably well for three of its four tests, which genuinely exercise real production code paths (evaluator/orchestrator/provisioner/participant-runtime/manifest-evidence-gap logic) rather than mocks. The one structural problem is in the newest and most security-relevant test,test_failure_surface_diagnostics_validate_and_do_not_leak_native_sentinels: it collects failure-path output from five different subsystems but only asserts the negative property (no leaked sentinel string), never the positive one (that each subsystem actually reported failure). The sibling file this test's own fixture was copied from (test_cyberbattlesim_backend.py) always pairs those two assertions; this one drops the half that would catch a fail-open regression.Blocking findings: 1
Finding 1 —
one-off- ID:
F1 - Title: (no title)
- Decision: fix
- Rationale: Addressed by next cycle
- ID:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devat1dd255a92ab7b68c76c27b1637661fea7cee6518 - Outcome:
already_current - Published feature head:
daaf8791fb3df6337325b6c07905001e5332286c - Verified tree:
28850a0c7f4a4679777311fb4e6da5a84637c6be
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devat1dd255a92ab7b68c76c27b1637661fea7cee6518 - Outcome:
already_current - Published feature head:
3cadef7b4b3772820679a170e2da920fad2c8139 - Verified tree:
fb3416328296316670c5c26de89a4bdd3cc3623b
- Source:
Ready for review — issue #28
PR: #63
Plan: #28 (comment)Outcome
CyberBattleSim adapter maintainers can now run RAES-shaped conformance probes that combine the canonical backend report with simulator-specific evidence checks. Manifest capability claims fail closed unless executable probe evidence covers the exact declared surface, and installed optional extras are verified without exposing native simulator state.
Added CyberBattleSim conformance composition helpers, RAES JSON-pointer diagnostics, fail-closed/no-leak tests, clean-install distribution probing, and documentation for the conformance guardrails.
Files changed
Added:
docs/decisions/cyberbattlesim-conformance-guardrails.mdsrc/raes_adapters/cyberbattlesim/backend/_diagnostics.pysrc/raes_adapters/cyberbattlesim/backend/conformance.pytests/test_cyberbattlesim_conformance.py
Modified:
README.mddocs/index.mdmkdocs.ymlnoxfile.pysrc/raes_adapters/cyberbattlesim/backend/__init__.pysrc/raes_adapters/cyberbattlesim/backend/evaluator.pysrc/raes_adapters/cyberbattlesim/backend/orchestrator.pysrc/raes_adapters/cyberbattlesim/backend/participant_runtime.pysrc/raes_adapters/cyberbattlesim/backend/provisioner.py
Reviews
- codex: Pre-push production-readiness review findings on manifest capability evidence gating were fixed by requiring exact capability-pointer probe coverage and fail-closed behavior for new affirmative surfaces.
- test-quality: The test-quality finding on no-leak failure coverage was fixed by asserting fail-closed outcomes across provisioner, orchestrator, participant runtime, evaluator, and cleanup paths.
Traceability reconciliation
- Pending — requirement status transition and IMPLEMENTS/TESTS reconciliation run in Phase E once the PR merges.
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge. Ground Control reconciliation (requirement status + traceability) runs on merge (Phase E).
Pre-PR base synchronization
- Source:
refs/remotes/origin/devat1dd255a92ab7b68c76c27b1637661fea7cee6518 - Outcome:
already_current - Published feature head:
bfb528061840d9883db956d2f53071af8895d26d - Verified tree:
3bfd39808fc15ec38647ef9943cee0d4255db84c
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd3e4187ecfc661e37da8fa9bfd7ae54facb27a71 - Outcome:
merged_clean - Published feature head:
89e6ef8e96da1562a636ebc0933450a4e4e7203a - Verified tree:
5604eff55a07e59520f8f79b791ceabe2274f77e
- Source:
Ready for review — issue #28
PR: #63
Plan: #28 (comment)Outcome
CyberBattleSim adapter maintainers can run RAES-shaped conformance probes that combine the canonical backend report with simulator-specific evidence checks. Downstream CyberBattleSim issues now require updating and passing the real native-readiness protocol when their adapter behavior changes.
Added CyberBattleSim conformance composition helpers, RAES JSON-pointer diagnostics, fail-closed/no-leak tests, clean-install distribution probing, and manual native-readiness protocol requirements for downstream CyberBattleSim work.
Files changed
Added:
docs/decisions/cyberbattlesim-conformance-guardrails.mdsrc/raes_adapters/cyberbattlesim/backend/_diagnostics.pysrc/raes_adapters/cyberbattlesim/backend/conformance.pytests/test_cyberbattlesim_conformance.py
Modified:
README.mddocs/decisions/cyberbattlesim-qualification-guardrails.mddocs/index.mdmkdocs.ymlnoxfile.pysrc/raes_adapters/cyberbattlesim/backend/__init__.pysrc/raes_adapters/cyberbattlesim/backend/evaluator.pysrc/raes_adapters/cyberbattlesim/backend/orchestrator.pysrc/raes_adapters/cyberbattlesim/backend/participant_runtime.pysrc/raes_adapters/cyberbattlesim/backend/provisioner.py
Reviews
- codex: Pre-push production-readiness review findings on manifest capability evidence gating were fixed by requiring exact capability-pointer probe coverage and fail-closed behavior for new affirmative surfaces.
- test-quality: The test-quality finding on no-leak failure coverage was fixed by asserting fail-closed outcomes across provisioner, orchestrator, participant runtime, evaluator, and cleanup paths.
Traceability reconciliation
- Pending — requirement status transition and IMPLEMENTS/TESTS reconciliation run in Phase E once the PR merges.
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge. Ground Control reconciliation (requirement status + traceability) runs on merge (Phase E).
gc workflow phase recorded:
traceability_reconciled(issue #28). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.no in-scope requirements; touched-files audit clean.
Final report — issue #28 complete
PR: #63
Plan: #28 (comment)Outcome
CyberBattleSim adapter maintainers can run RAES-shaped conformance probes that combine the canonical backend report with simulator-specific evidence checks. Downstream CyberBattleSim issues now require updating and passing the real native-readiness protocol when their adapter behavior changes.
Added CyberBattleSim conformance composition helpers, RAES JSON-pointer diagnostics, fail-closed/no-leak tests, clean-install distribution probing, and manual native-readiness protocol requirements for downstream CyberBattleSim work.
Files changed
Added:
docs/decisions/cyberbattlesim-conformance-guardrails.mdsrc/raes_adapters/cyberbattlesim/backend/_diagnostics.pysrc/raes_adapters/cyberbattlesim/backend/conformance.pytests/test_cyberbattlesim_conformance.py
Modified:
README.mddocs/decisions/cyberbattlesim-qualification-guardrails.mddocs/index.mdmkdocs.ymlnoxfile.pysrc/raes_adapters/cyberbattlesim/backend/__init__.pysrc/raes_adapters/cyberbattlesim/backend/evaluator.pysrc/raes_adapters/cyberbattlesim/backend/orchestrator.pysrc/raes_adapters/cyberbattlesim/backend/participant_runtime.pysrc/raes_adapters/cyberbattlesim/backend/provisioner.py
Reviews
- codex: Pre-push production-readiness review findings on manifest capability evidence gating were fixed by requiring exact capability-pointer probe coverage and fail-closed behavior for new affirmative surfaces.
- test-quality: The test-quality finding on no-leak failure coverage was fixed by asserting fail-closed outcomes across provisioner, orchestrator, participant runtime, evaluator, and cleanup paths.
Traceability reconciliation
- IMPLEMENTS / TESTS / DOCUMENTS added: 0
- Links updated: 0
- Stale links removed: 0
No requirement UIDs were in scope for issue #28. The PR body records issue-level IMPLEMENTS and TESTS links for the CyberBattleSim conformance helpers, clean-install distribution probe, documentation, and regression tests. Issues #29, #30, and #31 now require updating and passing the CyberBattleSim manual native-readiness protocol for their downstream paths.
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge.
Objective
Compose published RAES backend conformance with CyberBattleSim-specific mapping, control, and leakage probes.
Scope
Acceptance criteria
cyberbattlesimextra.