Repository navigation
test(cyborg): compose RAES backend conformance with simulator-specific probes #19
Description
Activity
🛠️ Picked up by /implement - driver codex, branch
19-cyborg-conformance-probes, 2026-08-01T21:05:34.877Z.gc workflow phase recorded:
preflight(issue #19). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.gc workflow phase recorded:
plan(issue #19). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Implementation plan
-
Drive the published RAES boundary test-first. Add
tests/test_cyborg_conformance.pywith failing cases for registry-loadedbackend-manifest-v2validation, published profile/corpus selection, an executable hermeticRuntimeTarget, exact RAES report projection/persistence, and fail-closed realization probes. Update the CybORG manifest only from RAES-owned profile/model identities, make its existing realization envelope constructive for the actually supported switch/VM projection, and provide an independent injected-driver realization harness. Do not add a local profile, fixture family, report case, aggregate pass result, or base semantic abstraction. -
Add adapter-local evidence without creating authority. Implement
raes_adapters.cyborg.conformancearound the existingrun_conformance_probe, manifest projector, source-ledger/qualification loaders, RAESDiagnosticvalidator/projector, and RAES report writer. Expose fixed PR seeds(3,)and full seeds(3, 153), manifest-derived capability-to-passing-evidence closure, declared qualification/loss weakness references, and safe reproduction argv. Extend package exports and README documentation. Keep the evaluator-seed limitation visible and state explicitly that finite conformance is not scientific equivalence. -
Close the diagnostic and leakage boundaries. Add the same JSON-Pointer conversion pattern already used by the CyberBattleSim module and route every CybORG diagnostic address through it. Add success/failure tests across provisioner, orchestration/lifecycle, participant action/observation, evaluator/reward, time, and cleanup using hostile native sentinels whose
str/reprare unsafe. Validate actual JSON-ready diagnostics, canonical report payloads, and persisted output; assert no native ids, observations, reward vectors/mappings, hidden truth, credentials, paths, argv/environment values, exception text, causes, or tracebacks escape. -
Exercise deterministic suites through existing gates. Extend the existing nox test/distribution owners rather than introduce a second verification authority: PR/hermetic conformance uses seed 3 and an injected driver; clean-install builds the one wheel, installs
[cyborg]in an isolated environment, loads packaged qualification/ledger resources, runs the canonical report plus local probes, and persists the report through the RAES writer. Extend the existing CI workflow with a scheduled/dispatch full-tier path using ordered seeds 3 and 153 and the existing qualified-source reproducer; do not add a second workflow or lockfile. Updatedocs/maintainers/ci.mdwith exact local reproduction commands. -
Verify clause by clause and run the canonical graph. Map every issue acceptance criterion to source/test/docs lines, run focused CybORG conformance and native qualification tests first, then the repository completion and policy gates with the issue’s requirement-free governance binding. Preserve the preflight-authored guardrail note and its docs navigation entries.
Design checks
- Security: closed target config remains the only config parser; manifest, realization, diagnostic, and report projectors validate every portable shape; native failures are replaced at component boundaries with stable input-free diagnostics; execution uses no secret input, shell interpolation, environment dump, or native value rendering.
- Maintainability: reuse
BackendRegistry,create_cyborg_target,run_conformance_probe, source-ledger validators, RAES diagnostics/report writer, the CyberBattleSim composition pattern, and the existing nox/CI owners. New code stays CybORG-local. - Extensibility: suite tier selects an immutable
EvidenceSelection, ordered seed tuple, driver/harness, and RAESExecutionBasis; a future qualified source does not require a new schema/profile/registry/base API. - Whole-repo: account for
pyproject.toml/singleuv.lock, package resources, source and tests, docs navigation, README, nox test/distribution sessions, the existing CI/PR Gate contract, qualification reproducer, strict typing/lint/docs/policy, and clean-install isolation.
-
gc_codex_review — cycle 1 of 1 (pre-push) on issue #19 (branch
19-cyborg-conformance-probes)
Diff mode: inline — the complete diff was supplied in one promptCore review
Verdict:
ship-with-fixesThe change is broadly shaped correctly: CybORG-specific evidence remains inside the backend module, published RAES profiles, diagnostics, report projection, persistence, and registry seams remain authoritative, and CI extends the incumbent workflow and distribution proof. The suite-tier seam cleanly selects fixed seeds without inventing shared semantics and leaves room for another qualified source selection. Two evidence gates can nevertheless report validation after observing a materially weaker outcome than intended, so I would fix those false-green paths before shipping.
Blocking findings (2):
- [one-off] The unsupported-case gate accepts more than the declared bounded exception —
src/raes_adapters/cyborg/conformance.py:850
This predicate accepts every unsupported case carrying any diagnostic whose code ends inno-witness. The design permits one specific runner-owned realization-envelope disposition, but this gate neither identifies that case nor limits its count. A future regression producing an additional unsupported no-witness case would therefore pass the suite gate and qualify as published-conformance evidence. Match the exact published exception and require every other case to pass; pin the case identity and cardinality in the test rather than mirroring this broad predicate. - [one-off] An earlier participant termination failure can be reported as lifecycle validation —
src/raes_adapters/cyborg/conformance.py:396
The termination loop overwritesterminatedon each participant, and the lifecycle disposition later checks only the final result. If Blue or Green termination fails but a later termination succeeds, the probe can emitlifecycle.validateddespite an incomplete aggregate lifecycle. Retain each termination disposition and require all of them to succeed, with a failure-path test where a non-final participant fails.
Security review
Verdict:
shipThis change is shaped as a bounded, offline conformance runner layered over the existing RAES manifest, diagnostic, report-projection, and redaction boundaries. Its security-sensitive seams are native-to-portable data projection, report persistence, operational output paths, lazy module loading, and CI artifact publication; the diff keeps imports and execution targets fixed, suppresses native exception details, uses the canonical report writer, and exposes no new request, authentication, authorization, or tenant boundary. The fixed suite-selection seam also accommodates the obvious next reviewed seed tier without introducing dynamic code execution or untrusted configuration. I found no concrete, exploitable security issue introduced by this diff.
No blocking findings.
- [one-off] The unsupported-case gate accepts more than the declared bounded exception —
gc_codex_review pre-push cycle 1 of 1 complete for issue #19 on branch '19-cyborg-conformance-probes'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Review decision record — codex cycle 1 (issue #19)
Reviewer: codex
Cycle: 1Architectural read:
Core reviewer: The change is broadly shaped correctly: CybORG-specific evidence remains inside the backend module, published RAES profiles, diagnostics, report projection, persistence, and registry seams remain authoritative, and CI extends the incumbent workflow and distribution proof. The suite-tier seam cleanly selects fixed seeds without inventing shared semantics and leaves room for another qualified source selection. Two evidence gates can nevertheless report validation after observing a materially weaker outcome than intended, so I would fix those false-green paths before shipping.
Security reviewer: This change is shaped as a bounded, offline conformance runner layered over the existing RAES manifest, diagnostic, report-projection, and redaction boundaries. Its security-sensitive seams are native-to-portable data projection, report persistence, operational output paths, lazy module loading, and CI artifact publication; the diff keeps imports and execution targets fixed, suppresses native exception details, uses the canonical report writer, and exposes no new request, authentication, authorization, or tenant boundary. The fixed suite-selection seam also accommodates the obvious next reviewed seed tier without introducing dynamic code execution or untrusted configuration. I found no concrete, exploitable security issue introduced by this diff.
Blocking findings: 2
Finding 1 —
one-off- ID:
F1 - Title: [core] The unsupported-case gate accepts more than the declared bounded exception
- Location:
src/raes_adapters/cyborg/conformance.py:850 - Decision: fix
- Rationale: This predicate accepts every unsupported case carrying any diagnostic whose code ends in
no-witness. The design permits one specific runner-owned realization-envelope disposition, but this gate neither identifies that case nor limits its…
Finding 2 —
one-off- ID:
F2 - Title: [core] An earlier participant termination failure can be reported as lifecycle validation
- Location:
src/raes_adapters/cyborg/conformance.py:396 - Decision: fix
- Rationale: The termination loop overwrites
terminatedon each participant, and the lifecycle disposition later checks only the final result. If Blue or Green termination fails but a later termination succeeds, the probe can emit `lifecycle.validate…
- ID:
gc_test_quality_review cycle 1 of 1 — issue #19
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:19-cyborg-conformance-probes
Cycle: 1 / 1
Findings: 2Finding 1 — [warning]
tests/test_cyborg_conformance.py:246Problem: test_failure_diagnostics_from_every_adapter_surface_validate_and_redact claims to validate redaction across provisioner, orchestrator, evaluator, and participant_runtime, but three of its four calls (orchestrator.start(object()), evaluator.start(object()), participant_runtime.reset(object())) hit early type-guard branches whose diagnostic messages are fixed string literals that never incorporate the invalid input or any driver-owned value.
Why it matters: If a future change to CyborgOrchestrator._workflow_operation, CyborgEvaluator._validate_plan, or CyborgParticipantRuntime.reset began interpolating driver-native details into their diagnostic messages, this test would still pass for those three surfaces, since passing object() can never trigger such content. Only the provisioner branch exercised here (already independently covered by test_hostile_construction_failure_never_leaks_to_portable_output) actually forces a driver exception carrying secret content.
Fix: Either drop the three non-provisioner calls from this test and rename it to reflect that it only checks generic RAES diagnostic-schema validity, or extend it to genuinely exercise a leak-capable path per surface (e.g. a driver whose step/reset/project_evaluation raises with a secret-bearing message, invoked through orchestrator/evaluator/participant_runtime call sites that actually reach the driver) so each surface gets real redaction coverage.Finding 2 — [warning]
tests/test_cyborg_conformance.py:287Problem: run_cyborg_conformance_suite (conformance.py) has three fail-closed guard clauses that raise RuntimeError to stop a broken adapter from publishing a falsely-clean conformance index (unvalidated adapter diagnostics, a published conformance report with unexpected failing cases, or manifest capability evidence gaps). test_suite_index_preserves_canonical_reports_and_non_claims is the only test that calls this function, and it exercises exclusively the all-passing happy path.
Why it matters: If any of these three guards were accidentally removed or weakened (e.g. the raise turned into a log-and-continue), no test would fail, so a genuinely broken adapter could silently emit an index.json that claims full conformance — the exact audit-integrity failure this function's guard logic exists to prevent.
Fix: Add negative-path tests that force each guard to fire independently: monkeypatch cyborg_adapter_diagnostics to return a non-'.validated' code, force run_cyborg_conformance to return a report with an unexpected failing case, and construct a scenario where cyborg_manifest_capability_evidence_gaps is non-empty; assert RuntimeError is raised (with pytest.raises) in each case.gc_test_quality_review cycle 1 of 1 complete for issue #19 on branch '19-cyborg-conformance-probes'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Review decision record — test-quality cycle 1 (issue #19)
Reviewer: test-quality
Cycle: 1Architectural read:
This changeset extends the CybORG/CAGE-2 RAES adapter with aggregate logical-turn execution, reward-outcome evaluation, and a new conformance/evidence module (conformance.py, _diagnostics.py), and the bulk of the test suite matches that shape well: tests route through the real target/orchestrator/participant-runtime/evaluator/provisioner stack with narrow fakes rather than mocking language internals, use a deliberate 'HostileNative' fixture that hard-fails (AssertionError) on any accidental str/repr to pin the no-native-leakage invariant across every surface, and drive genuine boundary conditions (malformed native turn projections, quarantine-then-recover via reconstruction, post-effect contract-diagnostic failures forced via monkeypatch, source-tree tamper/bytecode-substitution detection, RNG-isolation across the native boundary) with assertions on real state — snapshots, diagnostic codes, driver call counts, typed evidence/measure models — rather than mock call-counts. test_cyborg_qualification.py's fixed-value assertions against the packaged qualification.json are intentionally golden-record checks on a compliance/audit artifact, which is the right shape for that data, not a smell. The one place the shape slips is the new conformance.py module: run_cyborg_conformance_suite's three fail-closed RuntimeError guards (meant to stop a broken adapter from publishing a falsely-clean conformance index) have no negative-path test at all, and one 'covers every adapter surface' test bundles three vacuous branches (which structurally can never reach driver-owned content) alongside the one branch that actually matters for secret redaction. Neither is a structural defect in the adapter itself, but both leave the newly-added conformance/evidence-integrity surface under-verified relative to the rest of this otherwise disciplined suite.
Blocking findings: 2
Finding 1 —
one-off- ID:
F1 - Title: (no title)
- Decision: fix
- Rationale: Addressed by next cycle
Finding 2 —
one-off- ID:
F2 - Title: (no title)
- Decision: fix
- Rationale: Addressed by next cycle
- ID:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devataca6ca9aadf008396eb2abf80a0aac0c399eaf39 - Outcome:
already_current - Published feature head:
28f51931d3742dbbf2a9def2afb1881f3c306d93 - Verified tree:
96cc4fc427117e4f24c5671c85d6cd9b3d91fd16
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devataca6ca9aadf008396eb2abf80a0aac0c399eaf39 - Outcome:
already_current - Published feature head:
fd18ee70478787b63cbae10ce26387baae696210 - Verified tree:
d0a92de6c729f78bda26e143007eef682a49f2dc
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devataca6ca9aadf008396eb2abf80a0aac0c399eaf39 - Outcome:
already_current - Published feature head:
494dc5e06b6bb85398065d8f0784afcf047d5510 - Verified tree:
76a912d8dd6a396a479dad69b2cbcea06b4e43a2
- Source:
Ready for review — issue #19
PR: #65
Plan: #19 (comment)Outcome
CybORG now composes the published RAES backend conformance profile with deterministic simulator-specific lifecycle, action, observation, evaluation, cleanup, and portable-output probes; PR and weekly tiers publish canonical evidence without overstating native conformance.
Files changed
Added:
docs/decisions/cyborg-conformance-guardrails.mdsrc/raes_adapters/cyborg/_diagnostics.pysrc/raes_adapters/cyborg/conformance.pytests/test_cyborg_conformance.py
Modified:
.github/workflows/ci.ymlREADME.mddocs/index.mddocs/maintainers/ci.mdmkdocs.ymlnoxfile.pypyproject.tomlsrc/raes_adapters/cyborg/__init__.pysrc/raes_adapters/cyborg/evaluator.pysrc/raes_adapters/cyborg/manifest.pysrc/raes_adapters/cyborg/orchestrator.pysrc/raes_adapters/cyborg/participant_runtime.pysrc/raes_adapters/cyborg/provisioner.py
Reviews
- codex: Two false-green evidence gates were found and fixed; the user authorized proceeding without an over-cap rerun.
- test-quality: Two negative-path coverage gaps were found and fixed; the user authorized proceeding without an over-cap rerun.
Traceability reconciliation
- Pending — requirement status transition and IMPLEMENTS/TESTS reconciliation run in Phase E once the PR merges.
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge. Ground Control reconciliation (requirement status + traceability) runs on merge (Phase E).
gc workflow phase recorded:
traceability_reconciled(issue #19). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.no in-scope requirements; touched-files audit clean.
Final report — issue #19 complete
PR: #65
Plan: #19 (comment)Outcome
CybORG now composes the published RAES backend conformance profile with deterministic simulator-specific lifecycle, action, observation, evaluation, cleanup, and portable-output probes. PR and weekly tiers publish canonical evidence without overstating native conformance.
Files changed
Added:
docs/decisions/cyborg-conformance-guardrails.mdsrc/raes_adapters/cyborg/_diagnostics.pysrc/raes_adapters/cyborg/conformance.pytests/test_cyborg_conformance.py
Modified:
.github/workflows/ci.ymlREADME.mddocs/index.mddocs/maintainers/ci.mdmkdocs.ymlnoxfile.pypyproject.tomlsrc/raes_adapters/cyborg/__init__.pysrc/raes_adapters/cyborg/evaluator.pysrc/raes_adapters/cyborg/manifest.pysrc/raes_adapters/cyborg/orchestrator.pysrc/raes_adapters/cyborg/participant_runtime.pysrc/raes_adapters/cyborg/provisioner.py
Reviews
- codex: Two false-green evidence gates were found and fixed; the user authorized proceeding without an over-cap rerun.
- test-quality: Two negative-path coverage gaps were found and fixed; the user authorized proceeding without an over-cap rerun.
Traceability reconciliation
- IMPLEMENTS / TESTS / DOCUMENTS added: 0
- Links updated: 0
- Stale links removed: 0
No formal requirement UID was in scope; post-merge reconciliation completed the empty-requirement orphaned-link audit.
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge.
Objective
Provide one executable conformance and disclosure surface for the completed CybORG adapter.
Scope
backend-manifest-v2.Acceptance criteria
References