Repository navigation
REP-005 — Drive and validate CAGE-2 replication across the CybORG sim and the ACES reference backend #639
Description
Goal
Use one CAGE-2 SDL scenario (REP-003) with both the CybORG simulator adapter (REP-004) and the ACES reference backend. Capture reproducible, paper-ready evidence that both realize the authored scenario consistently under the equivalence rules in REP-001.
Done when
- The same SDL revision drives both supported paths.
- The evidence records each backend, configuration, result, limitation, and the REP-001 comparison.
- Another researcher can repeat the reported result.
CAGE-2 emulation-backend realization is outside this issue. Requirement: REP-005. Design: #635. Milestone: Paper Slice: CAGE-2 Replication (ACES-Driven Sim) (#58).
Activity
doublewhy commentedon Aug 13, 2026
Evidence and claim correction (2026-08-13)
The currently retained result does not satisfy REP-005:
-3112.2is the published CAGE-2 README leaderboard total for CCS Sleeper.-3112.6716is the observed primary total from the 9,000 valid OpenRAE-mediated runs in the same-CybORG behavioral reconstruction (-3112.6715999999997in the retained binary64 aggregate).bounded.The supported claim is bounded behavioral-baseline outcome reproduction. It is not evidence of two independent conformant backends, a RAES reference-backend realization, direct-native-versus-OpenRAE equivalence, submitted-agent artifact identity, or state/observation equivalence.
Evidence: OpenRAE/adapters#91 and the frozen aggregate at the retained research head in OpenRAE/research#75. The stronger paired direct-native-versus-mediated experiment is separately tracked by OpenRAE/research#76.
Keep #639 open until its actual two-backend acceptance criteria are met.
[-]REP-005 — Drive and validate CAGE-2 replication across the CybORG sim and the ACES reference backend[/-][+]Deferred: CAGE-2 cross-backend replication (REP-005)[/+][-]Deferred: CAGE-2 cross-backend replication (REP-005)[/-][+]REP-005 — Drive and validate CAGE-2 replication across the CybORG sim and the ACES reference backend[/+]