Repository navigation
test(primaite): compose RAES conformance with simulator-specific probes #42
Description
Objective
Compose published RAES backend conformance with PrimAITE-specific mapping, control, and leakage probes.
Scope
- Validate the adapter manifest with the published RAES model.
- Run the applicable published backend profile and canonical fixtures against a fully constructed RuntimeTarget.
- Add adapter-local probes for source-ledger coverage, source/profile identity, reset/seed/clock controls, action and observation projection, evaluator projection, cleanup, and portable-output leakage.
- Define a small deterministic PR suite and a broader study/readiness suite.
- Emit machine-readable RAES diagnostics, conformance reports, evidence references, and declared weaknesses.
- Keep adapter probes additive; do not create a second profile table, fixture corpus, schema registry, or manifest authority.
Acceptance criteria
- Every applicable published fixture passes or returns a justified machine-readable unsupported result.
- Every manifest capability links to passing evidence.
- Success and failure paths for all four backend surfaces are leak-tested.
- The selected source protocol's reset, stochastic, action, observation, reward, and terminal controls have executable probes.
- Clean-install conformance runs in CI under the
primaiteextra. - Reports distinguish backend conformance from source-protocol reproduction and research claims.
Activity
Brad-Edwards commentedon Aug 4, 2026
🛠️ Picked up by /implement - driver claude-code, branch
42-primaite-conformance-probes, 2026-08-04T17:14:08.826Z.Brad-Edwards commentedon Aug 4, 2026
gc workflow phase recorded:
preflight(issue #42). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Brad-Edwards commentedon Aug 4, 2026
gc workflow phase recorded:
plan(issue #42). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Implementation plan — compose RAES conformance with PrimAITE-specific probes
Context & design authority
test(primaite)run (issue has no## Requirementssection). Governance runs with--skip-requirement.docs/decisions/primaite-conformance-guardrails.md(authored by the architecture preflight this run), reconciled withprimaite-backend-guardrails.md(feat(primaite): implement the conformant simulator backend target #41/feat(primaite): implement the conformant simulator backend target #73).tests/test_cyberbattlesim_conformance.py,tests/test_nasim_conformance.py, and thecyborg/cyberbattlesimconformance modules.PrimaiteDriververifies source identity then refuses in-process construction (CPython-3.11-only qualified runtime; PrimAITE writes platform dirs on import). A fake/injected driver proves portable mechanics but can never certify live-native capability. Consequently capability evidence stays empty and every affirmative manifest capability is a disclosed gap — the inverse of the nasim/cbs closure. This change extends that established contract; it does not migrate the hand-rolledconformance.pyonto the gym framework.Acceptance criteria → deliverable
run_primaite_conformancereport (single publishedrealization-envelope-v1no-witness unsupported case),native_conformance=false.primaite_manifest_capability_evidence()stays{},…_gaps()= all affirmative pointers, and a newly-declared affirmative capability surfaces as a gap._LEAKAGE_MARKERS.primaiteextra → newPRIMAITE_CONFORMANCE_PROBE+venv-primaiteblock innoxfile.py_distributions().passed.Files
Add
tests/test_primaite_conformance.py— deterministic PR probe suite (injected driver, never imports PrimAITE): canonical report+payload shapes; source-protocol diagnostics + declared weaknesses are RAES models; capability evidence fails closed (evidence == {}, gaps ⊇/capabilities/cleanup/supported_action_kinds,/capabilities/participant_runtime/feature_support; mutated payload → new affirmative capability appears as a gap); PR bundle composes report+diagnostics+weaknesses+disclosed non-claims, JSON-serializable, no leakage; clock/time unsupported (manifest.time is None); four-surface failure/leakage cross-product + cleanup with every diagnostic validated bydiagnostic_model; manual-native-readiness protocol test asserting the guardrails-doc phrasing.Modify
src/raes_adapters/primaite/backend/conformance.py— addPR_CONFORMANCE_SEED(fixed literal) andrun_primaite_pr_conformance(*, driver, seed=PR_CONFORMANCE_SEED) -> dictcomposing the machine-readable disclosure bundle:seed,native_conformance(False),backend_conformance(payload),source_diagnostics(diagnostic_payload),capability_evidence({}),capability_gaps(disclosed non-claims),declared_weaknesses. It does not raise on gaps (gaps are the expected fail-closed disclosure, unlike nasim). Existing 6 functions unchanged.src/raes_adapters/primaite/backend/__init__.py— exportPR_CONFORMANCE_SEED,run_primaite_pr_conformance.noxfile.py— addPRIMAITE_CONFORMANCE_PROBEstring +venv-primaiteclean-install block in_distributions()mirroring the cyberbattlesim block; run the bundle from the installed[primaite]wheel with-I, clearedPYTHONPATH,PYTHONSAFEPATH=1; assertnative_conformance is False, the single non-passingrealization-envelope-v1case,capability_evidence == {}, non-empty gaps, JSON-serializable.docs/decisions/primaite-conformance-guardrails.md— add the manual-native-readiness anchor paragraph the readiness test asserts against (namesPrimaiteDriver,run_primaite_conformance(),backend_conformance_report_payload(),primaite_source_protocol_diagnostics(), the deterministic injected-driver lane, native blocked pending worker isolation + CPython-3.12 qualification).README.md— add a "PrimAITE backend conformance" section mirroring the NASim section (three distinct claims; fail-closed/blocked-native posture).docs/index.md,mkdocs.yml— nav link to the guardrails doc (already added by preflight; verify).No
CHANGELOG.mdedit (Release Please owns it); PR title will be Conventional Committest(primaite): ….Cross-cutting review
diagnostic_model). Leakage cross-product proves no native ids /Box(1652)/ masks /info/ reward / paths / tracebacks / tokens cross. Clean-install probe runs isolated (-I, clearedPYTHONPATH,PYTHONSAFEPATH=1).run_conformance_probe,_conformance_support.affirmative_capability_pointers, existing primaite surfaces +FakeDriverpatterns,diagnostic_payload/diagnostic_model. No new abstraction, controller, schema, or diagnostic hierarchy.EvidenceSelection+ injected driver factory + fixed seed; future live runtime enters behind a reviewed worker boundary + CPython-3.12 qualification — not this change.pyproject.toml/uv.lock, emptyprimaiteextra unchanged,_distributions()clean-wheel matrix + CIPR Gate, strictnox -s verify.Verification
uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verifywith--skip-requirement.nox -s tests -- tests/test_primaite_conformance.pyplus the primaite backend suite.Brad-Edwards commentedon Aug 4, 2026
gc_codex_review — cycle 1 of 1 (pre-push) on issue #42 (branch
42-primaite-conformance-probes)Diff mode: inline — the complete diff was supplied in one prompt
Core review
Verdict:
shipThis change is shaped as an adapter-local conformance composition layer: it keeps the RAES backend report as the canonical object, adds PrimAITE-specific source diagnostics and gap disclosure beside it, and documents the boundary rather than inventing a new schema or profile. The implementation seam is narrow (
run_primaite_pr_conformance), reuses the existing manifest/report/diagnostic projectors, and keeps the injected-driver lane explicitly non-native. The nox clean-install probe exercises the packaged extra without broadening workflow shape. I would ship this.No blocking findings.
Security review
Verdict:
shipThis change is shaped as a conformance-disclosure composition, not a new runtime/control-plane surface: it keeps the RAES backend report, PrimAITE source diagnostics, capability gaps, and weakness disclosures as separate outputs, and the only production code path added is a deterministic bundle projector around existing conformance and manifest helpers. The security-relevant boundary remains the existing driver/component boundary and validated RAES payload projection; this diff does not add request handling, persistence, auth scope, secret material, dynamic imports, shell interpolation, or user-controlled file execution paths. I would ship this from an application-security standpoint.
No blocking findings.
Brad-Edwards commentedon Aug 4, 2026
gc_codex_review pre-push cycle 1 of 1 complete for issue #42 on branch '42-primaite-conformance-probes'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Brad-Edwards commentedon Aug 4, 2026
Review decision record — codex cycle 1 (issue #42)
Reviewer: codex
Cycle: 1
Architectural read:
Blocking findings: 0 (clean run)
Brad-Edwards commentedon Aug 4, 2026
gc_test_quality_review cycle 1 of 1 — issue #42
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:
42-primaite-conformance-probesCycle: 1 / 1
Findings: 0 (clean run)
Brad-Edwards commentedon Aug 4, 2026
gc_test_quality_review cycle 1 of 1 complete for issue #42 on branch '42-primaite-conformance-probes'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Brad-Edwards commentedon Aug 4, 2026
Review decision record — test-quality cycle 1 (issue #42)
Reviewer: test-quality
Cycle: 1
Architectural read:
Blocking findings: 0 (clean run)
Brad-Edwards commentedon Aug 4, 2026
Ready for review — issue #42
PR: #75
Plan: #42 (comment)
Outcome
PrimAITE now ships a deterministic, offline conformance-composition test suite plus a clean-install CI proof that compose the published RAES backend conformance report with adapter-local source-protocol and leakage probes. Because the live PrimAITE driver cannot run in-process, the suite is fail-closed: it proves the portable mechanics and honestly discloses every unproven live capability as an open gap rather than certifying it, keeping backend conformance, source-protocol reproduction, and research claims distinct.
Files changed
Added:
tests/test_primaite_conformance.pydocs/decisions/primaite-conformance-guardrails.mdModified:
src/raes_adapters/primaite/backend/conformance.pysrc/raes_adapters/primaite/backend/__init__.pynoxfile.pyREADME.mddocs/index.mdmkdocs.ymlReviews
Traceability reconciliation
Status
Documentation
Updated: see diff.
doublewhy commentedon Aug 13, 2026
Verified complete in merged PR #75 (
8c39789f3198530caf303aa836aad11e5eb18260). The PR added the PrimAITE conformance composition, fail-closed capability evidence, four-surface leakage probes, clean-installprimaite-extra CI proof, manual native-readiness protocol, and documentation. All recorded required checks passed, including tests, distributions, typecheck, policy, docs, CodeQL, SonarCloud, GitGuardian, and the aggregate PR Gate.