Repository navigation
test(nasim): compose RAES conformance with simulator-specific probes #35
Description
Objective
Compose published RAES backend conformance with NASim-specific mapping, control, and leakage probes.
Scope
- Validate the adapter manifest with the published RAES model.
- Run the applicable published backend profile and canonical fixtures against a fully constructed RuntimeTarget.
- Add adapter-local probes for source-ledger coverage, source/profile identity, reset/seed/clock controls, action and observation projection, evaluator projection, cleanup, and portable-output leakage.
- Define a small deterministic PR suite and a broader study/readiness suite.
- Emit machine-readable RAES diagnostics, conformance reports, evidence references, and declared weaknesses.
- Keep adapter probes additive; do not create a second profile table, fixture corpus, schema registry, or manifest authority.
Acceptance criteria
- Every applicable published fixture passes or returns a justified machine-readable unsupported result.
- Every manifest capability links to passing evidence.
- Success and failure paths for all four backend surfaces are leak-tested.
- The selected source protocol's reset, stochastic, action, observation, reward, and terminal controls have executable probes.
- Clean-install conformance runs in CI under the
nasimextra. - Reports distinguish backend conformance from source-protocol reproduction and research claims.
- Manual integration test — honest NASim execution: run the deterministic PR conformance suite against a fully-constructed RuntimeTarget backed by a live installed NASim (qualified runtime per build(nasim): qualify and pin the public source and experiment profile #32; not a stub/mock), confirming every probe executes against the real simulator — a hands-on check beyond the automated fixtures.
Activity
Brad-Edwards commentedon Aug 3, 2026
🛠️ Picked up by /implement - driver claude, branch
35-nasim-conformance-probes, 2026-08-03T06:21:57.083Z.Brad-Edwards commentedon Aug 3, 2026
gc workflow phase recorded:
preflight(issue #35). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Brad-Edwards commentedon Aug 3, 2026
gc workflow phase recorded:
plan(issue #35). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Implementation plan — compose RAES conformance with NASim-specific probes
Requirement-free run (no in-scope requirement UID;
nox -s verify -- --skip-requirement). Design authority:docs/decisions/nasim-conformance-guardrails.md(authored by architecture preflight).Key finding from codebase assessment
The NASim production conformance API already exists and mirrors the CyberBattleSim (gym-sibling) template 1:1 —
run_nasim_conformance,nasim_backend_conformance_payload,nasim_manifest_capability_evidence[_gaps],nasim_declared_weaknesses,nasim_source_protocol_diagnostics, all composed through the sharedGymConformanceConfigin_gym_backend/conformance.py. #35 is therefore a test + docs + distribution-boundary deliverable: exercise that production composition and close the CBS-parity gaps. Per the guardrails, this must execute the production composition and reuse existing fixtures/patterns, not grow a second inline suite or a parallel test-only controller (so no cyborg-styleindex.json/CLI).Production code
src/raes_adapters/nasim/backend/conformance.py— addPR_CONFORMANCE_SEED = 20260802andrun_nasim_pr_conformance(*, driver, seed=PR_CONFORMANCE_SEED) -> dict. It composes, over an explicitly-supplied driver: published-profile conformance (run_nasim_conformance) → payload;nasim_source_protocol_diagnostics; fail-closed capability-evidence closure;nasim_declared_weaknesses. Returns a validated portable dict (seed,backend_conformance,native_conformance,source_diagnostics,capability_evidence,declared_weaknesses); raises a bounded error when capability gaps exist.driveris required — no silent injected/real fallback (guardrails gotcha). This is the single production entry the PR test suite, the clean-install proof, and the manual gate all call.src/raes_adapters/nasim/backend/__init__.py— exportrun_nasim_pr_conformanceandPR_CONFORMANCE_SEED.noxfile.py— generalize the inlineNASIM_CONFORMANCE_PROBEdistribution boundary to import and callrun_nasim_pr_conformancefrom the installed wheel (running the full PR suite, not just the basic payload assertion), asserting the returned shape,native_conformance is False, and no sentinel leakage. Keeps the deterministic injected driver.Tests —
tests/test_nasim_conformance.py(expand to CBS-parity)ProbeDriverwithfail_construct/fail_step/fail_evaluate/fail_close+ aSENTINEL, keeping NASim'sstep(action_kind, target_ref=None)signature andNasim*report types.test_canonical_conformance_report_and_payload_are_published_shapes— assertnasim_backend_conformance_payload(report) == backend_conformance_report_payload(report)andreport.native_conformance is False.test_source_protocol_diagnostics_and_declared_weaknesses_are_raes_models— every diagnostic validates viadiagnostic_model, addresses start with/, codes ⊇ {nasim.source-protocol.validated,.declared-weakness}, weaknesses ⊇ {limitation:action-success-rng-unbound-by-env-seed,loss:loss-unbound-action-rng:deterministic-replay}.test_manifest_capability_evidence_is_derived_and_fails_closed— existing backed-pointer coverage + gaps== (), plus the fails-closed branch: injecting an unbacked affirmative/capabilities/provisioner/supports_new_modesurfaces exactly that pointer as a gap.test_failure_surface_diagnostics_validate_and_do_not_leak_native_sentinels— driveNasimProvisioner,NasimOrchestrator,NasimParticipantRuntime,NasimEvaluator, andexecute_nasim_cleanupwith hostile drivers/sentinel; validate every diagnostic viadiagnostic_model; assert per-surface success/failure status; serialize all portable projections (ApplyResult, action result, cleanup receipt, diagnostics) and assert the sentinel and NASim native symbol markers (flatactionspace,servicescan, …) are absent. Covers "success and failure paths for all four backend surfaces are leak-tested."test_pr_conformance_suite_composes_published_and_source_evidence— callrun_nasim_pr_conformance(driver=ProbeDriver(), seed=PR_CONFORMANCE_SEED); assert the composed dict shape,native_conformanceFalse, non-emptycapability_evidence, and fail-closed behavior.test_clock_control_is_unsupported_consistent_with_manifest— assertcreate_nasim_manifest()declares no RAES time surface, i.e. clock control resolves to a validated unsupported/limitation disposition (NASim's serialized step count is an observation, not a logical clock).test_manual_native_readiness_protocol_covers_adapter_conformance_path— assert the conformance-guardrails doc documents the manual live-NASim lane with the exact commitments: realNasimDriver,run_nasim_conformance(),backend_conformance_report_payload(),nasim_source_protocol_diagnostics(), "deterministic injected-driver" CI probe, no fallback to the injected driver under the same label, and no native-coordinate leakage.Docs (ADR-054 coverage)
docs/decisions/nasim-conformance-guardrails.md,docs/index.md,mkdocs.yml— created/updated by preflight; align the guardrails prose with the doc-contract test phrases.README.md— add a NASim conformance-composition section mirroring the CyberBattleSim one (three distinct claims; exact published report/projector; additive probes; capability-evidence closure; link to the guardrails).Manual integration test (issue's final acceptance criterion)
During verification I will attempt the honest live-NASim run:
uv sync --extra nasim, construct a realNasimDriver+ fully-constructedRuntimeTarget, runrun_nasim_pr_conformanceagainst the real simulator, and confirm every probe executes against live NASim, recording only bounded portable evidence. If the environment cannot install/run NASim (e.g. missing Tk), that is reported as a failed/blocked manual-readiness result per the guardrails — never a switch to the injected driver under the same label.Design considerations
info, reward containers, random state, paths, and tracebacks never reach portable projections; leakage oracle inspects the actual JSON-ready projections (notrepr)._gym_backend/_conformance_supporttraversal andGymConformanceConfig; the extension seam stays an immutableEvidenceSelection+ explicit seed + declared-basis driver provider. No sibling-backend semantics imported.pyproject.toml/uv.lock, isolatednasimextra, existing nox graph and CIPR Gate, clean-wheel proof under_distributions(). NoCHANGELOG.mdedit (Release Please owns it); PR title will be a Conventional Committest(nasim): ….Verification
uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify -- --skip-requirement(hygiene, policy, lint, tool-tests, typecheck, tests incl. base +nasimextra, distributions clean-wheel proof, strict docs).Brad-Edwards commentedon Aug 3, 2026
gc_codex_review — cycle 1 of 1 (pre-push) on issue #35 (branch
35-nasim-conformance-probes)Diff mode: inline — the complete diff was supplied in one prompt
Core review
Verdict:
ship-with-fixesThe change chooses the correct seam: NASim composes the published RAES report projector, existing source diagnostics, shared capability-evidence closure, and declared weaknesses without inventing another profile or backend contract. That architecture fits the repository’s authority boundary and leaves the next qualified selection behind explicit driver/seed inputs. However, the exported function is presented as the executable PR/manual-live probe suite while implementing only aggregation of canonical conformance and static source evidence; the runtime probe inventory promised by the documentation remains outside that seam.
Blocking findings (1):
src/raes_adapters/nasim/backend/conformance.py:152run_nasim_pr_conformance()runs canonical target conformance, driver-independent source diagnostics, capability closure, and weakness collection, but it never executes the advertised additive source-runtime probes: hostile success/failure projections, independent terminal facts, source action/observation behavior, or explicit verified cleanup. Those exercises exist only as pytest code, while the installed-wheel probe and documented manual-live entry point invoke this function. Consequently, the clean-install lane can report a complete bundle without exercising the installed runtime evidence it claims; its sentinel assertion is also vacuous because that inline driver never introduces the sentinel. Put the required probe inventory behind the production composition callable and invoke it from both pytest and the installed-wheel lane, or narrow the API and documentation to the evidence it actually produces.Security review
Verdict:
shipThis change is shaped correctly: it adds a narrow composition seam over existing published report projection, validated diagnostics, manifest-derived capability evidence, and declared weaknesses. The explicit driver boundary preserves the distinction between deterministic injected and native-live execution, while capability closure fails closed. It introduces no request, authorization, persistence, secret-handling, dynamic-loading, or user-controlled file boundary, and it leaves room for the next driver or seed variation without weakening the existing security controls.
No blocking findings.
Brad-Edwards commentedon Aug 3, 2026
gc_codex_review pre-push cycle 1 of 1 complete for issue #35 on branch '35-nasim-conformance-probes'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Brad-Edwards commentedon Aug 3, 2026
Review decision record — codex cycle 1 (issue #35)
Reviewer: codex
Cycle: 1
Architectural read:
Blocking findings: 1
Finding 1 —
one-offF1src/raes_adapters/nasim/backend/conformance.py:152run_nasim_pr_conformance()runs canonical target conformance, driver-independent source diagnostics, capability closure, and weakness collection, but it never executes the advertised additive source-runtime probes: hostile success/failur…Brad-Edwards commentedon Aug 4, 2026
gc_test_quality_review cycle 1 of 1 — issue #35
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:
35-nasim-conformance-probesCycle: 1 / 1
Findings: 1
Finding 1 — [critical]
tests/test_nasim_conformance.py::test_pr_conformance_suite_composes_published_and_source_evidenceProblem: run_nasim_pr_conformance's fail-closed guarantee (
if gaps: raise RuntimeError(...)in src/raes_adapters/nasim/backend/conformance.py) is never driven to its failure branch anywhere in this file. The only test of the function exercises the real manifest, which always has zero gaps, so the raise is never triggered.Why it matters: If the fail-closed check were deleted or weakened (e.g. logged instead of raised, or the gaps computed against the wrong evidence set), no test in this suite would fail — the bundle would just silently ship with an unbacked capability, exactly the outcome the function's own docstring says it exists to prevent ("a new surface can never ship without an evidence join").
Fix: Add a negative test that monkeypatches nasim_manifest_capability_evidence_gaps (or the underlying conformance module reference used inside run_nasim_pr_conformance) to return a non-empty tuple, then assert pytest.raises(RuntimeError, match="incomplete") when calling run_nasim_pr_conformance — mirroring tests/test_cyborg_conformance.py::test_suite_refuses_manifest_capability_evidence_gaps.
Brad-Edwards commentedon Aug 4, 2026
gc_test_quality_review cycle 1 of 1 complete for issue #35 on branch '35-nasim-conformance-probes'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Brad-Edwards commentedon Aug 4, 2026
Review decision record — test-quality cycle 1 (issue #35)
Reviewer: test-quality
Cycle: 1
Architectural read:
Blocking findings: 1
Finding 1 —
one-offF1Brad-Edwards commentedon Aug 4, 2026
Pre-PR base synchronization
refs/remotes/origin/devatd8e0dbdece247704bf0403428d15b8975b455bd3already_current232f1861b1edf8e539c8258159c84a3755bd49f61e00ac507f3a9032471d763b872b7e12cb57a64bBrad-Edwards commentedon Aug 4, 2026
Pre-PR base synchronization
refs/remotes/origin/devatd8e0dbdece247704bf0403428d15b8975b455bd3already_current741308a9c3112839b207dcfee77149a77a63ec2621ce1c2550dc759b2f8c3c3bea0d7a35831c2420Brad-Edwards commentedon Aug 4, 2026
Ready for review — issue #35
PR: #74
Plan: #35 (comment)
Outcome
NASim now ships an executable RAES conformance suite: one entrypoint (run_nasim_pr_conformance) composes the published backend report, source-protocol diagnostics, fail-closed capability-evidence closure, and declared weaknesses into a portable bundle, run by both the deterministic PR tests and the clean-install wheel proof. It keeps three claims distinct — backend conformance vs. source-protocol reproduction vs. research/readiness — with hostile-failure and leakage probes over all four backend surfaces.
Compose published RAES backend conformance for NASim with simulator-specific probes over the shared _gym_backend seam; add the deterministic run_nasim_pr_conformance PR entrypoint, driven by the clean-install wheel proof.
Files changed
Added:
docs/decisions/nasim-conformance-guardrails.mdModified:
README.mddocs/index.mdmkdocs.ymlnoxfile.pysrc/raes_adapters/nasim/backend/__init__.pysrc/raes_adapters/nasim/backend/conformance.pytests/test_nasim_conformance.pyReviews
Traceability reconciliation
Status
Brad-Edwards commentedon Aug 4, 2026
Manual integration test — honest live-NASim execution (issue #35 final acceptance criterion)
Result: PASS. The deterministic PR conformance suite was executed against a real installed NASim — not a stub/mock, no fall back to the injected driver.
Command / lane
uv run --extra nasim pythondrivingrun_nasim_pr_conformance(driver=NasimDriver(), seed=PR_CONFORMANCE_SEED)— a liveraes_adapters.nasim.backend.driver.NasimDriverthrough a fully-constructedRuntimeTarget._tkinterpresent;import nasimsucceeds headless.Qualified identities (installed-source admission verified against
qualification.json)nasim==0.12.0(pinned),gymnasium==0.26.3,numpy==1.26.4;driver._artifacts_verified == True.Dispositions (bounded portable evidence only — no native dump)
backend_conformance.passed == True, 67 fixture cases;native_conformance == False(injected-vs-live label preserved).close()→closed=True, verified=True;verify_closed() == True.Scope note (consistent with the design authority). The live run exercises the published conformance composition (construct / reset / evaluate surfaces + fixture corpus). Per the guardrails and the resolved F1 boundary, the hostile-failure / portable-output-leakage / action-step probes are injected-driver constructs by nature — a real
NasimDrivercannot be made to raise on a chosen surface — so they run in the deterministic PR test suite, not against the live driver. This manual gate confirms the same composition against installed NASim; it does not replace the injected-driver CI probe.