feat(nasim): implement the conformant simulator backend target #34

Activity

1. Brad-Edwards

   added this to the NASim Research Demonstrator milestone on Jul 29, 2026
2. Brad-Edwards

   marked this as blocked by 3 issues on Jul 29, 2026

   • build(nasim): qualify and pin the public source and experiment profile #32
   • docs(nasim): author the RAES scenario and pinned source mapping ledger #33
   • feat(base): implement shared simulator-adapter plumbing over published RAES contracts #14
3. Brad-Edwards

   marked this as blocking 2 issues on Jul 29, 2026

   • test(nasim): compose RAES conformance with simulator-specific probes #35
   • feat(nasim): add the example environment pack and researcher workflow #36
4. Brad-Edwards

   added

   in-progressAn agent is actively working this issue via /implementAn agent is actively working this issue via /implement

   on Aug 2, 2026
5. Brad-Edwards commented on Aug 2, 2026

   

   Brad-Edwards

   on Aug 2, 2026

   CollaboratorAuthor

   More actions

   Picked up by /implement (driver: claude) on branch 34-nasim-backend-target at 2026-08-02T22:58:53Z.

   Note: running under the documented GC#1502 manual-drive workaround — the mechanical MCP tooling is blocked by a stale launch-time identity pin in this linked worktree, so branch/commit/push/PR/reviews are being driven by hand this run.
6. Brad-Edwards commented on Aug 2, 2026

   

   Brad-Edwards

   on Aug 2, 2026

   CollaboratorAuthor

   More actions

   gc workflow phase recorded: preflight (issue #34). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.
7. Brad-Edwards commented on Aug 2, 2026

   

   Brad-Edwards

   on Aug 2, 2026

   CollaboratorAuthor

   More actions

   gc workflow phase recorded: plan (issue #34). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

   Implementation plan — feat(nasim): conformant simulator backend target (#34)

   Implements raes_adapters.nasim.backend as a conformant RAES backend over the admitted NASIM_TINY selection (nasim==0.12.0, gymnasium==0.26.3, numpy==1.26.4), honoring docs/decisions/nasim-backend-guardrails.md (architecture preflight). The qualified source, protocol, authored SDL, experiment task/spec, source ledger and loss disclosures already exist (#32/#33); this issue adds the executable backend.

   Approach: adapt the CyberBattleSim gym-backend pattern, not cyborg's flat layout

   NASim and CyberBattleSim (CBS) are both index-published gym backends using the shared _scenario_ledger.ScenarioLedger. nasim/backend/ mirrors cyberbattlesim/backend/'s shape (one private driver shared by provisioner/participant/evaluator), with NASim-specific semantics plus two additions the guardrails require: a logical-time component and global-NumPy RNG isolation.

   Duplication control (SonarCloud new_duplicated_lines_density gate) — shared extraction, refactor CBS

   Copying CBS's backend verbatim would trip the duplication gate. Per the guardrails ("extract only a backend-neutral private verifier"; "do not add a third copied diagnostic-address helper"), the backend-neutral mechanical blocks move into shared private modules consumed by both CBS and nasim (CBS refactored to consume them; its existing test suite pins behavior):

   • src/raes_adapters/_source_admission.py — the installed-distribution / artifact / import-root-tree / direct-archive-provenance / module-origin verifier (the ~470-line block). Backend-neutral: takes the qualification dict + package/import-root names; expected roots/versions/digests stay backend-local evidence.
   • src/raes_adapters/_diagnostics.py — diagnostic_address + JSON-Pointer escaping (shared; CBS backend/_diagnostics.py and nasim re-export it).
   • src/raes_adapters/_experiment_evidence.py — backend-neutral builders for the evaluator's ExperimentEvidenceRecordModel / ExperimentDerivedMeasureModel / ExperimentCaptureSpecModel from a sanitized summary (step_count, cumulative_reward, projection_ref, task_id, backend name, limitations). Reuses the contract shapes, not either backend's semantics.

   Backend-semantic code (capability declarations, diagnostic codes, action mappings, resource types, terminal-cause values, limitation text) stays nasim-local.

   New files (src/raes_adapters/nasim/backend/)

   • driver.py — NasimDriverProtocol + NasimDriver (the redaction boundary). construct() lazily imports nasim/gymnasium/numpy, runs _source_admission against qualification.json identities, verifies the Tk prerequisite, then nasim.make_benchmark("tiny", fully_obs=True, flat_actions=True, flat_obs=True, render_mode=None). Native env / obs vector / action space / info never leave. Sanitized frozen dataclasses: DriverResetReport, DriverStep, DriverEvaluation, DriverCleanupReport. terminal_cause ∈ {"goal","step-limit"}. Both terminated and truncated retained; on simultaneous truth, goal precedence for participant completion, truncation kept as evaluator evidence + limitation (no fact-erasing if/elif). Global-NumPy RNG isolation: a process-wide lock + driver-private saved numpy.random.get_state(), restoring caller state in finally; reset seeds the driver's saved state with the admitted seed AND calls env.reset(seed=), reporting the two streams (global-NumPy action-success vs gym-reset) separately. Action resolution: portable kind → first unambiguous native FlatActionSpace action of that class; unsupported/ambiguous/unavailable fails before mutation; flat indices / target tuples stay private.
   • manifest.py — create_nasim_manifest() → BackendManifest with backend-manifest-v2 in realization_support.disclosure_kinds, TimeCapabilities, and NASim-local capability declarations. Constraints and loss descriptors state the native source is fully observed and the RAES view is a lossy default-deny projection. supported_contract_versions derived from RAES profiles + capability models.
   • target.py — create_nasim_components(*, driver, seed) / create_nasim_target(*, driver=None, seed=None) via build_runtime_target. Components: provisioner, orchestrator, evaluator, participant_runtime, time_runtime=ReferenceTimeRuntime().
   • provisioner.py — admits only the compiled selected static scenario; rejects unsupported resources before construction; preserves ownership on failure for retryable cleanup.
   • orchestrator.py — portable workflow lifecycle only; never steps the driver.
   • participant_runtime.py — BaseParticipantRuntime subclass; one admitted portable action contract → one native env.step(); advances the logical clock one tick per committed step (apply_logical_clock_transition / time_runtime.advance), no advance on reject, reset starts a new logical segment; truncation guarded at max_steps=1000; observation envelope is a default-deny lossy projection (native vector / index / info / host-truth / evaluator facts withheld); one adapter-generated driver-operation ref per action; typed terminal result joined to participant / episode / action-instance / contract / generation / observation-point. Failed projection after a native mutation quarantines the episode (no false success).
   • evaluator.py — read-only projection of committed driver facts (never advances/resets); objective/proposition truth kept distinct from reward; evidence record + cumulative-reward derived measure + capture spec via _experiment_evidence; limitations disclose stochastic-bounded reproduction and the retained truncation fact.
   • cleanup.py — execute_nasim_cleanup over base execute_cleanup; deterministic, idempotent, failure-preserving driver close/verify/reset dispatch.
   • conformance.py — run_nasim_conformance (base run_conformance_probe), source-protocol diagnostics via the ledger, capability-evidence pointer helpers.
   • __init__.py — eager backend exports; conformance stays directly importable.

   Edits to existing files

   • src/raes_adapters/nasim/scenario_ledger.py — add module-level EvidenceSelection, load_loss_disclosures = LEDGER.load_loss_disclosures, validate_all = LEDGER.validate_all for the backend importers (mirrors CBS scenario_ledger). No change to the pinned digest or ledger data.
   • src/raes_adapters/cyberbattlesim/backend/{driver,_diagnostics,evaluator}.py — refactor to consume the three new shared modules (behavior-preserving; pinned by the existing CBS suite).
   • pyproject.toml — add nasim backend files to the ruff UP004 per-file-ignores (the class X(object) pattern the repo uses).
   • noxfile.py _distributions() — add a hermetic (driver=None, no native execution/import) nasim conformance probe mirroring the CBS clean-wheel probe; no live-Tk requirement added to the base graph.
   • docs/decisions/nasim-backend-guardrails.md, docs/index.md, mkdocs.yml — the preflight-added guardrails doc + nav entry (already written on this branch).

   Tests (TDD, tests/test_nasim_*.py) — fake driver, no real simulator

   • test_nasim_backend.py — mirrors test_cyberbattlesim_backend.py with a FakeDriver: dependency-light lazy import (no nasim/gymnasium at import); manifest/target claimed surfaces incl. has_time; provisioner reconcile + unsupported-rejection-before-construct; one-admitted-action → one-driver-op → typed-terminal-result; unsupported action rejected before mutation; positive + negative disclosure (native vector / index / info / evaluator-only sensitive-host truth never enter participant views/histories); evaluator distinct facts without advancing/leaking; both terminated and truncated retained incl. simultaneous-truth goal-precedence; global-NumPy seed dispositions reported separately; termination closes episode + restart resets same seed; cleanup idempotent/verified/failure-preserving; selected scenario realizes and runs across surfaces via ReferenceProcessor / RuntimeControlPlane; canonical conformance probe passes.
   • test_nasim_driver.py — monkeypatched nasim/gymnasium/numpy (no real sim): lazy construct, source-admission rejection (tampered tree / injected extension / editable install / wrong module origin), global-NumPy save/restore isolation, exactly one native step per action, terminated=goal / truncated=step-limit mapping, terminal_cause, idempotent close.
   • test_nasim_source_admission.py — the shared verifier's negative/positive paths (also covers the CBS refactor).
   • Existing test_nasim_qualification.py and the CBS/cyborg suites stay green.

   Verification, plan-rules, and gates

   • Single distribution / one uv.lock; nasim extra + qualified pins already present; no tool.uv.conflicts needed (documented). Base-only install unaffected; simulator imports lazy so base + each extra typecheck/test alone.
   • raes_adapters.base stays semantics-free; no CAGE/NASim SDL/schema/profile/vocabulary/policy added; no native objects leak into portable artifacts.
   • No CHANGELOG.md edit / no changelog.d/ fragment (Release Please owns it); the PR title will be a valid Conventional Commit (feat(nasim): ...).
   • Requirement-free run (make verify / make policy with --skip-requirement); the full nox graph (hygiene, policy, lint, tool-tests, per-extra typecheck+test, distributions, docs) must pass; coverage ≥ 80%.

   Manual integration test (acceptance criterion 8, done by hand after gates are green)

   On the qualified installed runtime with Tk present, construct the default live create_nasim_target(), drive reset → a representative admitted step → source termination → verified cleanup through the RAES control plane, recording only reward, both terminal booleans, step/control dispositions, and cleanup outcome. Bounded operational evidence; not a portable native-state dump or a replay/equivalence claim.

   ---

   Operational note: this run is executing under the documented GC#1502 manual-drive workaround — the mechanical /implement MCP tooling is blocked by a stale launch-time identity pin in this linked worktree (concurrent run in the sibling checkout), so branch/commit/push/PR/reviews are driven by hand. The workflow gates (preflight, plan, TDD, completion, reviews, CI, Sonar) are all still honored.
8. Brad-Edwards commented on Aug 3, 2026

   

   Brad-Edwards

   on Aug 3, 2026

   CollaboratorAuthor

   More actions

   Pre-push review record (manual, GC#1502 workaround)

   The automated gc_codex_review_cycle / gc_test_quality_review_cycle are blocked by the stale launch-time identity pin in this linked worktree (implement_repo_identity_changed), the same GC#1502 condition that blocked bootstrap/pickup. The architecture preflight (read-only) ran; the review-cycle wrappers (which bind the pinned identity for decision-record posting) do not. Substituting an independent adversarial review of the committed branch diff.

   Dimensions verified clean: global-NumPy RNG isolation (save/install/restore under the process lock, caller state restored in finally); NASim wheel-tree admission (verified via runtime_artifacts roots, not the 42-file git runtime_source_tree; editable/VCS/symlink/digest/version rejected); terminated/truncated retained distinctly with goal precedence; capability-declaration sharing has no cross-backend value bleed (provisioner stays backend-local); CyberBattleSim refactor behavior-preserving (its suite still green, evaluator-cutoff semantics preserved for CBS and correctly not inherited by NASim); driver-contract redaction sound (no native object / flat index / observation vector / host state / info crosses).

   Findings addressed (all fixed this run):

   1. (correctness / guardrail) A supplied-but-unparseable participant target silently degraded to the first native action of the class (wrong host). Now a supplied target that does not resolve to a native coordinate fails before mutation (source_transition=False → action-unavailable); added test_live_driver_fails_before_mutation_on_unresolvable_target.
   2. (hardening / guardrail) Top-level module-origin verification (nasim/gymnasium/numpy) now runs BEFORE the native import (find_spec does not execute a top-level module), closing the import-shadowing window; submodule-origin checks remain after import (their file contents are digest-verified beforehand).
   3. (guardrail) Per-run terminal facts were dropped at the evaluator projection, leaving only a static limitation. EvaluatorSummary and the evidence payload_summary now retain terminated/truncated/terminal_cause, so truncation that fires alongside a goal survives as evaluator evidence; asserted in the evaluator test.
   4. (test quality) Strengthened the live-driver test to assert the native observation/info payload the fake environment emits does not survive into the sanitized driver outputs (a real redaction path, versus the backend fake's inert decoy).

   Local gates green after fixes: ruff, ruff format, mypy (strict, base + nasim), full pytest suite (85% coverage), make policy, tool unittests, mkdocs --strict, and the hermetic nasim clean-wheel distribution conformance probe. The cyborg distribution probe (needs raes-env-packs from the index) and SonarCloud run in CI.
9. Brad-Edwards commented on Aug 3, 2026

   

   Brad-Edwards

   on Aug 3, 2026

   CollaboratorAuthor

   More actions

   Ready for review and merge (Phase D complete)

   PR #71 (feat(nasim): implement the conformant simulator backend target → dev) is green and mergeable.

   All CI checks pass: Fast checks (hygiene + lint), Policy, Typecheck, Tests, Tool tests, Distributions (incl. the new hermetic nasim clean-wheel conformance probe and the cyborg/cyberbattlesim probes), Docs, CodeQL, GitGuardian, PR-title lint, and the aggregate PR Gate.

   SonarCloud quality gate: passed — new_violations 0, new coverage 88.4%, new duplicated-lines density 0.2%, reliability/security/maintainability all A, security hotspots reviewed 100%.

   What shipped: raes_adapters.nasim.backend over the admitted NASim tiny selection, a neutral shared _gym_backend framework (with CyberBattleSim refactored onto it, behavior preserved), and the supporting tests/probes — per docs/decisions/nasim-backend-guardrails.md. The plan, the manual pre-push review record, and the Sonar-fix follow-up are on this thread. dev (#70 PrimAITE) was merged in.

   Plain-English outcome: installing raes-adapters[nasim] now yields a runnable, conformant RAES backend for the NASim tiny benchmark: the RAES control plane can provision the scenario, admit red attacker actions one native step at a time, and read evaluator reward/goal facts — while the native simulator, its observation vector, action indices, and host state stay behind the driver, and goal termination vs step-limit truncation are reported as distinct facts.

   This run is under the documented GC#1502 manual-drive workaround, so this readiness record is posted by hand rather than through the mechanical gc_assert_completion gate. Merging is the single human touchpoint — the requirement transition, traceability reconciliation, post-merge report, and issue close (Phase E) run when /implement 34 is re-invoked after the merge.

   Acceptance criterion 8 (honest live NASim execution) remains a bounded manual check performed on the qualified runtime with Tk present, post-merge.
10. Brad-Edwards commented on Aug 3, 2026

    

    Brad-Edwards

    on Aug 3, 2026

    CollaboratorAuthor

    More actions

    Acceptance criterion 8 — honest live NASim execution (done)

    Ran the adapter against a live installed NASim on the qualified runtime (CPython 3.12.3, nasim==0.12.0 / gymnasium==0.26.3 / numpy==1.26.4 cp312, tkinter present) — a hands-on check beyond the fake-driver suite. Driven through the backend's NasimDriver (the component that runs the real simulator).

    This surfaced two real bugs in the shared installed-source verifier that the fake-driver tests could never catch (they used trees the verifier itself constructed), and which would have made the adapter refuse to construct against any real install. Both are fixed at the root:

    1. _runtime_root_paths counted __pycache__/*.pyc files, which are absent from the recorded clean tree ("pycache and *.pyc excluded") — so any imported runtime failed on file count/digest. Now excluded.
    2. _installed_tree_digest hashed each file by its root-prefixed path, but the qualification records the digest over root-relative names — so the tree digest never matched a real install. Verified by reproducing the recorded nasim root digest (813821d5…) with relative names. Now relative. The CyberBattleSim fake-driver test and the nasim driver test encoded the same prefixed convention (which hid the bug); both were corrected, and a __pycache__ decoy was added so the exclusion is regression-tested.

    Live run evidence (after the fixes):

    • Construct/admission: real source admission passed (versions, complete import-root trees, direct-archive provenance, module origins) and make_benchmark('tiny', fully_obs=True, flat_actions=True, flat_obs=True) built the env; flat action space n=18 (ServiceScan/OSScan/SubnetScan/ProcessScan/Exploit/PrivilegeEscalation × hosts (1,0)/(2,0)/(3,0)).
    • Reset: seed 20260802 bound both streams (numpy-global-action-success, gym-environment-reset).
    • Representative step: service-discovery → first ServiceScan target (1,0) → reward −1.0, terminated=False, truncated=False, step 1 — matches the qualification smoke record exactly.
    • Goal termination: driving the attack path (pivot host-1-0 → sensitive hosts (2,0),(3,0)) reached the goal in 27 steps, cumulative reward 173.0, terminated=True, truncated=False, terminal_cause=goal.
    • Step-limit truncation: repeating an unproductive action hit the 1000-step limit → terminated=False, truncated=True, terminal_cause=step-limit — the two terminal facts stay distinct on the real sim.
    • Cleanup: verified idempotent close in every case (closed=True, verified=True, verify_closed()=True).

    The admission fixes are being pushed to PR #71 (they change only the shared verifier + two test helpers; all gates re-run).
11. Brad-Edwards commented on Aug 3, 2026

    

    Brad-Edwards

    on Aug 3, 2026

    CollaboratorAuthor

    More actions

    Final report — shipped (Phase E)

    Both PRs are merged into dev:

    • feat(nasim): implement the conformant simulator backend target #71 feat(nasim): implement the conformant simulator backend target — the backend + shared _gym_backend framework (with CyberBattleSim refactored onto it).
    • fix(base): admit real installed source trees in the shared verifier #72 fix(base): admit real installed source trees in the shared verifier — the two source-admission fixes the honest live run surfaced.

    Acceptance criteria — all met:

    • The selected RAES scenario constructs and runs through all applicable backend surfaces.
    • Manifest capabilities have executable evidence; unsupported facts are rejected or disclosed.
    • Every admitted action joins one driver operation, a typed terminal result, and participant/episode identity.
    • Observation and evaluator-only boundaries pass positive and negative disclosure tests.
    • Reset, seed/control reporting, termination, and cleanup match the selected source protocol.
    • Portable artifacts contain only RAES contract data and stable source references.
    • The canonical verification graph passes with this extra and with unrelated extras isolated.
    • Honest live NASim execution — on the qualified runtime: real admission → reset → representative step (reward −1.0, matches smoke) → goal termination (27 steps, reward 173.0) → step-limit truncation (1000 steps) → verified cleanup. Evidence and the two bugs it caught are recorded above.

    Traceability / requirements: requirement-free run (issue had no Requirements section); no new ADR-backed structural gate was introduced, so there are no requirement→artifact links to reconcile and no DRAFT→ACTIVE transition.

    Process: executed under the documented GC#1502 manual-drive workaround (mechanical /implement MCP tooling blocked by a stale identity pin in this linked worktree); plan, architecture preflight, review, readiness, live-run evidence, and this report were driven by hand. The gated pipeline (preflight → plan → TDD → completion → review → CI → Sonar → readiness → merge) was honored throughout.

    Closing as complete.
12. Brad-Edwards

    closed this as completedon Aug 3, 2026
13. Brad-Edwards

    removed

    in-progressAn agent is actively working this issue via /implementAn agent is actively working this issue via /implement

    on Aug 3, 2026