You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(nasim): add the example environment pack and researcher workflow #36
Provide one validated NASim environment pack and a no-code researcher workflow for the selected case.
Scope
Add environments/nasim-tiny/ from the published raes-env-packs template.
Include pack identity, the canonical SDL, provenance/licensing/safety review, compatibility claims, associated-artifact identity where applicable, source-ledger references, and clear participant/operator/evaluator boundaries.
Integrate the adapter into the repository's installed researcher command so a clean environment can inspect identities, validate the pack, run one episode or seeded batch, and retain a portable artifact inventory.
Provide a reference build, automated rehearsal, and matching researcher walkthrough.
Keep runtime implementation selection and simulator-native configuration outside pack semantics.
Mark draft, built, or golden only according to evidence actually produced.
Keep the pack out of the Python wheel; publish it as versioned repository/release content.
Acceptance criteria
raes-pack-validate and raes-pack-release check pass.
A clean installed environment can acquire the immutable pack, run the documented short path, and validate its outputs.
Build, tests, and walkthrough use the same scenario, controls, agent, objective, and evaluator.
Participant, operator, evaluator/restricted, native-source, and public artifact boundaries pass leak checks.
Provenance covers all upstream sources, exclusions, generated content, and redistribution decisions.
The command rejects mismatched pins/digests, unsafe output reuse, unsupported controls, and invalid participant bindings before execution.
The pack defines no RAES semantics and RAESystem/env-packs receives no scenario content.
Manual integration test — honest NASim execution: in a clean environment, run the researcher command's documented short path (inspect identities → validate pack → run one episode/seeded batch) against a live installed NASim (qualified runtime per build(nasim): qualify and pin the public source and experiment profile #32) and confirm the portable artifact inventory is produced — a hands-on end-to-end check beyond the automated rehearsal.
RAESystem/hub#10 has now established the federated catalog boundary for reusable example packs. Pack source belongs in an appropriate catalog repository (with RAESystem/reference-packs as the first-party curated catalog) under environments/<pack-id>/, using a history- and provenance-preserving one-way transfer. This adapter repository retains simulator realization and backend-native evidence, then consumes the released pack identity; it should not retain a competing editable pack copy. The upstream format, validation, release, provenance, trust, and compatibility contracts remain owned by RAESystem/env-packs.
🛠️ Picked up by /implement — driver claude, branch 36-nasim-env-pack, 2026-08-04T20:46:48Z. Scope confirmed with maintainer: NASim example pack lives in-repo beside the nasim adapter (mirroring cyborg/examples/cage2-research), conforming to the published OpenRAE/env-packs (raes-env-packs) format — no reinvented RAE SDL/mechanics or competing pack format.
Implementation plan — NASim example environment pack + researcher workflow
Requirement-free run (no in-scope requirement UID; nox -s verify -- --skip-requirement). Mirrors the shipped CyBORG sibling (src/raes_adapters/cyborg/examples/cage2-research/ + cyborg/researcher.py), conforming to the published raes-env-packs (OpenRAE/env-packs) format and RAE SDL — no reinvented format/semantics/mechanics.
Scope decisions (maintainer-confirmed)
Pack lives in-repo beside the nasim adapter at src/raes_adapters/nasim/examples/nasim-tiny/ (in-wheel, like CyBORG). The issue body's environments/… + "out of the wheel" wording is superseded by this decision; flagged in the PR.
Format/tooling owned by raes-env-packs; the pack consumes it and adds nothing to raes_adapters.base and no CAGE/RAES semantics.
Key finding from assessment
The shipped CyBORG pack passes only the library validate_pack() API the CLI calls — it failsraes-pack-validate (missing golden checklist) and is skipped by raes-pack-release check (no compatibility manifest). Satisfying #36's acceptance criteria is therefore net-new work beyond mirroring CyBORG: the NASim pack adds docs/golden-readiness-checklist.md + pack.compatibility.yaml (artifact_boundaries), and the rehearsal actually runs both pack CLIs.
pack.yaml — manifest (name nasim-tiny; provenance/associated-artifact/compatibility pointers; status set to evidence actually produced).
pack.content-manifest.json — derived via raes_env_packs.derive_pack_content_manifest; its set_digest is the pinned --pack-digest.
pack.compatibility.yaml — net-new (no CyBORG precedent); artifact_boundaries so raes-pack-release check treats the pack as releasable.
sdl/nasim-tiny.sdl.yaml — copy of the authored nasim/scenario/nasim-tiny.sdl.yaml (RAES-valid, pinned digest).
experiment/nasim-tiny.spec.exp.json / nasim-tiny.task.exp.json — JSON ports of the existing YAML spec/task, pack:/… artifact refs byte-bound to the participant files.
participant/nasim-red-bruteforce.{manifest,selection,configuration}.json — the red bruteforce attacker (role red, backend nasim-tiny); withheld_refs names the NASim native-state markers.
New src/raes_adapters/nasim/researcher.py — NASim RunControls/execute_episode/archival_run/archival_collection over create_nasim_target() + the exported Nasim runtime components (red-attacker semantics; metrics steps_to_termination, cumulative_attacker_reward, goal_reached, terminal_cause; NASim logical clock).
New src/raes_adapters/nasim/runtime_plans.py — NASim orchestration/evaluation/time plan builders (analog of cyborg/runtime_plans.py).
src/raes_adapters/nasim/__init__.py (or backend/driver.py) — expose public verify_selected_nasim_source() wrapping the existing _source_admission calls, for the CLI native gate.
src/raes_adapters/cli.py — backend-parameterized refactor. Extend --backend to nasim-tiny; resolve pack-root/example-members, inspection payload, native-import + source-verify gate, and RunControls/execute-episode dispatch per backend, keeping NASim's red-attacker semantics inside raes_adapters.nasim (per the CyBORG guardrails' extension seam). Must not regress the CyBORG path.
Rehearsal + walkthrough + guardrails
noxfile.py_distributions — add NASim clean-install inspect/validate/run blocks mirroring CyBORG, plusraes-pack-validate --pack … and raes-pack-release check --pack … (env-packs console scripts, via the cyborg/env-packs dist available to the rehearsal env).
tests/test_nasim_researcher_cli.py — subprocess acceptance suite mirroring tests/test_researcher_cli.py: every exit class, output reserve/reuse, symlink/traversal escape, hostile native stdout/stderr suppression, duplicate-JSON-key rejection, tamper/stale-checksum rejection, participant-binding mismatch, full study seal + exact inventory. Native-leakage invariants over NASim outputs (no host paths, no secret sentinel, no Traceback, relative inventory paths, sha256 len 64).
Boundaries / determinism disclosure
Participant/operator/evaluator/native-source/public-artifact boundaries enforced via the participant selection.jsonexposure_policy (disclosed vs withheld) + the scenario-ledger native-leakage scan. The ADR-069 determinism loss (unbound global-NumPy action RNG; qualification.json known_defects) stays disclosed in provenance and non-claims — never erased.
Verification
uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify -- --skip-requirement (hygiene, policy, lint, tool-tests, typecheck, base + nasim tests incl. the new researcher CLI suite, distributions clean-wheel proof now running the NASim pack + pack CLIs, strict docs). Single pyproject.toml/uv.lock; no CHANGELOG.md edit (Release Please owns it); PR title feat(nasim): ….
Manual integration test (final acceptance criterion)
In a clean environment, run the researcher command's documented short path (inspect identities → validate pack → run one episode/seeded batch) against live installed NASim (qualified runtime per #32) and confirm the portable artifact inventory is produced. If NASim can't install/run, report a failed/blocked manual-readiness result — never a silent fall back.
Station observation — codex review cycle 1 — Opened
Observed state: the station ran but rendered no verdict, so this gate is unobserved. Failure class:incomplete_reviewer_coverage Attempt: 1 Impact: the review cap was not consumed and no findings record exists for this cycle. Current obligation: observe this station before the run completes.
This is a missing observation, not a defect. It is resolved by observing the station, not by
dispositioning a finding.
gc_codex_review — cycle 1 of 1 (pre-push) on issue #36 (branch 36-nasim-env-pack) Diff mode: inline — the complete diff was supplied in one prompt
Core review
Verdict:ship-with-fixes
The change is mostly shaped correctly: the installed shell grows a closed per-backend adapter seam, NASim semantics stay inside raes_adapters.nasim, and the implementation composes RAES contracts, raes_env_packs, runtime planning, participant configuration, diagnostics, and conformance report helpers instead of inventing local schemas. The obvious extension path is still published pack/participant/backend artifacts plus another adapter entry, not a cross-backend semantic layer. The one production-readiness issue is in the NASim runtime lifecycle: startup/provision/orchestration work happens before the cleanup guard, which cuts across the repo's explicit cleanup invariant.
Blocking findings (1):
[one-off] Runtime startup failures bypass cleanup — src/raes_adapters/nasim/researcher.py:625 execute_episode calls _start_runtime before entering the try/finally that runs RuntimeManager.destroy(). _start_runtime plans, applies provisioning, and starts orchestration; if provisioning or orchestration partially allocates the target and then fails, the exception returns to the CLI without the verified cleanup path. The researcher command contract says cleanup is always attempted and a failed cleanup cannot be downgraded, so startup needs to be inside a cleanup-protected region or _start_runtime needs to destroy the target on every partial-start failure.
Security review
Verdict:ship
This change is shaped as a closed per-backend adapter dispatch for the installed local command: NASim-specific participant admission, pack validation, source verification, native execution, and evidence archival stay behind the NASim module while the shared CLI owns mode dispatch and output reservation. The security-relevant boundaries are the closed argparse surface, pack child confinement through the pack validator and _pack_child, installed-source verification before native import, native stdout/stderr suppression, and bounded error envelopes. I do not see a concrete exploitable security regression introduced by this diff.
Station observation — codex review cycle 1 — Re-observed
Disposition: reobserved Corrective action: a bounded automatic re-attempt observed the station and it rendered a verdict. Evidence:#36 (comment)
Verification
The linked findings record is this station's validated verdict for this logical cycle.
Re-observation closes only the missing-observation obligation. Any findings in that record
remain subject to the existing fix / wontfix / not-applicable rules.
gc_codex_review pre-push cycle 1 of 1 complete for issue #36 on branch '36-nasim-env-pack'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.
Review decision record — codex cycle 1 (issue #36)
Reviewer: codex Cycle: 1
Architectural read:
Core reviewer: The change is mostly shaped correctly: the installed shell grows a closed per-backend adapter seam, NASim semantics stay inside raes_adapters.nasim, and the implementation composes RAES contracts, raes_env_packs, runtime planning, participant configuration, diagnostics, and conformance report helpers instead of inventing local schemas. The obvious extension path is still published pack/participant/backend artifacts plus another adapter entry, not a cross-backend semantic layer. The one production-readiness issue is in the NASim runtime lifecycle: startup/provision/orchestration work happens before the cleanup guard, which cuts across the repo's explicit cleanup invariant.
Security reviewer: This change is shaped as a closed per-backend adapter dispatch for the installed local command: NASim-specific participant admission, pack validation, source verification, native execution, and evidence archival stay behind the NASim module while the shared CLI owns mode dispatch and output reservation. The security-relevant boundaries are the closed argparse surface, pack child confinement through the pack validator and _pack_child, installed-source verification before native import, native stdout/stderr suppression, and bounded error envelopes. I do not see a concrete exploitable security regression introduced by this diff.
Rationale:execute_episode calls _start_runtime before entering the try/finally that runs RuntimeManager.destroy(). _start_runtime plans, applies provisioning, and starts orchestration; if provisioning or orchestration partially allocates t…
gc_test_quality_review cycle 1 of 1 complete for issue #36 on branch '36-nasim-env-pack'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next gc_test_quality_review invocation to count cycles.
Review decision record — test-quality cycle 1 (issue #36)
Reviewer: test-quality Cycle: 1
Architectural read:
This PR adds the NASim researcher-command acceptance suite (test_nasim_researcher_cli.py) plus updated qualification golden-record tests (test_cyborg_qualification.py, test_nasim_qualification.py) and the nasim-tiny experiment authoring fixture. The CLI test suite deliberately mirrors the existing, previously-shipped CybORG suite's shape: a FakeNasimDriver injected at the NasimDriverProtocol seam so the standard test job never imports the real nasim/Tk dependency, while every admission/tamper/escape path is driven end-to-end through the real cli.main() and the real production validators (raes_env_packs.validate_pack/verify_pack_content_digest, strict duplicate-key JSON loading, _pack_child traversal confinement, spec-artifact byte/digest binding, participant manifest/selection/configuration digest joins). This is the correct seam: native simulator construction/import is already covered by the pre-existing, out-of-scope test_nasim_driver.py, so mocking only resolve_selected_distribution / _verify_pre_import_source here is a boundary choice, not a shortcut around the security gate — the gate itself (duplicate-key rejection, traversal rejection, digest/checksum mismatch rejection, symlink-escape rejection) is exercised for real and asserted by exit code plus absence of any created output directory, which is the right assertion shape for a fail-closed contract. The tampered-bytes tests correctly re-derive the pack-level content digest with the real derive_pack_content_manifest so only the intended layer (spec-level artifact-byte binding) is what fails, isolating exactly what's under test. No test-visible production special-casing on fixture names/paths was found, and the qualification tests are consistent golden-record checks matching the pre-existing sibling pattern. I'd ship this.
Manual integration test — honest live-NASim execution (issue #36 final acceptance criterion)
Result: PASS. In a clean installed environment, the researcher command's documented short path was run against a real installed NASim — not a stub/mock, no fall back to an injected driver.
Environment
Clean venv: raes-adapters[nasim] installed from the built wheel, Python 3.12.3 (the qualified runtime's Python), live nasim==0.12.0. _tkinter present; import nasim succeeds.
raes-adapters validate --backend nasim-tiny --mode study (packaged example, all pins) → {"disposition":"validated","pack":"admitted","participant":"nasim-red-bruteforce","run_count":1}.
raes-adapters run --backend nasim-tiny --mode study … --output live-episode (one seeded native episode, seed 20260802, trial-length 1000) → {"disposition":"succeeded","inventory":"inventory.json","mode":"study","run_count":1}.
Portable artifact inventory (bounded evidence — no native dump)
All inventory paths relative; every checksum a 64-char sha256; study.json present.
Leakage oracle over every produced JSON: no host/tmp path, no Traceback, no native observation/action/state. The only NASim native symbol names present (servicescan, osscan, subnetscan, processscan, flatactionspace) appear solely under the participant selection's exposure_policy.withheld_refs — the default-deny declaration of what is withheld, carried in provenance exactly as the design requires — not in any observation, action, reward, or host-truth field.
Distinct claims preserved. The report distinguishes backend conformance from source-protocol reproduction and research claims; the ADR-069 determinism loss (unbound global-NumPy action RNG) stays disclosed as a non-claim, never erased.
Researchers get a ready-to-run NASim example: a clean pip install raes-adapters[nasim] can inspect identities, validate the packaged nasim-tiny environment, and run one episode/seeded batch to a sealed portable evidence bundle — no simulator wiring, and no native details leak out. The pack conforms to the published env-packs format (validated by both pack CLIs), and adding future simulators now reuses one shared researcher/CLI seam instead of copied boilerplate.
Add the in-repo NASim nasim-tiny example environment pack and wire it into the installed raes-adapters researcher command via a closed per-backend CLI adapter, with shared researcher/plan helpers extracted from the cyborg sibling.
codex: Pre-push 1/1: ship-with-fixes, security ship. F1 (startup ran before the cleanup guard, so a partial start bypassed cleanup) fixed: startup moved inside try/finally + a regression test. Coverage 30/30 files.
test-quality: Pre-push 1/1: clean, 0 findings.
Traceability reconciliation
Pending — requirement status transition and IMPLEMENTS/TESTS reconciliation run in Phase E once the PR merges.
Status
CI: ✅ green
SonarCloud: ✅ passed
PR ready for user review and merge. Ground Control reconciliation (requirement status + traceability) runs on merge (Phase E).
Kept open after evidence audit. PR #76 (961ff2fe9989d730663162c0e4313650592b2f25) merged the automated pack, researcher workflow, validators, clean-install rehearsal, and tests. Brad recorded an explicit live-NASim PASS in comment 5188079000, but that run predates the final refactor commits 17832bb / a0c064f and the merge. The PR final test plan also states that the manual live-NASim short path remained to be run. The missing proof is therefore for the exact final merged/released artifact, not proof that the manual path was never exercised. Because the final-artifact acceptance criterion is unverified, #36 remains open. The stale in-progress label has been removed; no experiment was started during this freeze-safe pass.
Live NASim short-path run on the current dev tip (informational evidence, no code change).
Commit:0272949fa964920a7e06fb56f6d7051054756b5b (origin/dev, unchanged since 2026-08-19). It descends from the #76 merge 961ff2f. It also contains #90 (8db4fbd, 2026-08-12), which added the evidence-admission gate seen below. Run on 2026-10-09 between 06:01 and 06:05 UTC.
Environment and versions
A linux/amd64 ubuntu:24.04 container (24.04.5) under Docker Desktop on an arm64 Mac, so x86_64 is emulated. glibc 2.39.
System Python 3.12.3 with python3-tk (Tk 8.6). PYTHONPATH unset, PYTHONSAFEPATH=1, uv 0.12.1.
raes-adapters is not on PyPI (the JSON API returns 404). The wheel was therefore built from a git archive of the commit, following the distributions nox session: raes_adapters-0.0.0-py3-none-any.whl, sha256 068871d1866b944cf65a695f5c49c1a6c8705b334aea565a54b19819d341ebc6. A host build produced the same digest.
Commands
The CLI lines come from docs/nasim-researcher-command.md. <doc args> stands for that page's unchanged --pack nasim-tiny … --trial-length 1000 --seed 20260802 --run-id nasim-tiny-run block.
ENVIRONMENT-PACK CONTENT CI: PASS, including [ok] anti-extension guard
raes-pack-release check
0
PACK RELEASE GATE: PASS
The conformance root contains exactly index.json, inventory.json and runs/nasim-pr-seed-20260802/conformance/backend-conformance.json.
The inventory paths are relative, and each sha256 is 64 hex characters.
index.json records native_conformance: false, 7 declared weaknesses and the non-claim "A probe over the installed source is not a native attacker episode."
A grep of every produced file found no host path, Traceback or site-packages string.
On the host itself (macOS arm64, CPython 3.12.13), the same wheel reports "native_available": false. verify_runtime_artifacts rejects numpy, because the qualified artifact in nasim/qualification.json is the manylinux_2_17_x86_64 wheel tree.
Against the acceptance criteria, at 0272949
Satisfied:raes-pack-validate and raes-pack-release check pass.
Partly evidenced: clean install and acquisition of the packaged pack.
The documented validate reaches the evidence gate. _admit_native_run raises that gate only after the pack digest, content digest, SDL/spec/task/participant loading and binding checks have been admitted.
Rejection before execution was observed for a mismatched pack digest and for output reuse. Unsupported controls and invalid participant bindings were not exercised in this run.
The leak check covers the conformance bundle only.
Not satisfied: the manual live-NASim criterion.
validate, run --mode smoke and run --mode study exit 3 before runtime planning. No native episode ran, and no episode inventory was produced.
Activity
Brad-Edwards commentedon Jul 29, 2026
RAESystem/hub#10 has now established the federated catalog boundary for reusable example packs. Pack source belongs in an appropriate catalog repository (with
RAESystem/reference-packsas the first-party curated catalog) underenvironments/<pack-id>/, using a history- and provenance-preserving one-way transfer. This adapter repository retains simulator realization and backend-native evidence, then consumes the released pack identity; it should not retain a competing editable pack copy. The upstream format, validation, release, provenance, trust, and compatibility contracts remain owned byRAESystem/env-packs.Brad-Edwards commentedon Aug 4, 2026
🛠️ Picked up by /implement — driver claude, branch
36-nasim-env-pack, 2026-08-04T20:46:48Z. Scope confirmed with maintainer: NASim example pack lives in-repo beside the nasim adapter (mirroring cyborg/examples/cage2-research), conforming to the published OpenRAE/env-packs (raes-env-packs) format — no reinvented RAE SDL/mechanics or competing pack format.Brad-Edwards commentedon Aug 4, 2026
Implementation plan — NASim example environment pack + researcher workflow
Requirement-free run (no in-scope requirement UID;
nox -s verify -- --skip-requirement). Mirrors the shipped CyBORG sibling (src/raes_adapters/cyborg/examples/cage2-research/+cyborg/researcher.py), conforming to the publishedraes-env-packs(OpenRAE/env-packs) format and RAE SDL — no reinvented format/semantics/mechanics.Scope decisions (maintainer-confirmed)
src/raes_adapters/nasim/examples/nasim-tiny/(in-wheel, like CyBORG). The issue body'senvironments/…+ "out of the wheel" wording is superseded by this decision; flagged in the PR.raes-env-packs; the pack consumes it and adds nothing toraes_adapters.baseand no CAGE/RAES semantics.Key finding from assessment
The shipped CyBORG pack passes only the library
validate_pack()API the CLI calls — it failsraes-pack-validate(missing golden checklist) and is skipped byraes-pack-release check(no compatibility manifest). Satisfying #36's acceptance criteria is therefore net-new work beyond mirroring CyBORG: the NASim pack addsdocs/golden-readiness-checklist.md+pack.compatibility.yaml(artifact_boundaries), and the rehearsal actually runs both pack CLIs.Pack —
src/raes_adapters/nasim/examples/nasim-tiny/pack.yaml— manifest (namenasim-tiny; provenance/associated-artifact/compatibility pointers;statusset to evidence actually produced).pack.content-manifest.json— derived viaraes_env_packs.derive_pack_content_manifest; itsset_digestis the pinned--pack-digest.pack.compatibility.yaml— net-new (no CyBORG precedent);artifact_boundariessoraes-pack-release checktreats the pack as releasable.sdl/nasim-tiny.sdl.yaml— copy of the authorednasim/scenario/nasim-tiny.sdl.yaml(RAES-valid, pinned digest).experiment/nasim-tiny.spec.exp.json/nasim-tiny.task.exp.json— JSON ports of the existing YAML spec/task,pack:/…artifact refs byte-bound to the participant files.participant/nasim-red-bruteforce.{manifest,selection,configuration}.json— the red bruteforce attacker (role red, backendnasim-tiny);withheld_refsnames the NASim native-state markers.docs/provenance-ledger.yaml(environment-pack-provenance/v3; NASim MIT source, attribution required + adapter-example source; review gates licensing/attribution/sensitive-data/offensive-tooling),docs/attack-path.md,docs/concepts.md,docs/golden-readiness-checklist.md.Researcher command
src/raes_adapters/nasim/researcher.py— NASimRunControls/execute_episode/archival_run/archival_collectionovercreate_nasim_target()+ the exported Nasim runtime components (red-attacker semantics; metricssteps_to_termination,cumulative_attacker_reward,goal_reached,terminal_cause; NASim logical clock).src/raes_adapters/nasim/runtime_plans.py— NASim orchestration/evaluation/time plan builders (analog ofcyborg/runtime_plans.py).src/raes_adapters/nasim/__init__.py(orbackend/driver.py) — expose publicverify_selected_nasim_source()wrapping the existing_source_admissioncalls, for the CLI native gate.src/raes_adapters/cli.py— backend-parameterized refactor. Extend--backendtonasim-tiny; resolve pack-root/example-members, inspection payload, native-import + source-verify gate, and RunControls/execute-episode dispatch per backend, keeping NASim's red-attacker semantics insideraes_adapters.nasim(per the CyBORG guardrails' extension seam). Must not regress the CyBORG path.Rehearsal + walkthrough + guardrails
noxfile.py_distributions— add NASim clean-installinspect/validate/runblocks mirroring CyBORG, plusraes-pack-validate --pack …andraes-pack-release check --pack …(env-packs console scripts, via thecyborg/env-packs dist available to the rehearsal env).docs/nasim-researcher-command.md+docs/decisions/nasim-researcher-command-guardrails.md(+mkdocs.yml/docs/index.mdnav — strict docs build gate).tests/test_nasim_researcher_cli.py— subprocess acceptance suite mirroringtests/test_researcher_cli.py: every exit class, output reserve/reuse, symlink/traversal escape, hostile native stdout/stderr suppression, duplicate-JSON-key rejection, tamper/stale-checksum rejection, participant-binding mismatch, full study seal + exact inventory. Native-leakage invariants over NASim outputs (no host paths, no secret sentinel, noTraceback, relative inventory paths, sha256 len 64).Boundaries / determinism disclosure
Participant/operator/evaluator/native-source/public-artifact boundaries enforced via the participant
selection.jsonexposure_policy(disclosed vs withheld) + the scenario-ledger native-leakage scan. The ADR-069 determinism loss (unbound global-NumPy action RNG;qualification.jsonknown_defects) stays disclosed in provenance and non-claims — never erased.Verification
uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify -- --skip-requirement(hygiene, policy, lint, tool-tests, typecheck, base +nasimtests incl. the new researcher CLI suite, distributions clean-wheel proof now running the NASim pack + pack CLIs, strict docs). Singlepyproject.toml/uv.lock; noCHANGELOG.mdedit (Release Please owns it); PR titlefeat(nasim): ….Manual integration test (final acceptance criterion)
In a clean environment, run the researcher command's documented short path (inspect identities → validate pack → run one episode/seeded batch) against live installed NASim (qualified runtime per #32) and confirm the portable artifact inventory is produced. If NASim can't install/run, report a failed/blocked manual-readiness result — never a silent fall back.
Brad-Edwards commentedon Aug 5, 2026
Station observation — codex review cycle 1 — Opened
Observed state: the station ran but rendered no verdict, so this gate is unobserved.
Failure class:
incomplete_reviewer_coverageAttempt: 1
Impact: the review cap was not consumed and no findings record exists for this cycle.
Current obligation: observe this station before the run completes.
This is a missing observation, not a defect. It is resolved by observing the station, not by
dispositioning a finding.
Brad-Edwards commentedon Aug 5, 2026
gc_codex_review — cycle 1 of 1 (pre-push) on issue #36 (branch
36-nasim-env-pack)Diff mode: inline — the complete diff was supplied in one prompt
Core review
Verdict:
ship-with-fixesThe change is mostly shaped correctly: the installed shell grows a closed per-backend adapter seam, NASim semantics stay inside
raes_adapters.nasim, and the implementation composes RAES contracts,raes_env_packs, runtime planning, participant configuration, diagnostics, and conformance report helpers instead of inventing local schemas. The obvious extension path is still published pack/participant/backend artifacts plus another adapter entry, not a cross-backend semantic layer. The one production-readiness issue is in the NASim runtime lifecycle: startup/provision/orchestration work happens before the cleanup guard, which cuts across the repo's explicit cleanup invariant.Blocking findings (1):
src/raes_adapters/nasim/researcher.py:625execute_episodecalls_start_runtimebefore entering thetry/finallythat runsRuntimeManager.destroy()._start_runtimeplans, applies provisioning, and starts orchestration; if provisioning or orchestration partially allocates the target and then fails, the exception returns to the CLI without the verified cleanup path. The researcher command contract says cleanup is always attempted and a failed cleanup cannot be downgraded, so startup needs to be inside a cleanup-protected region or_start_runtimeneeds to destroy the target on every partial-start failure.Security review
Verdict:
shipThis change is shaped as a closed per-backend adapter dispatch for the installed local command: NASim-specific participant admission, pack validation, source verification, native execution, and evidence archival stay behind the NASim module while the shared CLI owns mode dispatch and output reservation. The security-relevant boundaries are the closed argparse surface, pack child confinement through the pack validator and
_pack_child, installed-source verification before native import, native stdout/stderr suppression, and bounded error envelopes. I do not see a concrete exploitable security regression introduced by this diff.No blocking findings.
Brad-Edwards commentedon Aug 5, 2026
Station observation — codex review cycle 1 — Re-observed
Disposition: reobserved
Corrective action: a bounded automatic re-attempt observed the station and it rendered a verdict.
Evidence: #36 (comment)
Verification
remain subject to the existing
fix/wontfix/not-applicablerules.Brad-Edwards commentedon Aug 5, 2026
gc_codex_review pre-push cycle 1 of 1 complete for issue #36 on branch '36-nasim-env-pack'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Brad-Edwards commentedon Aug 5, 2026
Review decision record — codex cycle 1 (issue #36)
Reviewer: codex
Cycle: 1
Architectural read:
Blocking findings: 1
Finding 1 —
one-offF1src/raes_adapters/nasim/researcher.py:625execute_episodecalls_start_runtimebefore entering thetry/finallythat runsRuntimeManager.destroy()._start_runtimeplans, applies provisioning, and starts orchestration; if provisioning or orchestration partially allocates t…Brad-Edwards commentedon Aug 5, 2026
gc_test_quality_review cycle 1 of 1 — issue #36
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:
36-nasim-env-packCycle: 1 / 1
Findings: 0 (clean run)
Brad-Edwards commentedon Aug 5, 2026
gc_test_quality_review cycle 1 of 1 complete for issue #36 on branch '36-nasim-env-pack'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Brad-Edwards commentedon Aug 5, 2026
Review decision record — test-quality cycle 1 (issue #36)
Reviewer: test-quality
Cycle: 1
Architectural read:
Blocking findings: 0 (clean run)
Brad-Edwards commentedon Aug 5, 2026
Pre-PR base synchronization
refs/remotes/origin/devatec6fa3b0b38d4cc175e0abd5b41d6e315db65d18already_current034b3b7b6e0775e8680e7dd178adfd51c5408d53cf59ea04d98bb6ce56db63d523262fd5954d6d07Brad-Edwards commentedon Aug 5, 2026
Manual integration test — honest live-NASim execution (issue #36 final acceptance criterion)
Result: PASS. In a clean installed environment, the researcher command's documented short path was run against a real installed NASim — not a stub/mock, no fall back to an injected driver.
Environment
raes-adapters[nasim]installed from the built wheel, Python 3.12.3 (the qualified runtime's Python), livenasim==0.12.0._tkinterpresent;import nasimsucceeds.Documented short path executed
raes-adapters inspect --backend nasim-tiny→native_available: true, backendnasim-tiny,examples: ["nasim-tiny"], qualification identities projected.raes-adapters validate --backend nasim-tiny --mode study(packaged example, all pins) →{"disposition":"validated","pack":"admitted","participant":"nasim-red-bruteforce","run_count":1}.raes-adapters run --backend nasim-tiny --mode study … --output live-episode(one seeded native episode, seed 20260802, trial-length 1000) →{"disposition":"succeeded","inventory":"inventory.json","mode":"study","run_count":1}.Portable artifact inventory (bounded evidence — no native dump)
inventory.jsonseals 10 artifacts:runs/manual-live-1/{evidence-records,derived-measures,diagnostics,participant-provenance,run,summary}.json,study.json,machine-inventory.json,provenance.json,summary.json.study.jsonpresent.Traceback, no native observation/action/state. The only NASim native symbol names present (servicescan,osscan,subnetscan,processscan,flatactionspace) appear solely under the participant selection'sexposure_policy.withheld_refs— the default-deny declaration of what is withheld, carried in provenance exactly as the design requires — not in any observation, action, reward, or host-truth field.Distinct claims preserved. The report distinguishes backend conformance from source-protocol reproduction and research claims; the ADR-069 determinism loss (unbound global-NumPy action RNG) stays disclosed as a non-claim, never erased.
Brad-Edwards commentedon Aug 5, 2026
Pre-PR base synchronization
refs/remotes/origin/devatec6fa3b0b38d4cc175e0abd5b41d6e315db65d18already_current17832bb0867a1987c6614d12edc556a9d38f9d58b6d4197a50dbe6759c8861e2cb4ba241917aca36Brad-Edwards commentedon Aug 5, 2026
Pre-PR base synchronization
refs/remotes/origin/devatec6fa3b0b38d4cc175e0abd5b41d6e315db65d18already_currenta0c064f329e4212c9fdfa8ff4d26d0a423cf19c035bad53d5ad8e39fcd1a65bcbbf19d4d219e4536Brad-Edwards commentedon Aug 5, 2026
Ready for review — issue #36
PR: #76
Plan: #36 (comment)
Outcome
Researchers get a ready-to-run NASim example: a clean
pip install raes-adapters[nasim]can inspect identities, validate the packagednasim-tinyenvironment, and run one episode/seeded batch to a sealed portable evidence bundle — no simulator wiring, and no native details leak out. The pack conforms to the published env-packs format (validated by both pack CLIs), and adding future simulators now reuses one shared researcher/CLI seam instead of copied boilerplate.Add the in-repo NASim nasim-tiny example environment pack and wire it into the installed raes-adapters researcher command via a closed per-backend CLI adapter, with shared researcher/plan helpers extracted from the cyborg sibling.
Files changed
Added:
docs/decisions/nasim-researcher-command-guardrails.mddocs/nasim-researcher-command.mdsrc/raes_adapters/_researcher_support.pysrc/raes_adapters/nasim/researcher.pysrc/raes_adapters/nasim/runtime_plans.pytests/test_nasim_researcher_cli.pysrc/raes_adapters/nasim/examples/nasim-tiny/pack.yamlsrc/raes_adapters/nasim/examples/nasim-tiny/pack.content-manifest.jsonsrc/raes_adapters/nasim/examples/nasim-tiny/pack.compatibility.yamlsrc/raes_adapters/nasim/examples/nasim-tiny/sdl/nasim-tiny.sdl.yamlsrc/raes_adapters/nasim/examples/nasim-tiny/experiment/nasim-tiny.spec.exp.jsonsrc/raes_adapters/nasim/examples/nasim-tiny/experiment/nasim-tiny.task.exp.jsonsrc/raes_adapters/nasim/examples/nasim-tiny/participant/nasim-red-bruteforce.manifest.jsonsrc/raes_adapters/nasim/examples/nasim-tiny/participant/nasim-red-bruteforce.selection.jsonsrc/raes_adapters/nasim/examples/nasim-tiny/participant/nasim-red-bruteforce.configuration.jsonsrc/raes_adapters/nasim/examples/nasim-tiny/docs/provenance-ledger.yamlsrc/raes_adapters/nasim/examples/nasim-tiny/docs/attack-path.mdsrc/raes_adapters/nasim/examples/nasim-tiny/docs/concepts.mdsrc/raes_adapters/nasim/examples/nasim-tiny/docs/golden-readiness-checklist.mdModified:
src/raes_adapters/cli.pysrc/raes_adapters/cyborg/researcher.pysrc/raes_adapters/cyborg/runtime_plans.pysrc/raes_adapters/cyborg/qualification.jsonsrc/raes_adapters/nasim/backend/driver.pysrc/raes_adapters/nasim/backend/__init__.pysrc/raes_adapters/nasim/qualification.jsonnoxfile.pypyproject.tomluv.lockmkdocs.ymldocs/index.mdtests/test_cyborg_qualification.pytests/test_nasim_qualification.pyReviews
Traceability reconciliation
Status
doublewhy commentedon Aug 13, 2026
Kept open after evidence audit. PR #76 (
961ff2fe9989d730663162c0e4313650592b2f25) merged the automated pack, researcher workflow, validators, clean-install rehearsal, and tests. Brad recorded an explicit live-NASim PASS in comment 5188079000, but that run predates the final refactor commits17832bb/a0c064fand the merge. The PR final test plan also states that the manual live-NASim short path remained to be run. The missing proof is therefore for the exact final merged/released artifact, not proof that the manual path was never exercised. Because the final-artifact acceptance criterion is unverified, #36 remains open. The stalein-progresslabel has been removed; no experiment was started during this freeze-safe pass.doublewhy commentedon Oct 9, 2026
Live NASim short-path run on the current
devtip (informational evidence, no code change).Commit:
0272949fa964920a7e06fb56f6d7051054756b5b(origin/dev, unchanged since 2026-08-19). It descends from the #76 merge961ff2f. It also contains #90 (8db4fbd, 2026-08-12), which added the evidence-admission gate seen below. Run on 2026-10-09 between 06:01 and 06:05 UTC.Environment and versions
ubuntu:24.04container (24.04.5) under Docker Desktop on an arm64 Mac, so x86_64 is emulated. glibc 2.39.python3-tk(Tk 8.6).PYTHONPATHunset,PYTHONSAFEPATH=1, uv 0.12.1.raes-adaptersis not on PyPI (the JSON API returns 404). The wheel was therefore built from agit archiveof the commit, following thedistributionsnox session:raes_adapters-0.0.0-py3-none-any.whl, sha256068871d1866b944cf65a695f5c49c1a6c8705b334aea565a54b19819d341ebc6. A host build produced the same digest.Commands
The CLI lines come from
docs/nasim-researcher-command.md.<doc args>stands for that page's unchanged--pack nasim-tiny … --trial-length 1000 --seed 20260802 --run-id nasim-tiny-runblock.Two negative probes were also run:
validateline with the last hex digit of--pack-digestchanged;--output.Observed (trimmed)
inspect"native_available": true; qualification source7c732bc4…/0.12.0; examples["nasim-tiny"]run --mode conformance{"disposition": "succeeded", "evidence_basis": "installed-source-probe", "inventory": "inventory.json", "mode": "conformance", "run_count": 1}validate --mode studyresearcher.validation.evidence-unverifiable: task evidence requirements cannot be verified by the backend manifestrun --mode smokenasim-smoke-evidencenot createdrun --mode studynasim-study-evidencenot createdvalidate, wrong pack digestresearcher.validation.controls-invalid: run controls were not admittedresearcher.output.unavailable: output root is unavailableverify-bundlebundle.integrity.verified: 1 inventory, 2 entries, 3 filesraes-pack-validateENVIRONMENT-PACK CONTENT CI: PASS, including[ok] anti-extension guardraes-pack-release checkPACK RELEASE GATE: PASSThe conformance root contains exactly
index.json,inventory.jsonandruns/nasim-pr-seed-20260802/conformance/backend-conformance.json.index.jsonrecordsnative_conformance: false, 7 declared weaknesses and the non-claim "A probe over the installed source is not a native attacker episode."Tracebackorsite-packagesstring.On the host itself (macOS arm64, CPython 3.12.13), the same wheel reports
"native_available": false.verify_runtime_artifactsrejects numpy, because the qualified artifact innasim/qualification.jsonis themanylinux_2_17_x86_64wheel tree.Against the acceptance criteria, at
0272949raes-pack-validateandraes-pack-release checkpass.validatereaches the evidence gate._admit_native_runraises that gate only after the pack digest, content digest, SDL/spec/task/participant loading and binding checks have been admitted.validate,run --mode smokeandrun --mode studyexit 3 before runtime planning. No native episode ran, and no episode inventory was produced.docs/nasim-researcher-command.mdL46–52). Thedistributionssession asserts it (noxfile.pyL757).