Skip to content

feat(nasim): add the example environment pack and researcher workflow #36

Open

Description

@Brad-Edwards

Objective

Provide one validated NASim environment pack and a no-code researcher workflow for the selected case.

Scope

  • Add environments/nasim-tiny/ from the published raes-env-packs template.
  • Include pack identity, the canonical SDL, provenance/licensing/safety review, compatibility claims, associated-artifact identity where applicable, source-ledger references, and clear participant/operator/evaluator boundaries.
  • Integrate the adapter into the repository's installed researcher command so a clean environment can inspect identities, validate the pack, run one episode or seeded batch, and retain a portable artifact inventory.
  • Provide a reference build, automated rehearsal, and matching researcher walkthrough.
  • Keep runtime implementation selection and simulator-native configuration outside pack semantics.
  • Mark draft, built, or golden only according to evidence actually produced.
  • Keep the pack out of the Python wheel; publish it as versioned repository/release content.

Acceptance criteria

  • raes-pack-validate and raes-pack-release check pass.
  • A clean installed environment can acquire the immutable pack, run the documented short path, and validate its outputs.
  • Build, tests, and walkthrough use the same scenario, controls, agent, objective, and evaluator.
  • Participant, operator, evaluator/restricted, native-source, and public artifact boundaries pass leak checks.
  • Provenance covers all upstream sources, exclusions, generated content, and redistribution decisions.
  • The command rejects mismatched pins/digests, unsafe output reuse, unsupported controls, and invalid participant bindings before execution.
  • The pack defines no RAES semantics and RAESystem/env-packs receives no scenario content.
  • Manual integration test — honest NASim execution: in a clean environment, run the researcher command's documented short path (inspect identities → validate pack → run one episode/seeded batch) against a live installed NASim (qualified runtime per build(nasim): qualify and pin the public source and experiment profile #32) and confirm the portable artifact inventory is produced — a hands-on end-to-end check beyond the automated rehearsal.

Shared command reference

Activity

  1. Brad-Edwards commented on Jul 29, 2026

    @Brad-Edwards
    CollaboratorAuthor

    RAESystem/hub#10 has now established the federated catalog boundary for reusable example packs. Pack source belongs in an appropriate catalog repository (with RAESystem/reference-packs as the first-party curated catalog) under environments/<pack-id>/, using a history- and provenance-preserving one-way transfer. This adapter repository retains simulator realization and backend-native evidence, then consumes the released pack identity; it should not retain a competing editable pack copy. The upstream format, validation, release, provenance, trust, and compatibility contracts remain owned by RAESystem/env-packs.

  2. added
    in-progressAn agent is actively working this issue via /implement
    on Aug 4, 2026
  3. Brad-Edwards commented on Aug 4, 2026

    @Brad-Edwards
    CollaboratorAuthor

    🛠️ Picked up by /implement — driver claude, branch 36-nasim-env-pack, 2026-08-04T20:46:48Z. Scope confirmed with maintainer: NASim example pack lives in-repo beside the nasim adapter (mirroring cyborg/examples/cage2-research), conforming to the published OpenRAE/env-packs (raes-env-packs) format — no reinvented RAE SDL/mechanics or competing pack format.

  4. Brad-Edwards commented on Aug 4, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Implementation plan — NASim example environment pack + researcher workflow

    Requirement-free run (no in-scope requirement UID; nox -s verify -- --skip-requirement). Mirrors the shipped CyBORG sibling (src/raes_adapters/cyborg/examples/cage2-research/ + cyborg/researcher.py), conforming to the published raes-env-packs (OpenRAE/env-packs) format and RAE SDL — no reinvented format/semantics/mechanics.

    Scope decisions (maintainer-confirmed)

    • Pack lives in-repo beside the nasim adapter at src/raes_adapters/nasim/examples/nasim-tiny/ (in-wheel, like CyBORG). The issue body's environments/… + "out of the wheel" wording is superseded by this decision; flagged in the PR.
    • Format/tooling owned by raes-env-packs; the pack consumes it and adds nothing to raes_adapters.base and no CAGE/RAES semantics.

    Key finding from assessment

    The shipped CyBORG pack passes only the library validate_pack() API the CLI calls — it fails raes-pack-validate (missing golden checklist) and is skipped by raes-pack-release check (no compatibility manifest). Satisfying #36's acceptance criteria is therefore net-new work beyond mirroring CyBORG: the NASim pack adds docs/golden-readiness-checklist.md + pack.compatibility.yaml (artifact_boundaries), and the rehearsal actually runs both pack CLIs.

    Pack — src/raes_adapters/nasim/examples/nasim-tiny/

    • pack.yaml — manifest (name nasim-tiny; provenance/associated-artifact/compatibility pointers; status set to evidence actually produced).
    • pack.content-manifest.json — derived via raes_env_packs.derive_pack_content_manifest; its set_digest is the pinned --pack-digest.
    • pack.compatibility.yaml — net-new (no CyBORG precedent); artifact_boundaries so raes-pack-release check treats the pack as releasable.
    • sdl/nasim-tiny.sdl.yaml — copy of the authored nasim/scenario/nasim-tiny.sdl.yaml (RAES-valid, pinned digest).
    • experiment/nasim-tiny.spec.exp.json / nasim-tiny.task.exp.json — JSON ports of the existing YAML spec/task, pack:/… artifact refs byte-bound to the participant files.
    • participant/nasim-red-bruteforce.{manifest,selection,configuration}.json — the red bruteforce attacker (role red, backend nasim-tiny); withheld_refs names the NASim native-state markers.
    • docs/provenance-ledger.yaml (environment-pack-provenance/v3; NASim MIT source, attribution required + adapter-example source; review gates licensing/attribution/sensitive-data/offensive-tooling), docs/attack-path.md, docs/concepts.md, docs/golden-readiness-checklist.md.

    Researcher command

    • New src/raes_adapters/nasim/researcher.py — NASim RunControls/execute_episode/archival_run/archival_collection over create_nasim_target() + the exported Nasim runtime components (red-attacker semantics; metrics steps_to_termination, cumulative_attacker_reward, goal_reached, terminal_cause; NASim logical clock).
    • New src/raes_adapters/nasim/runtime_plans.py — NASim orchestration/evaluation/time plan builders (analog of cyborg/runtime_plans.py).
    • src/raes_adapters/nasim/__init__.py (or backend/driver.py) — expose public verify_selected_nasim_source() wrapping the existing _source_admission calls, for the CLI native gate.
    • src/raes_adapters/cli.py — backend-parameterized refactor. Extend --backend to nasim-tiny; resolve pack-root/example-members, inspection payload, native-import + source-verify gate, and RunControls/execute-episode dispatch per backend, keeping NASim's red-attacker semantics inside raes_adapters.nasim (per the CyBORG guardrails' extension seam). Must not regress the CyBORG path.

    Rehearsal + walkthrough + guardrails

    • noxfile.py _distributions — add NASim clean-install inspect/validate/run blocks mirroring CyBORG, plus raes-pack-validate --pack … and raes-pack-release check --pack … (env-packs console scripts, via the cyborg/env-packs dist available to the rehearsal env).
    • docs/nasim-researcher-command.md + docs/decisions/nasim-researcher-command-guardrails.md (+ mkdocs.yml / docs/index.md nav — strict docs build gate).
    • tests/test_nasim_researcher_cli.py — subprocess acceptance suite mirroring tests/test_researcher_cli.py: every exit class, output reserve/reuse, symlink/traversal escape, hostile native stdout/stderr suppression, duplicate-JSON-key rejection, tamper/stale-checksum rejection, participant-binding mismatch, full study seal + exact inventory. Native-leakage invariants over NASim outputs (no host paths, no secret sentinel, no Traceback, relative inventory paths, sha256 len 64).

    Boundaries / determinism disclosure

    Participant/operator/evaluator/native-source/public-artifact boundaries enforced via the participant selection.json exposure_policy (disclosed vs withheld) + the scenario-ledger native-leakage scan. The ADR-069 determinism loss (unbound global-NumPy action RNG; qualification.json known_defects) stays disclosed in provenance and non-claims — never erased.

    Verification

    uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify -- --skip-requirement (hygiene, policy, lint, tool-tests, typecheck, base + nasim tests incl. the new researcher CLI suite, distributions clean-wheel proof now running the NASim pack + pack CLIs, strict docs). Single pyproject.toml/uv.lock; no CHANGELOG.md edit (Release Please owns it); PR title feat(nasim): ….

    Manual integration test (final acceptance criterion)

    In a clean environment, run the researcher command's documented short path (inspect identities → validate pack → run one episode/seeded batch) against live installed NASim (qualified runtime per #32) and confirm the portable artifact inventory is produced. If NASim can't install/run, report a failed/blocked manual-readiness result — never a silent fall back.

  5. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Station observation — codex review cycle 1 — Opened

    Observed state: the station ran but rendered no verdict, so this gate is unobserved.
    Failure class: incomplete_reviewer_coverage
    Attempt: 1
    Impact: the review cap was not consumed and no findings record exists for this cycle.
    Current obligation: observe this station before the run completes.

    This is a missing observation, not a defect. It is resolved by observing the station, not by
    dispositioning a finding.

  6. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_codex_review — cycle 1 of 1 (pre-push) on issue #36 (branch 36-nasim-env-pack)
    Diff mode: inline — the complete diff was supplied in one prompt

    Core review

    Verdict: ship-with-fixes

    The change is mostly shaped correctly: the installed shell grows a closed per-backend adapter seam, NASim semantics stay inside raes_adapters.nasim, and the implementation composes RAES contracts, raes_env_packs, runtime planning, participant configuration, diagnostics, and conformance report helpers instead of inventing local schemas. The obvious extension path is still published pack/participant/backend artifacts plus another adapter entry, not a cross-backend semantic layer. The one production-readiness issue is in the NASim runtime lifecycle: startup/provision/orchestration work happens before the cleanup guard, which cuts across the repo's explicit cleanup invariant.

    Blocking findings (1):

    1. [one-off] Runtime startup failures bypass cleanup — src/raes_adapters/nasim/researcher.py:625
      execute_episode calls _start_runtime before entering the try/finally that runs RuntimeManager.destroy(). _start_runtime plans, applies provisioning, and starts orchestration; if provisioning or orchestration partially allocates the target and then fails, the exception returns to the CLI without the verified cleanup path. The researcher command contract says cleanup is always attempted and a failed cleanup cannot be downgraded, so startup needs to be inside a cleanup-protected region or _start_runtime needs to destroy the target on every partial-start failure.

    Security review

    Verdict: ship

    This change is shaped as a closed per-backend adapter dispatch for the installed local command: NASim-specific participant admission, pack validation, source verification, native execution, and evidence archival stay behind the NASim module while the shared CLI owns mode dispatch and output reservation. The security-relevant boundaries are the closed argparse surface, pack child confinement through the pack validator and _pack_child, installed-source verification before native import, native stdout/stderr suppression, and bounded error envelopes. I do not see a concrete exploitable security regression introduced by this diff.

    No blocking findings.

  7. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Station observation — codex review cycle 1 — Re-observed

    Disposition: reobserved
    Corrective action: a bounded automatic re-attempt observed the station and it rendered a verdict.
    Evidence: #36 (comment)

    Verification

    • The linked findings record is this station's validated verdict for this logical cycle.
    • Re-observation closes only the missing-observation obligation. Any findings in that record
      remain subject to the existing fix / wontfix / not-applicable rules.
  8. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_codex_review pre-push cycle 1 of 1 complete for issue #36 on branch '36-nasim-env-pack'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

  9. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Review decision record — codex cycle 1 (issue #36)

    Reviewer: codex
    Cycle: 1

    Architectural read:

    Core reviewer: The change is mostly shaped correctly: the installed shell grows a closed per-backend adapter seam, NASim semantics stay inside raes_adapters.nasim, and the implementation composes RAES contracts, raes_env_packs, runtime planning, participant configuration, diagnostics, and conformance report helpers instead of inventing local schemas. The obvious extension path is still published pack/participant/backend artifacts plus another adapter entry, not a cross-backend semantic layer. The one production-readiness issue is in the NASim runtime lifecycle: startup/provision/orchestration work happens before the cleanup guard, which cuts across the repo's explicit cleanup invariant.

    Security reviewer: This change is shaped as a closed per-backend adapter dispatch for the installed local command: NASim-specific participant admission, pack validation, source verification, native execution, and evidence archival stay behind the NASim module while the shared CLI owns mode dispatch and output reservation. The security-relevant boundaries are the closed argparse surface, pack child confinement through the pack validator and _pack_child, installed-source verification before native import, native stdout/stderr suppression, and bounded error envelopes. I do not see a concrete exploitable security regression introduced by this diff.

    Blocking findings: 1

    Finding 1 — one-off

    • ID: F1
    • Title: [core] Runtime startup failures bypass cleanup
    • Location: src/raes_adapters/nasim/researcher.py:625
    • Decision: fix
    • Rationale: execute_episode calls _start_runtime before entering the try/finally that runs RuntimeManager.destroy(). _start_runtime plans, applies provisioning, and starts orchestration; if provisioning or orchestration partially allocates t…
  10. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_test_quality_review cycle 1 of 1 — issue #36

    Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
    Branch: 36-nasim-env-pack
    Cycle: 1 / 1
    Findings: 0 (clean run)

  11. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_test_quality_review cycle 1 of 1 complete for issue #36 on branch '36-nasim-env-pack'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next gc_test_quality_review invocation to count cycles.

  12. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Review decision record — test-quality cycle 1 (issue #36)

    Reviewer: test-quality
    Cycle: 1

    Architectural read:

    This PR adds the NASim researcher-command acceptance suite (test_nasim_researcher_cli.py) plus updated qualification golden-record tests (test_cyborg_qualification.py, test_nasim_qualification.py) and the nasim-tiny experiment authoring fixture. The CLI test suite deliberately mirrors the existing, previously-shipped CybORG suite's shape: a FakeNasimDriver injected at the NasimDriverProtocol seam so the standard test job never imports the real nasim/Tk dependency, while every admission/tamper/escape path is driven end-to-end through the real cli.main() and the real production validators (raes_env_packs.validate_pack/verify_pack_content_digest, strict duplicate-key JSON loading, _pack_child traversal confinement, spec-artifact byte/digest binding, participant manifest/selection/configuration digest joins). This is the correct seam: native simulator construction/import is already covered by the pre-existing, out-of-scope test_nasim_driver.py, so mocking only resolve_selected_distribution / _verify_pre_import_source here is a boundary choice, not a shortcut around the security gate — the gate itself (duplicate-key rejection, traversal rejection, digest/checksum mismatch rejection, symlink-escape rejection) is exercised for real and asserted by exit code plus absence of any created output directory, which is the right assertion shape for a fail-closed contract. The tampered-bytes tests correctly re-derive the pack-level content digest with the real derive_pack_content_manifest so only the intended layer (spec-level artifact-byte binding) is what fails, isolating exactly what's under test. No test-visible production special-casing on fixture names/paths was found, and the qualification tests are consistent golden-record checks matching the pre-existing sibling pattern. I'd ship this.

    Blocking findings: 0 (clean run)

  13. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Pre-PR base synchronization

    • Source: refs/remotes/origin/dev at ec6fa3b0b38d4cc175e0abd5b41d6e315db65d18
    • Outcome: already_current
    • Published feature head: 034b3b7b6e0775e8680e7dd178adfd51c5408d53
    • Verified tree: cf59ea04d98bb6ce56db63d523262fd5954d6d07
  14. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Manual integration test — honest live-NASim execution (issue #36 final acceptance criterion)

    Result: PASS. In a clean installed environment, the researcher command's documented short path was run against a real installed NASim — not a stub/mock, no fall back to an injected driver.

    Environment

    • Clean venv: raes-adapters[nasim] installed from the built wheel, Python 3.12.3 (the qualified runtime's Python), live nasim==0.12.0. _tkinter present; import nasim succeeds.

    Documented short path executed

    1. raes-adapters inspect --backend nasim-tiny → native_available: true, backend nasim-tiny, examples: ["nasim-tiny"], qualification identities projected.
    2. raes-adapters validate --backend nasim-tiny --mode study (packaged example, all pins) → {"disposition":"validated","pack":"admitted","participant":"nasim-red-bruteforce","run_count":1}.
    3. raes-adapters run --backend nasim-tiny --mode study … --output live-episode (one seeded native episode, seed 20260802, trial-length 1000) → {"disposition":"succeeded","inventory":"inventory.json","mode":"study","run_count":1}.

    Portable artifact inventory (bounded evidence — no native dump)

    • inventory.json seals 10 artifacts: runs/manual-live-1/{evidence-records,derived-measures,diagnostics,participant-provenance,run,summary}.json, study.json, machine-inventory.json, provenance.json, summary.json.
    • All inventory paths relative; every checksum a 64-char sha256; study.json present.
    • Leakage oracle over every produced JSON: no host/tmp path, no Traceback, no native observation/action/state. The only NASim native symbol names present (servicescan, osscan, subnetscan, processscan, flatactionspace) appear solely under the participant selection's exposure_policy.withheld_refs — the default-deny declaration of what is withheld, carried in provenance exactly as the design requires — not in any observation, action, reward, or host-truth field.

    Distinct claims preserved. The report distinguishes backend conformance from source-protocol reproduction and research claims; the ADR-069 determinism loss (unbound global-NumPy action RNG) stays disclosed as a non-claim, never erased.

  15. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Pre-PR base synchronization

    • Source: refs/remotes/origin/dev at ec6fa3b0b38d4cc175e0abd5b41d6e315db65d18
    • Outcome: already_current
    • Published feature head: 17832bb0867a1987c6614d12edc556a9d38f9d58
    • Verified tree: b6d4197a50dbe6759c8861e2cb4ba241917aca36
  16. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Pre-PR base synchronization

    • Source: refs/remotes/origin/dev at ec6fa3b0b38d4cc175e0abd5b41d6e315db65d18
    • Outcome: already_current
    • Published feature head: a0c064f329e4212c9fdfa8ff4d26d0a423cf19c0
    • Verified tree: 35bad53d5ad8e39fcd1a65bcbbf19d4d219e4536
  17. Brad-Edwards commented on Aug 5, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Ready for review — issue #36

    PR: #76
    Plan: #36 (comment)

    Outcome

    Researchers get a ready-to-run NASim example: a clean pip install raes-adapters[nasim] can inspect identities, validate the packaged nasim-tiny environment, and run one episode/seeded batch to a sealed portable evidence bundle — no simulator wiring, and no native details leak out. The pack conforms to the published env-packs format (validated by both pack CLIs), and adding future simulators now reuses one shared researcher/CLI seam instead of copied boilerplate.

    Add the in-repo NASim nasim-tiny example environment pack and wire it into the installed raes-adapters researcher command via a closed per-backend CLI adapter, with shared researcher/plan helpers extracted from the cyborg sibling.

    Files changed

    Added:

    • docs/decisions/nasim-researcher-command-guardrails.md
    • docs/nasim-researcher-command.md
    • src/raes_adapters/_researcher_support.py
    • src/raes_adapters/nasim/researcher.py
    • src/raes_adapters/nasim/runtime_plans.py
    • tests/test_nasim_researcher_cli.py
    • src/raes_adapters/nasim/examples/nasim-tiny/pack.yaml
    • src/raes_adapters/nasim/examples/nasim-tiny/pack.content-manifest.json
    • src/raes_adapters/nasim/examples/nasim-tiny/pack.compatibility.yaml
    • src/raes_adapters/nasim/examples/nasim-tiny/sdl/nasim-tiny.sdl.yaml
    • src/raes_adapters/nasim/examples/nasim-tiny/experiment/nasim-tiny.spec.exp.json
    • src/raes_adapters/nasim/examples/nasim-tiny/experiment/nasim-tiny.task.exp.json
    • src/raes_adapters/nasim/examples/nasim-tiny/participant/nasim-red-bruteforce.manifest.json
    • src/raes_adapters/nasim/examples/nasim-tiny/participant/nasim-red-bruteforce.selection.json
    • src/raes_adapters/nasim/examples/nasim-tiny/participant/nasim-red-bruteforce.configuration.json
    • src/raes_adapters/nasim/examples/nasim-tiny/docs/provenance-ledger.yaml
    • src/raes_adapters/nasim/examples/nasim-tiny/docs/attack-path.md
    • src/raes_adapters/nasim/examples/nasim-tiny/docs/concepts.md
    • src/raes_adapters/nasim/examples/nasim-tiny/docs/golden-readiness-checklist.md

    Modified:

    • src/raes_adapters/cli.py
    • src/raes_adapters/cyborg/researcher.py
    • src/raes_adapters/cyborg/runtime_plans.py
    • src/raes_adapters/cyborg/qualification.json
    • src/raes_adapters/nasim/backend/driver.py
    • src/raes_adapters/nasim/backend/__init__.py
    • src/raes_adapters/nasim/qualification.json
    • noxfile.py
    • pyproject.toml
    • uv.lock
    • mkdocs.yml
    • docs/index.md
    • tests/test_cyborg_qualification.py
    • tests/test_nasim_qualification.py

    Reviews

    • codex: Pre-push 1/1: ship-with-fixes, security ship. F1 (startup ran before the cleanup guard, so a partial start bypassed cleanup) fixed: startup moved inside try/finally + a regression test. Coverage 30/30 files.
    • test-quality: Pre-push 1/1: clean, 0 findings.

    Traceability reconciliation

    • Pending — requirement status transition and IMPLEMENTS/TESTS reconciliation run in Phase E once the PR merges.

    Status

    • CI: ✅ green
    • SonarCloud: ✅ passed
    • PR ready for user review and merge. Ground Control reconciliation (requirement status + traceability) runs on merge (Phase E).
  18. removed
    in-progressAn agent is actively working this issue via /implement
    on Aug 13, 2026
  19. doublewhy commented on Aug 13, 2026

    @doublewhy

    Kept open after evidence audit. PR #76 (961ff2fe9989d730663162c0e4313650592b2f25) merged the automated pack, researcher workflow, validators, clean-install rehearsal, and tests. Brad recorded an explicit live-NASim PASS in comment 5188079000, but that run predates the final refactor commits 17832bb / a0c064f and the merge. The PR final test plan also states that the manual live-NASim short path remained to be run. The missing proof is therefore for the exact final merged/released artifact, not proof that the manual path was never exercised. Because the final-artifact acceptance criterion is unverified, #36 remains open. The stale in-progress label has been removed; no experiment was started during this freeze-safe pass.

  20. doublewhy commented on Oct 9, 2026

    @doublewhy

    Live NASim short-path run on the current dev tip (informational evidence, no code change).

    Commit: 0272949fa964920a7e06fb56f6d7051054756b5b (origin/dev, unchanged since 2026-08-19). It descends from the #76 merge 961ff2f. It also contains #90 (8db4fbd, 2026-08-12), which added the evidence-admission gate seen below. Run on 2026-10-09 between 06:01 and 06:05 UTC.

    Environment and versions

    • A linux/amd64 ubuntu:24.04 container (24.04.5) under Docker Desktop on an arm64 Mac, so x86_64 is emulated. glibc 2.39.
    • System Python 3.12.3 with python3-tk (Tk 8.6). PYTHONPATH unset, PYTHONSAFEPATH=1, uv 0.12.1.
    • Installed: raes-adapters 0.0.0, raes 3.3.0, raes-env-packs 3.6.2, nasim 0.12.0, gymnasium 0.26.3, numpy 1.26.4.
    • raes-adapters is not on PyPI (the JSON API returns 404). The wheel was therefore built from a git archive of the commit, following the distributions nox session: raes_adapters-0.0.0-py3-none-any.whl, sha256 068871d1866b944cf65a695f5c49c1a6c8705b334aea565a54b19819d341ebc6. A host build produced the same digest.

    Commands

    The CLI lines come from docs/nasim-researcher-command.md. <doc args> stands for that page's unchanged --pack nasim-tiny … --trial-length 1000 --seed 20260802 --run-id nasim-tiny-run block.

    git archive 0272949 | tar -x -C /work-src && cd /work-src
    uv build --out-dir /work/dist
    uv venv --python /usr/bin/python3.12 /work/venv
    uv pip install --python /work/venv --find-links /work/dist '/work/dist/raes_adapters-0.0.0-py3-none-any.whl[nasim]'
    cd /work/nasim-researcher-work   # new, empty
    raes-adapters inspect --backend nasim-tiny
    raes-adapters run --backend nasim-tiny --mode conformance --suite pr --output nasim-conformance-evidence
    raes-adapters validate --backend nasim-tiny --mode study <doc args>
    raes-adapters run --backend nasim-tiny --mode smoke <doc args> --output nasim-smoke-evidence
    raes-adapters run --backend nasim-tiny --mode study <doc args> --output nasim-study-evidence
    raes-adapters verify-bundle --bundle nasim-conformance-evidence --format json
    raes-pack-validate --pack /work-src/src/raes_adapters/nasim/examples/nasim-tiny
    raes-pack-release check --pack /work-src/src/raes_adapters/nasim/examples/nasim-tiny

    Two negative probes were also run:

    • the validate line with the last hex digit of --pack-digest changed;
    • the conformance line repeated with the same --output.

    Observed (trimmed)

    step exit output
    inspect 0 "native_available": true; qualification source 7c732bc4… / 0.12.0; examples ["nasim-tiny"]
    run --mode conformance 0 {"disposition": "succeeded", "evidence_basis": "installed-source-probe", "inventory": "inventory.json", "mode": "conformance", "run_count": 1}
    validate --mode study 3 researcher.validation.evidence-unverifiable: task evidence requirements cannot be verified by the backend manifest
    run --mode smoke 3 same diagnostic; nasim-smoke-evidence not created
    run --mode study 3 same diagnostic; nasim-study-evidence not created
    validate, wrong pack digest 3 researcher.validation.controls-invalid: run controls were not admitted
    conformance, reused output 4 researcher.output.unavailable: output root is unavailable
    verify-bundle 0 bundle.integrity.verified: 1 inventory, 2 entries, 3 files
    raes-pack-validate 0 ENVIRONMENT-PACK CONTENT CI: PASS, including [ok] anti-extension guard
    raes-pack-release check 0 PACK RELEASE GATE: PASS

    The conformance root contains exactly index.json, inventory.json and runs/nasim-pr-seed-20260802/conformance/backend-conformance.json.

    • The inventory paths are relative, and each sha256 is 64 hex characters.
    • index.json records native_conformance: false, 7 declared weaknesses and the non-claim "A probe over the installed source is not a native attacker episode."
    • A grep of every produced file found no host path, Traceback or site-packages string.

    On the host itself (macOS arm64, CPython 3.12.13), the same wheel reports "native_available": false. verify_runtime_artifacts rejects numpy, because the qualified artifact in nasim/qualification.json is the manylinux_2_17_x86_64 wheel tree.

    Against the acceptance criteria, at 0272949

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions