Skip to content

feat(primaite): implement the conformant simulator backend target #41

Closed

Description

@Brad-Edwards

Objective

Implement raes_adapters.primaite as a conformant simulator backend over the selected source profile.

Scope

  • Add the primaite optional extra and module without changing base-only installation.
  • Implement Provisioner target construction and an evidence-backed backend-manifest-v2.
  • Implement Orchestrator logical steps, reset/termination, admitted-action translation, receipts, and mapped state transitions.
  • Implement ParticipantRuntime lifecycle, participant-relative observations, action admission, histories, sequence/linkage, and exposure controls.
  • Implement Evaluator projection of simulator reward/result facts into distinct RAES objectives, evidence, derived measures, and limitations.
  • Reuse raes_adapters.base and published RAES contracts directly.
  • Keep native simulator objects, IDs, tuples, arrays, logs, paths, hidden state, exceptions, argv/environment data, credentials, and tracebacks inside the driver.
  • Provide deterministic cleanup and failure-path tests.

Acceptance criteria

  • The selected RAES scenario constructs and runs through all applicable backend surfaces.
  • Manifest capabilities have executable evidence and unsupported facts are rejected or disclosed.
  • Every admitted action joins to one driver operation, typed terminal result, and participant/episode identity.
  • Observation and evaluator-only boundaries pass positive and negative disclosure tests.
  • Reset, seed/control reporting, termination, and cleanup match the selected source protocol.
  • Portable artifacts contain only RAES contract data and stable source references.
  • The canonical verification graph passes with this extra and with unrelated extras isolated.

Dependencies

Activity

  1. added
    in-progressAn agent is actively working this issue via /implement
    on Aug 2, 2026
  2. Brad-Edwards commented on Aug 2, 2026

    @Brad-Edwards
    CollaboratorAuthor

    🛠️ Picked up by /implement - driver claude, branch 41-primaite-backend, 2026-08-02T23:56:52.673Z.

  3. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc workflow phase recorded: plan (issue #41). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

    Implementation plan — raes_adapters.primaite conformant backend (#41)

    This run is requirement-free (issue #41 has no #### Requirements section; the GC
    project holds no local requirements). The governance gate runs requirement-free
    (--skip-requirement / no ACES_REQUIREMENT_UID). The binding architecture
    preflight guardrails live at docs/decisions/primaite-backend-guardrails.md;
    every design point below is subordinate to that record.

    Process note: the Step 2.5 preflight produced the complete guardrails doc but its
    codex process was killed at the 20-minute hard timeout on both runs before the
    preflight marker could be recorded; the plan gate was overridden (with an
    audited reason and explicit user authorization) so work could proceed on the
    already-complete guardrails.

    Shape and layout

    Add the runtime target under src/raes_adapters/primaite/backend/, mirroring
    cyberbattlesim/backend/ (the selection evidence — qualification.json,
    scenario_ledger.py, SDL, experiment task/spec, mapping — stays flat at
    primaite/, already shipped by #39/#40). The already-declared dependency-light
    primaite = [] extra is unchanged; the backend is pure in-process plumbing over
    published RAES contracts, so hatchling packages the new subpackage automatically
    and base-only / unrelated-extra installs stay byte-identical (ADR-003).

    New modules (src/raes_adapters/primaite/backend/):
    __init__.py, _diagnostics.py (dotted→JSON-Pointer, /primaite fallback),
    driver.py (private native lifecycle + source-identity verification + the
    representability gate), provisioner.py, orchestrator.py,
    participant_runtime.py, evaluator.py, manifest.py (+ realization envelope),
    cleanup.py, conformance.py, target.py.

    Semantics (from the guardrails)

    • RL seat is BLUE. Participant runtime supports role blue; RED/GREEN are
      source-internal (scripted / probabilistic), never separate admissions or steps.
    • Representability gate (hard). The SDL blue action contracts are broad
      families with no operation argument; the only qualified native action is
      do-nothing 0, which has no portable contract. So the live driver/participant
      runtime rejects every broad blue contract before any env.step with a
      bounded unsupported/unrepresentable disposition — it never silently picks a
      native op, never exposes a native id/option dict, and never maps a contract to
      action 0 to "make the smoke run". Admission remains fully modelled (request →
      validated selection → typed rejected terminal result → participant/episode
      identity join), which the acceptance criterion on admission requires.
    • Aggregate turn. One admitted BLUE action ⇒ one driver operation ⇒ ≤1
      env.step. Orchestrator never calls env.step; Evaluator never advances the
      source; observation projection makes no source call.
    • Termination. terminated is hard-coded false; only fixed-horizon
      truncation at step 128. No invented success/failure terminal, no N+1 step.
    • Stochastic controls. Descriptive only (broken torch seed seam, uncontrolled
      GREEN RNG). Report applied/broken/absent/unbound streams as distinct
      dispositions; no RandomStreamControlBindingModel, no global reseed.
    • Python honesty. Live source is qualified CPython 3.11-only; the repo is
      3.12+. The live source-installed driver fails the source-backed target honestly
      on an unqualified runtime; the fake driver proves portable mechanics only and
      cannot upgrade the live-runtime claim.
    • Process/workspace isolation. Lazy import under a driver-owned temporary
      HOME/workspace established before native import; restore process state on
      every path; one active native target per process; base never imports
      primaite/gymnasium/numpy. Cleanup removes and verifies the workspace.
    • Quarantine. If a native mutation succeeds but projection/RAES validation
      fails, quarantine the native episode, return the unchanged portable baseline
      with a bounded failure, and require aggregate reset/cleanup before another
      action.

    Component design

    • manifest.py — create_primaite_manifest() builds a validated
      BackendManifest via RAES capability models + backend_manifest_payload()
      (never hand-authored). It declares a BackendRealizationEnvelopeModel
      (cyborg pattern) binding scenario/experiment/source selection/mapping losses +
      stochastic disposition, digested via realization_envelope_digest /
      validate_backend_realization_envelope. Capabilities disclose the abstracted
      participant interface as a disclosed-weak feature referencing the guardrails
      doc; blue role declared but constrained "no qualified single-op action
      argument → contracts rejected until an argument seam is qualified".
    • driver.py — private PrimaiteDriver (source-installed) +
      PrimaiteDriverProtocol; sanitized dataclasses (DriverResetReport,
      DriverStep, DriverEvaluation, DriverCleanupReport). Verifies the selected
      PrimAITE distribution version, complete import-root file count/digest, critical
      source digests, and module origin before native import (guardrails attest only
      the PrimAITE root; gymnasium/numpy/setuptools are observed evidence, not
      separately attested). Owns the temp HOME/workspace, the aggregate env.step,
      and the representability rejection. Native objects never leave the class.
    • provisioner.py — accepts only a plan that joins completely to the
      selected scenario/experiment/source/realization envelope; portable snapshot
      carries only planned RAES state + stable qualification/ledger refs; no native
      handle stored.
    • orchestrator.py — owns workflow lifecycle only; empty compiled-SDL
      orchestration is valid and is not backfilled with invented operations.
    • participant_runtime.py — subclasses BaseParticipantRuntime; _model_action
      is the backend seam that rejects unrepresentable blue contracts; observation
      envelopes built from an allowlist of evidence-backed semantic facts (never from
      the Box(1652) array / info / mask).
    • evaluator.py — cbs-style scalar projection: reads the one committed driver
      fact without advancing the source; emits distinct proposition-truth
      (unknown, lossy-evidence — there is no source terminal to assert),
      evidence-record, derived-measure, terminal-cause, and limitation models.
    • cleanup.py — execute_primaite_cleanup() over execute_cleanup();
      close + workspace removal + verification as deterministic idempotent private ops.
    • conformance.py — run_primaite_conformance() via run_conformance_probe;
      declared-weakness + source-protocol-diagnostic composition over
      scenario_ledger.validate_all and the qualification limitations/losses.
    • target.py — create_primaite_target() / create_primaite_components()
      via base.build_runtime_target; validates the manifest realization envelope is
      the selected one; injectable driver for tests.

    DRY / SonarCloud duplication (root-cause, not suppression)

    Copying cbs/cyborg boilerplate verbatim would trip `new_duplicated_lines_density

    3%. Extract the genuinely mechanical, literal-free machinery into raes_adapters.base` as the single canonical copy (the guardrails explicitly bless
    extracting "complete-root enumeration, tree digesting, module-origin
    verification") and route all backends through it:

    1. base/source_identity.py — complete import-root enumeration, installed-tree
      digest, is_sha256, module-origin verification, direct-install archive checks.
      Migrate the cyberbattlesim driver to it (delete its copies); primaite uses it.
    2. base/manifest_evidence.py — the JSON-Pointer capability-pointer walk
      (affirmative_capability_pointers, _escape_pointer_token, evidence
      intersection, backend_conformance_payload). Migrate cyberbattlesim and
      cyborg
      conformance to it (delete their copies) so exactly one copy exists;
      primaite uses it.

    Backend-specific files (evaluator/participant/manifest) carry primaite literals
    and diverge naturally. This is a genuine debt-reducing refactor, covered by the
    existing cbs/cyborg test suites + make verify.

    Tests (TDD, tests/)

    • test_primaite_backend.py — contract-level behaviour with an injected
      FakeDriver (cbs pattern): manifest + realization-envelope validity; target
      construction; provisioner join/rejection; representability rejection of
      blue contracts (positive: admission modelled + typed rejected result +
      identity join; negative: no native op chosen, no native id leaks); aggregate
      reset/termination at 128; evaluator projection (unknown truth, one
      evidence-record + measure, no leakage); cleanup success + failure path
      (quarantine); positive/negative participant & evaluator disclosure boundaries;
      leakage markers (must-not-cross, <primaite, Box(1652)) absent from every
      portable artifact.
    • test_primaite_conformance.py — run_primaite_conformance passes; declared
      weaknesses + source-protocol diagnostics; capability-evidence gaps.
    • Migration-safety: existing test_cyberbattlesim_backend/conformance.py and
      test_cyborg_conformance.py continue to pass after the base extraction.
    • test_base_plumbing.py — extend for the new base helpers.

    Whole-repo / four-lens check

    • Security: source/dependency authority (version + tree digest + origin
      before import), no secret/argv/env/traceback leakage, no network/subprocess/HTTP,
      process-state restore, quarantine on irreversible native mutation.
    • Maintainability: compose the canonical incumbents (base, published RAES
      models, the pinned EvidenceSelection, noxfile extras/verify graph); no new
      DTO/schema/registry; extract shared plumbing rather than a third copy.
    • Extensibility: the seam is an immutable EvidenceSelection + driver
      factory; future action variants belong at the RAES action-contract argument
      seam, future runtimes at the qualified-selection seam — not a generic Gym driver
      or an options mapping in base.
    • Whole-repo: pyproject.toml/single uv.lock/ADR-003; noxfile.py isolated
      extra matrix + clean-wheel identity probe; SonarCloud roots already src/tests;
      the guardrails design note is wired into docs/ nav (mkdocs). Release Please
      owns CHANGELOG.md/version — no manual edits; the eventual PR title is a valid
      Conventional Commit (feat(primaite): ...).

    Verification

    Run the canonical graph once the tree is final:
    uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify (= make verify), plus make policy. Base-only and unrelated-extra installs must remain
    unchanged; the primaite extra isolation is exercised by the existing nox extra
    matrix.

  4. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_codex_review — cycle 1 of 1 (pre-push) on issue #41 (branch 41-primaite-backend)
    Diff mode: inline — the complete diff was supplied in one prompt

    Core review

    Verdict: don't-ship

    The change aims at the right decomposition—backend-local PrimAITE components over published RAES contracts, with reusable identity and evidence mechanics—but it violates its own guardrails at the decisive seams. The fixed native realization sits behind a permissive provisioner, live isolation mutates process-global state, the live driver cannot run on any interpreter supported by this distribution, and base now interprets a named conformance exception. These are structural boundary failures, and the missing closed selection-to-plan join also forecloses the obvious next qualified EvidenceSelection/driver-factory variation.

    Blocking findings (4):

    1. [one-off] Architecture-fit chore: use one pre-push Codex review, one pre-push test-quality review, and disable automatic review disposition #1: Fixed-scenario provisioning accepts unrelated plans — src/raes_adapters/primaite/backend/provisioner.py:37
      Validation rejects only unsupported resource types. Any network/node payload then constructs the fixed data_manipulation source and is recorded as status="applied", even though no check joins the plan to the selected scenario topology, experiment controls, source selection, or realization-envelope digest. The test accepting an arbitrary database-host demonstrates the mismatch: portable state can claim a resource that the native realization never created. Add the closed EvidenceSelection/configuration gate and exact fixed-realization plan validation before construction.
    2. [one-off] Architecture-fit ci: align branch protections, promotion tooling, and external project services #2: In-process isolation mutates process-global state — src/raes_adapters/primaite/backend/driver.py:512
      The driver overwrites HOME and the process current directory for the lifetime of the native session, guarded only by an instance-local lock. Other threads and driver instances can observe those values, and an exception during import or environment construction exits without teardown, leaving the process altered. This directly contradicts the new guardrail that blocks in-process operation unless PrimAITE exposes an explicit path binding. Use such a binding or a separately reviewed worker boundary.
    3. [one-off] Architecture-fit docs: align project documentation with OpenSSF Best Practices #3: Base now owns a semantic conformance exception — src/raes_adapters/base/manifest_evidence.py:144
      This helper recognizes a specific contract, case name, outcome, and diagnostic-code set and decides that the report is admissible for capability evidence. That is conformance policy rather than mechanical projection plumbing, crossing the repository's binding base-layer invariant. The interpretation belongs in the authoritative RAES conformance API; adapters should consume that published result rather than establish the exception in raes_adapters.base.
    4. [one-off] Code-quality chore: use one pre-push Codex review, one pre-push test-quality review, and disable automatic review disposition #1: The live backend cannot run on a supported interpreter — src/raes_adapters/primaite/backend/driver.py:364
      The checked-in qualification is CPython 3.11 while this distribution requires Python 3.12+, and this equality gate therefore rejects every supported production interpreter before construction. The live-driver fixture rewrites the qualification runtime to the current interpreter, so it proves gate mechanics against synthetic evidence rather than production viability. Obtain bounded Python 3.12 qualification evidence or an approved compatible isolation design before exposing the live target.

    Security review

    Verdict: don't-ship

    The change is broadly shaped around a backend-private driver and published RAES boundaries, with source-identity and manifest-evidence mechanics shared appropriately. The security seam is the native workspace: the in-process driver attempts per-target filesystem isolation by changing process-wide home-directory and working-directory state for the environment’s lifetime. That crosses driver and tenant boundaries, and construction or cleanup failures can make the change persistent. Safe concurrency requires explicit source-supported path binding or process isolation.

    Blocking findings (1):

    1. [class] Process-global path changes break tenant isolation — src/raes_adapters/primaite/backend/driver.py:512
      Attacker model: a low-privilege caller who can start or keep a PrimAITE target alive in a long-running process while another request performs home-relative or working-directory-relative file I/O. _establish_workspace() changes the process-wide home directory and current working directory and leaves both changed until cleanup. The per-driver lock cannot protect other threads or other driver instances, and exceptions during environment construction, close, or workspace cleanup can bypass restoration entirely. Consequently, unrelated credential/configuration reads and writes can be redirected into the simulator workspace, enabling cross-request disclosure, policy confusion, or removal of security-relevant artifacts when that workspace is cleaned. Use a source-supported explicit per-instance workspace binding; if none exists, isolate the native runtime in a worker process and fail closed instead of mutating process-global path state.
  5. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_codex_review pre-push cycle 1 of 1 complete for issue #41 on branch '41-primaite-backend'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

  6. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Review decision record — codex cycle 1 (issue #41)

    Reviewer: codex
    Cycle: 1

    Architectural read:

    Core reviewer: The change aims at the right decomposition—backend-local PrimAITE components over published RAES contracts, with reusable identity and evidence mechanics—but it violates its own guardrails at the decisive seams. The fixed native realization sits behind a permissive provisioner, live isolation mutates process-global state, the live driver cannot run on any interpreter supported by this distribution, and base now interprets a named conformance exception. These are structural boundary failures, and the missing closed selection-to-plan join also forecloses the obvious next qualified EvidenceSelection/driver-factory variation.

    Security reviewer: The change is broadly shaped around a backend-private driver and published RAES boundaries, with source-identity and manifest-evidence mechanics shared appropriately. The security seam is the native workspace: the in-process driver attempts per-target filesystem isolation by changing process-wide home-directory and working-directory state for the environment’s lifetime. That crosses driver and tenant boundaries, and construction or cleanup failures can make the change persistent. Safe concurrency requires explicit source-supported path binding or process isolation.

    Blocking findings: 5

    Finding 1 — one-off

    Finding 2 — one-off

    • ID: F2
    • Title: [core] Architecture-fit ci: align branch protections, promotion tooling, and external project services #2: In-process isolation mutates process-global state
    • Location: src/raes_adapters/primaite/backend/driver.py:512
    • Decision: fix
    • Rationale: The driver overwrites HOME and the process current directory for the lifetime of the native session, guarded only by an instance-local lock. Other threads and driver instances can observe those values, and an exception during import or env…

    Finding 3 — one-off

    • ID: F3
    • Title: [core] Architecture-fit docs: align project documentation with OpenSSF Best Practices #3: Base now owns a semantic conformance exception
    • Location: src/raes_adapters/base/manifest_evidence.py:144
    • Decision: fix
    • Rationale: This helper recognizes a specific contract, case name, outcome, and diagnostic-code set and decides that the report is admissible for capability evidence. That is conformance policy rather than mechanical projection plumbing, crossing the …

    Finding 4 — one-off

    Finding 5 — class (2 instances)

    • ID: F5
    • Title: [security] Process-global path changes break tenant isolation
    • Location: src/raes_adapters/primaite/backend/driver.py:512
    • Decision: fix
    • Rationale: Attacker model: a low-privilege caller who can start or keep a PrimAITE target alive in a long-running process while another request performs home-relative or working-directory-relative file I/O. _establish_workspace() changes the proces…
    • Instances:
      • src/raes_adapters/primaite/backend/driver.py:512
      • src/raes_adapters/primaite/backend/driver.py:513
  7. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_codex_review — cycle 2 of 1 (pre-push) on issue #41 (branch 41-primaite-backend)
    Diff mode: inline — the complete diff was supplied in one prompt

    Core review

    Verdict: don't-ship

    The shared manifest-evidence and source-identity extractions are appropriately mechanical base plumbing, while PrimAITE-specific lifecycle and mappings remain backend-local and compose published RAES models. The production shape is nevertheless incomplete: the public target advertises an operational backend although its default driver always refuses construction, while the conformance seam can turn an injected fake driver's portable-mechanics results into affirmative capability evidence. The evaluator also promotes cumulative reward into scores and evidence despite this change's own guardrails stating that the source-member evidence closure for semantic reward projection is not complete. Those are boundary and claim-integrity problems, not implementation polish; the obvious live-driver variation would have to retract or re-prove existing claims rather than simply fill the driver seam.

    Blocking findings (2):

    1. [class] Injected fake-driver conformance can certify an unusable production target — src/raes_adapters/primaite/backend/conformance.py:73
      The canonical conformance entrypoint accepts any injected PrimaiteDriverProtocol and the tests use that seam to establish manifest capability evidence, but create_primaite_target() defaults to PrimaiteDriver and PrimaiteDriver.construct() unconditionally raises after identity verification. This makes a contract-harness double capable of certifying a backend that cannot execute through its public production path, directly contradicting the guardrail that fake-driver tests cannot upgrade live-runtime claims. Implement and qualify the isolated live driver, then obtain capability evidence through that path; alternatively, keep the backend's affirmative runtime capabilities unavailable and prevent injected-driver reports from serving as production claim evidence.
    2. [class] Publishes semantic reward scoring before the source evidence closure exists — src/raes_adapters/primaite/backend/manifest.py:269
      The manifest affirmatively declares scoring, and the evaluator turns DriverEvaluation.cumulative_reward into an evaluation score, evidence summary, and derived measure. The accompanying architecture record explicitly says the current ledger does not close the source members needed for semantic reward projection and requires digest and mutation evidence before consuming them. A fake-provided scalar cannot establish that mapping. Remove these scoring and reward-evidence claims for the current selection, or first extend and validate the backend-local source ledger for the exact live fields and mapping, then exercise that projection against the qualified source.

    Security review

    Verdict: ship

    This change is shaped correctly: the simulator remains behind a private driver boundary, unsafe in-process execution fails closed, installed-source verification is centralized as mechanical plumbing, and portable runtime components use bounded diagnostics and projections rather than native payloads. The change touches supply-chain identity, participant disclosure, evaluator evidence, cleanup, and failure hygiene without introducing a new external request or authentication surface. The driver seam leaves the obvious next variation—a separately reviewed isolated worker implementation—open. I found no concrete, exploitable security issue introduced by the diff.

    No blocking findings.

  8. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_codex_review pre-push cycle 2 (USER-AUTHORIZED OVERRIDE past cap 1) complete for issue #41 on branch '41-primaite-backend'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.
    Override reason: User explicitly authorized one over-cap cycle to re-review the substantial fixes for the 5 cycle-1 blocking findings (fail-closed driver removing process-global HOME/cwd mutation, provisioner realization-envelope join, and moving the conformance-case interpretation out of base). make verify passes on the post-fix tree.

  9. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Review decision record — codex cycle 2 (issue #41)

    Reviewer: codex
    Cycle: 2

    Architectural read:

    Core reviewer: The shared manifest-evidence and source-identity extractions are appropriately mechanical base plumbing, while PrimAITE-specific lifecycle and mappings remain backend-local and compose published RAES models. The production shape is nevertheless incomplete: the public target advertises an operational backend although its default driver always refuses construction, while the conformance seam can turn an injected fake driver's portable-mechanics results into affirmative capability evidence. The evaluator also promotes cumulative reward into scores and evidence despite this change's own guardrails stating that the source-member evidence closure for semantic reward projection is not complete. Those are boundary and claim-integrity problems, not implementation polish; the obvious live-driver variation would have to retract or re-prove existing claims rather than simply fill the driver seam.

    Security reviewer: This change is shaped correctly: the simulator remains behind a private driver boundary, unsafe in-process execution fails closed, installed-source verification is centralized as mechanical plumbing, and portable runtime components use bounded diagnostics and projections rather than native payloads. The change touches supply-chain identity, participant disclosure, evaluator evidence, cleanup, and failure hygiene without introducing a new external request or authentication surface. The driver seam leaves the obvious next variation—a separately reviewed isolated worker implementation—open. I found no concrete, exploitable security issue introduced by the diff.

    Blocking findings: 2

    Finding 1 — class (3 instances)

    • ID: F1
    • Title: [core] Injected fake-driver conformance can certify an unusable production target
    • Location: src/raes_adapters/primaite/backend/conformance.py:73
    • Decision: fix
    • Rationale: The canonical conformance entrypoint accepts any injected PrimaiteDriverProtocol and the tests use that seam to establish manifest capability evidence, but create_primaite_target() defaults to PrimaiteDriver and PrimaiteDriver.construct() …
    • Instances:
      • src/raes_adapters/primaite/backend/conformance.py:73
      • src/raes_adapters/primaite/backend/target.py:52
      • src/raes_adapters/primaite/backend/driver.py:165

    Finding 2 — class (6 instances)

    • ID: F2
    • Title: [core] Publishes semantic reward scoring before the source evidence closure exists
    • Location: src/raes_adapters/primaite/backend/manifest.py:269
    • Decision: fix
    • Rationale: The manifest affirmatively declares scoring, and the evaluator turns DriverEvaluation.cumulative_reward into an evaluation score, evidence summary, and derived measure. The accompanying architecture record explicitly says the current ledge…
    • Instances:
      • src/raes_adapters/primaite/backend/manifest.py:269
      • src/raes_adapters/primaite/backend/conformance.py:55
      • src/raes_adapters/primaite/backend/driver.py:105
      • src/raes_adapters/primaite/backend/evaluator.py:424
      • src/raes_adapters/primaite/backend/evaluator.py:570
      • src/raes_adapters/primaite/backend/evaluator.py:662
  10. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_test_quality_review cycle 1 of 1 — issue #41

    Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
    Branch: 41-primaite-backend
    Cycle: 1 / 1
    Findings: 1

    Finding 1 — [warning] tests/test_primaite_backend.py:747

    Problem: test_cleanup_is_idempotent_verified_and_preserves_bounded_failure asserts failed.cleanup_status in {"failed", "partial"} for a FakeDriver(fail_close=True) cleanup run. Tracing execute_cleanup's _cleanup_status (src/raes_adapters/base/lifecycle.py:125-146) against this fixture's obligations (destroy fails via the raised close(), verify is force-skipped because its depends_on=[destroy] dependency did not succeed) yields exactly "failed" deterministically — "partial" is not a reachable value here (that requires at least one obligation to succeed). The or "partial" branch is dead slack that only weakens the assertion.
    Why it matters: A regression that made _cleanup_status/the cleanup dispatch return "partial" here instead of the correct "failed" would pass silently, even though the directly analogous cyberbattlesim test for the same scenario (tests/test_cyberbattlesim_backend.py:889, assert failed.cleanup_status == "failed") would catch exactly that regression.
    Fix: Tighten to assert failed.cleanup_status == "failed", matching the equivalent cyberbattlesim assertion.

  11. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    gc_test_quality_review cycle 1 of 1 complete for issue #41 on branch '41-primaite-backend'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next gc_test_quality_review invocation to count cycles.

  12. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Review decision record — test-quality cycle 1 (issue #41)

    Reviewer: test-quality
    Cycle: 1

    Architectural read:

    Both files are contract-level black-box tests built around an injected FakeDriver that models the private native simulator boundary, plus a dedicated live-driver test that monkeypatches only importlib.metadata.distribution/load_qualification (never the simulator itself) to exercise the real supply-chain source-identity verification code. This mirrors the production shape: the previously shared _gym_backend/_source_admission abstraction has been de-genericized into two parallel, independently-readable backend implementations (primaite is new, cyberbattlesim's participant_runtime/evaluator/driver got inlined from the same source), and the test files track that shape symmetrically — same FakeDriver pattern, same _LEAKAGE_MARKERS/portable-boundary checks, same tamper-and-reject live-driver scenarios (editable install, byte-level drift, wrong runtime, unselected module origin). The tests earn their assurance by actually driving the driver through rejection boundaries and asserting the specific outcome rather than checking for a rule's mere existence, which is the right level for security-adjacent source-identity and hidden-truth-leak enforcement. The one place this discipline slips is a single loosened assertion in the primaite cleanup test that the directly-analogous cyberbattlesim test gets right — not a structural problem, just a one-off that should be tightened to match its sibling.

    Blocking findings: 1

    Finding 1 — one-off

    • ID: F1
    • Title: (no title)
    • Decision: fix
    • Rationale: Addressed by next cycle
  13. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Pre-PR base synchronization

    • Source: refs/remotes/origin/dev at 0166f2943e19d9b3ff86c1d383bfe94525ec6b28
    • Outcome: merged_conflicts_resolved
    • Published feature head: 60e8c17d18fb7dd4a06bfb9fe095dfcde7608587
    • Verified tree: 55c2f5fc004028e7833bb84515be3d718a10fc73
  14. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Pre-PR base synchronization

    • Source: refs/remotes/origin/dev at 0166f2943e19d9b3ff86c1d383bfe94525ec6b28
    • Outcome: already_current
    • Published feature head: 80149464fef06d1f1a4d6924e84c603aa24999c5
    • Verified tree: bb276cc4794fe4f8041d19b11c407401c96d3f90
  15. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Pre-PR base synchronization

    • Source: refs/remotes/origin/dev at 0166f2943e19d9b3ff86c1d383bfe94525ec6b28
    • Outcome: already_current
    • Published feature head: c0577b0f905661f08dbda5193b26e356924ff054
    • Verified tree: 31618f2b36b715d971db91af76944d11e2062b97
  16. Brad-Edwards commented on Aug 3, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Ready for review — issue #41

    PR: #73

    Outcome

    Ships raes_adapters.primaite as a fail-closed conformant backend over the selected PrimAITE data_manipulation profile: the live driver verifies source identity, then refuses in-process construction (PrimAITE writes platform dirs on import; the qualified runtime is 3.11 while the distribution needs 3.12+). For claim integrity it certifies no live capability (every one an open gap) and withholds the cumulative BLUE reward pending evidence closure. It composes dev's shared gym-backend framework, so no shared machinery is duplicated. All gates are green.

    Files changed

    Added:

    • docs/decisions/primaite-backend-guardrails.md
    • src/raes_adapters/primaite/backend/__init__.py
    • src/raes_adapters/primaite/backend/_diagnostics.py
    • src/raes_adapters/primaite/backend/cleanup.py
    • src/raes_adapters/primaite/backend/conformance.py
    • src/raes_adapters/primaite/backend/driver.py
    • src/raes_adapters/primaite/backend/evaluator.py
    • src/raes_adapters/primaite/backend/manifest.py
    • src/raes_adapters/primaite/backend/orchestrator.py
    • src/raes_adapters/primaite/backend/participant_runtime.py
    • src/raes_adapters/primaite/backend/provisioner.py
    • src/raes_adapters/primaite/backend/target.py
    • tests/test_primaite_backend.py

    Modified:

    • docs/index.md
    • mkdocs.yml
    • pyproject.toml
    • src/raes_adapters/_experiment_evidence.py
    • src/raes_adapters/_gym_backend/evaluator.py

    Reviews

    • codex: Architecture review over two cycles: reworked the isolation seam into a fail-closed driver (no process-global HOME/cwd mutation), and retracted over-claimed reward scoring and fake-driver conformance certification. Both resolved.

    Traceability reconciliation

    • Pending — requirement status transition and IMPLEMENTS/TESTS reconciliation run in Phase E once the PR merges.

    Status

    • CI: ✅ green
    • SonarCloud: ✅ passed
    • PR ready for user review and merge. Ground Control reconciliation (requirement status + traceability) runs on merge (Phase E).
  17. Brad-Edwards commented on Aug 4, 2026

    @Brad-Edwards
    CollaboratorAuthor

    Final report — issue #41 complete

    PR: #73
    Plan: #41 (comment)

    Outcome

    raes_adapters.primaite ships as a fail-closed conformant backend over the selected PrimAITE data_manipulation profile. Maintainers can install the primaite extra and construct the target through published RAES contracts; the live driver verifies source identity, then refuses in-process construction (PrimAITE writes platform dirs on import; its qualified runtime is CPython 3.11 while the distribution needs 3.12+). No capability is over-claimed — each is a disclosed open gap and the BLUE reward is withheld pending evidence closure, leaving a clean seam for a future isolated-worker driver.

    Files changed

    Added:

    • docs/decisions/primaite-backend-guardrails.md
    • src/raes_adapters/primaite/backend/__init__.py
    • src/raes_adapters/primaite/backend/_diagnostics.py
    • src/raes_adapters/primaite/backend/cleanup.py
    • src/raes_adapters/primaite/backend/conformance.py
    • src/raes_adapters/primaite/backend/driver.py
    • src/raes_adapters/primaite/backend/evaluator.py
    • src/raes_adapters/primaite/backend/manifest.py
    • src/raes_adapters/primaite/backend/orchestrator.py
    • src/raes_adapters/primaite/backend/participant_runtime.py
    • src/raes_adapters/primaite/backend/provisioner.py
    • src/raes_adapters/primaite/backend/target.py
    • tests/test_primaite_backend.py

    Modified:

    • docs/index.md
    • mkdocs.yml
    • pyproject.toml
    • src/raes_adapters/_experiment_evidence.py
    • src/raes_adapters/_gym_backend/evaluator.py

    Reviews

    • codex: Two cycles (one over-cap, user-authorized); all blocking findings fixed: fail-closed driver, provisioner realization-envelope join, conformance moved out of base, over-claimed reward/certification retracted. Security: ship.
    • test-quality: One cycle, one warning: a loosened cleanup-status assertion was tightened to == "failed" to match the analogous cyberbattlesim test. Resolved.

    Traceability reconciliation

    • IMPLEMENTS / TESTS / DOCUMENTS added: 0
    • Links updated: 0
    • Stale links removed: 0

    Requirement-free run: issue #41 has no Requirements section and the aces-adapters GC project holds no local requirement files (docs/requirements/ does not exist). Steps 15 and 16 are no-ops.

    Status

    • CI: ✅ green
    • SonarCloud: ✅ passed
    • PR ready for user review and merge.
  18. removed
    in-progressAn agent is actively working this issue via /implement
    on Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions