diff --git a/README.md b/README.md index 41dcdce..d40318b 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,10 @@ A single distribution, **`raes-adapters`**, that qualifies and realizes [RAES](https://github.com/RAESystem/rae) scenarios against concrete simulator backends. It ships shared adapter plumbing plus one importable module per simulator. Implemented simulator dependencies are exposed as optional extras; -qualification evidence can fail closed before such an extra is advertised. +qualification evidence bounds the claims made for each selected backend. +Maintainer selection determines admission: a qualification record documents +source identity, attainable evidence, limitations, and claim strength, but it +does not veto implementation of a selected simulator. RAES — Reproducible Agentic Environments System — is the semantic authority. Its scope is agentic environments generally: cyber, AI security, AI safety, @@ -31,22 +34,28 @@ pip install raes-adapters # shared base plumbing + qualification evidence ``` The selected CybORG backend is admitted and usable through its documented -source installation. The `cyborg` extra key is reserved but currently empty. Issue +source installation. The `cyborg` extra key is dependency-light. Issue [#12](https://github.com/RAESystem/adapters/issues/12) qualified the official CAGE Challenge 2 source and a packaging-only fix, but the upstream wheel omits the version and Scenario2 runtime data. The fixed wheel passed a clean Python -3.12 smoke. Until that fix is published, install the pinned CAGE-2 source at +3.12 smoke locally, but it is not a governed public artifact and the +distribution's declared Python/platform range is not yet qualified. Those +limitations bound installability and reproducibility claims; they do not veto +the maintainer-selected backend. Until that fix is published, install the +pinned CAGE-2 source at commit `26ce1c1253fa9e2e73f25e6a7f2da32860c11257`, apply `src/raes_adapters/cyborg/cage2-wheel-package-data.patch`, and install its `CybORG/` package into the environment. The adapter validates the installed version and selected source-file digests before constructing the backend. +The empty dependency list avoids advertising an editable checkout, +install-time clone, or unpublished wheel as an automatic installation route. -There is also intentionally no `cyberbattlesim` extra. The +The `cyberbattlesim` extra is likewise dependency-light. The [qualification record](src/raes_adapters/cyberbattlesim/qualification.json) -found Microsoft's source legally usable and runnable, but not currently -admissible: no official index/release artifact exists, the public evaluator -does not bind every random stream, and material upstream benchmark findings -remain open. +binds Microsoft's legally usable, runnable source and admits the selected +profile. Because no official index/release artifact exists, users install the +pinned simulator source separately. Unbound random streams and open benchmark +findings remain explicit limits on deterministic-replay and outcome claims. ## One distribution, optional simulator extras @@ -116,8 +125,9 @@ raes-adapters/ ## Program status -This repository is stood up under **REP-002** (RAES issue #636). The modules are -buildable skeletons; adapter logic is downstream: +This repository was stood up under **REP-002** (RAES issue #636). Shared +plumbing and the CyberBattleSim backend are implemented; remaining simulator +backends land issue by issue: | Requirement | Scope | |-------------|-------| @@ -153,11 +163,59 @@ CI on source drift, a missing category, a duplicate row, an unresolvable target, a broken cross-artifact reference, an undisclosed loss, or leakage of a known native identifier or object representation (raw native arrays, reward vectors, and action ids are excluded structurally by the closed RAES models). The -evidence set does not change -the `not-admissible` decision or claim installability, replay, or equivalence; +evidence set does not turn source identity into a claim of dependency +installability, deterministic replay, or outcome equivalence; the [scenario/ledger guardrails](docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md) fix its boundaries. +## CyberBattleSim backend + +Issue [#27](https://github.com/RAESystem/adapters/issues/27) implements a RAES +runtime target for the admitted size-10 `CyberBattleChain-v0` profile: + +```python +from raes_adapters.cyberbattlesim.backend import create_cyberbattlesim_target +from raes_adapters.base import run_conformance_probe + +target = create_cyberbattlesim_target(seed=20260729) +report = run_conformance_probe(target) +``` + +Target creation is dependency-light and does not import the simulator. +Provisioning verifies the qualified simulator import-root identity and selected +dependency wheel identities, complete installed import-root trees (including +native libraries and unexpected files), +critical-file digests, dependency versions, and module origins for the directly +imported source, Gymnasium, and NumPy packages before importing and constructing +the separately installed source. The locally built simulator wheel is admitted +by its complete root because upstream publishes no reproducible wheel artifact; +direct Gymnasium/NumPy wheel installs must also match the recorded archive hash. +Editable/directory installations, symlinks, unqualified direct dependency +artifacts, absence, or an identity mismatch become a bounded +RAES diagnostic (and a source-backed conformance probe therefore fails rather +than pretending to run). The +reference processor compiles the checked-in scenario against the manifest, and +the shared target then realizes the applicable provisioning, orchestration, +participant, evaluation, observation, and cleanup surfaces. + +One admitted attacker action maps to at most one serialized native `env.step`. +The selected scan-and-reimage defender runs source-internally during that step; +it is not exposed as a second participant-admitted transition. +Native observations, masks, credentials, action coordinates, `info`, reward +vectors, and exceptions remain driver-private. Participant observations and +typed action results carry only RAES references admitted by their disclosure +boundary. Because the representative authored topology cannot identify the +selected native action coordinate, requested targets remain intent and are not +echoed as realized effect targets. Cumulative reward is evaluator-owned. Seed +bindings report the Gym environment and action-space streams as applied and the +Python/NumPy global streams as unbound. These controls improve run attestation +and bound repeatability without claiming byte-identical replay of a stochastic +experiment. + +The [backend architecture guardrails](docs/decisions/cyberbattlesim-backend-guardrails.md) +record the component ownership, failure hygiene, capability claims, and +acceptance-test mapping. + ## CybORG/CAGE-2 runtime qualification Issue [#12](https://github.com/RAESystem/adapters/issues/12) selects the diff --git a/docs/decisions/cyberbattlesim-backend-guardrails.md b/docs/decisions/cyberbattlesim-backend-guardrails.md new file mode 100644 index 0000000..01719b8 --- /dev/null +++ b/docs/decisions/cyberbattlesim-backend-guardrails.md @@ -0,0 +1,211 @@ +# CyberBattleSim backend architecture guardrails + +GitHub issue #27 is the authority for the implemented backend deliverable. This +record fixes the boundaries that implementation must respect; the document +itself defines neither an implementation plan nor a new RAES contract. + +## Admission and claim qualification + +The maintainer-selected source profile is admitted for issue #27. The canonical +`qualification.json` records `admission.decision = "admitted"` and preserves +these limitations: + +- no official index or release artifact for the selected immutable source; +- incomplete binding of the Gym environment, action-space, Python-global, and + NumPy-global random streams through reset and evaluation; and +- unresolved upstream benchmark/evaluation defects. + +The scenario and source ledger deliberately preserve those limitations. +Implementation may attest source identity, portable configuration, and observed +run evidence while reporting partial stochastic control and stochastic-bounded +outcome reproduction. It must not claim published dependency installability, +deterministic replay, or scientific equivalence that the evidence does not +support. A future patch or repackaging becomes part of the qualified identity +and must carry source, tree, patch, artifact, license, notice, dependency, and +behavioral evidence; none of that is a precondition for implementing the +selected adapter. + +## Keep the concepts separate + +| Concept | Boundary | +| --- | --- | +| Qualification and public protocol | Immutable backend-local source evidence, maintainer admission, and attainable claim strength. They are not a manifest, scenario, or runtime result. | +| Authored scenario | Portable RAES intent. The representative four-node authored topology is not the generated native `size=10` topology, an instantiated scenario snapshot, or live simulator state. | +| Source ledger and loss disclosures | Backend-local traceability and claim bounds. They are not portable state and do not extend RAES vocabulary. | +| Backend manifest | A truthful declaration of implemented RAES capabilities and realization limits, serialized with the published manifest-v2 model. It is not a source qualification record. | +| Provisioned target | RAES runtime state and references resulting from a plan. It is not a native Gym environment, reset observation, or hidden source truth. | +| Participant action | A validated RAES admission request and typed terminal result. Its contract address and action-instance identity are not a native action id. | +| Orchestration, time, and simulator execution | Workflow progress, logical clock transitions, and a native `env.step` are distinct. Exactly one component owns each native transition. | +| Participant observation and evaluation | Participant-relative disclosed information is separate from evaluator-only facts, objective truth, reward, evidence, and derived measures. | +| Termination and cleanup | Source termination, truncation, evaluator cutoff, ownership/SLA outcomes, participant termination, and verified resource cleanup are distinct facts. | + +Do not hide a mismatch between these layers behind a generic “environment,” +“step,” “result,” “profile,” or “done” abstraction. + +## Compose the canonical incumbents + +`raes_adapters.base` remains plumbing and published RAES contracts remain the +semantic owners. The backend must build on these incumbents instead of copying +their DTOs, validators, registries, receipts, or workflow logic: + +| Concern | Canonical incumbent and required use | +| --- | --- | +| Selected evidence | The existing `EvidenceSelection`, `qualification.json`, `public-protocol.md`, authored SDL, experiment task/spec, source ledger, and loss disclosures under `raes_adapters.cyberbattlesim`. Consume one admitted immutable selection; do not repeat its constants in runtime code. | +| Packaging and isolation | `pyproject.toml`, the single `uv.lock`, ADR-003, and `_extras()` / `_verification_envs()` in `noxfile.py`. A real `cyberbattlesim` extra is verified alone and must not change a base-only import or install. Add `tool.uv.conflicts` only for a demonstrated mutually incompatible extra stack. | +| Manifest | `raes_backend_protocols.backend_manifest.BackendManifest`, the capability models and validators, `backend_manifest_v2_model()` / `backend_manifest_payload()`, and the RAES-owned supported-contract catalog. Do not hand-author manifest JSON, capability vocabularies, or a supported-contract table. | +| Realization disclosure | `BackendRealizationEnvelopeModel` and its digest/cross-concern validation. Disclose the authored-to-native topology transformation and every applicable realization concern; do not claim exact topology realization when the evidence supports only a representative pattern. | +| Runtime target | `raes_adapters.base.build_runtime_target()` over `raes_runtime.registry.RuntimeTarget` and `RuntimeTargetComponents`. Let RAES check component presence, capability agreement, and method signatures; add no backend registry. | +| Runtime lifecycle and validation | `raes_runtime.RuntimeManager` / `RuntimeControlPlane` and the published provisioning, workflow, evaluation, proposition, participant, time, snapshot-transition, and `ApplyResult` gates. Never invoke private `_call_backend_*` helpers or fork their validation. | +| Participant lifecycle | `raes_backend_protocols.participant_runtime_base.BaseParticipantRuntime` and `ParticipantNativeActionExecution`. Reuse its initialize/reset/restart/terminate state machine, participant/episode identities, generation, append-only history, sequence, linkage, status, and results. The backend-specific seam is `_model_action`, not a second lifecycle controller. | +| Action admission and result | `ParticipantActionAdmissionRequest`, the published implementation-selection/exposure/evidence validators, `ParticipantActionResultModel`, and `autonomous_action_result_violation()`. Every admitted request must yield one typed terminal result matching its participant, episode, action instance, contract, generation, and temporal observation point. | +| Observation and retrieval | `ParticipantObservationEnvelopeModel`, published observation-boundary validation, `ParticipantRetrievalMixin`, and the participant-scoped context/status/history view models. Do not define a custom observation or history DTO, and do not return native dictionaries or masks. | +| Logical time | `ReferenceTimeRuntime`, `TimeCoordinator`, `apply_logical_clock_transition()`, and RAES time conformance validation. Define the explicit source-event-to-logical-transition mapping; never infer that one action or `env.step` is one RAES tick. | +| Stochastic controls | The checked-in experiment controls, published stochastic-control models, and `apply_seed_controls()` when an executable binding is supplied. The selected public task deliberately records descriptive controls without executable bindings; the driver separately reports which runtime seed calls were applied or remain unbound. A setter call or top-level integer is not a replay guarantee. | +| Workflow and evaluation | The compiled workflow contracts, `WorkflowExecutionStateModel`, evaluation lifecycle/history models, proposition/objective truth models, and their RAES validators. Orchestrator receipts do not substitute for participant outcomes or evaluator evidence. | +| Evidence and measures | `ExperimentEvidenceRecordModel`, `ExperimentRawEvidenceContentModel`, `ExperimentDerivedMeasureModel`, and `ExperimentRunModel`. Keep source reward/result facts, objective truth, evidence, derived measures, limitations, and evaluator lifecycle in their distinct published shapes. | +| Cleanup | `TrialCleanupPlanModel`, `CleanupCapabilities`, `require_cleanup_plan_capability()`, `execute_cleanup()`, `TrialCleanupReceiptModel`, and `validate_trial_cleanup_receipt()`. Native `close`/destroy/verify operations remain driver-local, deterministic, idempotent, and failure-preserving. | +| Receipts and persistence | `RuntimeControlPlane` operation receipts/statuses and `ControlPlaneStore`. Use the existing idempotency, audit, and persistence seam; create no CyberBattleSim receipt, repository, cache, database, or evidence store. | +| Failure hygiene and conformance | Base projection/redaction helpers, RAES `Diagnostic` / `DiagnosticModel` / `ApplyResult`, and `run_target_conformance()`. Return bounded portable failures and the canonical conformance report; do not create an exception hierarchy, fallback native payload, custom case family, or second report. | + +## Driver boundary and ownership + +The legitimate backend-specific seam is a private, in-process driver bound to +one admitted evidence selection and one native environment lifecycle. It may +hold native environment objects, observations, action masks, identifiers, +arrays, reward components, hidden truth, and compensation state. None of those +values crosses into a RAES contract, diagnostic, receipt, history, log, +exception message, or stable reference. + +Provisioner, Orchestrator, ParticipantRuntime, and Evaluator may compose that +driver, but native state has one owner. One admitted participant action maps to +exactly one driver operation and at most one source transition. The +Orchestrator must not replay the action, the Evaluator must not advance the +environment while projecting facts, and observation projection must not +trigger another source call. + +The driver must serialize access unless the qualified source proves safe +bounded concurrency; the manifest must not claim concurrent execution merely +because RAES supports it. Portable inputs should be completely admitted before +native mutation. After mutation, projection and portable result validation must +either succeed or leave the episode in an explicit failed/terminated state with +compensation or cleanup available. RAES rolls back an invalid portable +snapshot, but it cannot roll back a mutated native simulator. + +Reset must bind every declared random stream and distinguish reset from restart, +new episode identity, deterministic replay, termination, and cleanup. Cleanup +must be safe after partial construction, failed admission, projection failure, +termination, repeated calls, and ordinary success. `RuntimeManager.destroy()` +stopping components and removing portable provisioning entries is not, by +itself, verified native cleanup. + +## Cross-cutting validation and security path + +The intended design is an in-process optional backend. It introduces no new +authentication service, HTTP route, environment-variable configuration, CLI, +subprocess, network fetch, or durable store. + +| Layer the design passes | How it passes | +| --- | --- | +| Source/admission and dependency authority | The admitted qualification identity remains package evidence. `pyproject.toml` publishes a dependency-light adapter extra because upstream provides no index/release artifact; a separately installed simulator must match the qualified complete import-root tree (including native libraries and unexpected files), critical-file digests, and the qualified Gymnasium/NumPy wheel identities, complete roots, versions, and module origins. The locally built simulator wheel is admitted by its complete root because upstream publishes no reproducible wheel artifact; a direct Gymnasium/NumPy archive install must also match the selected public wheel hash. Editable/directory/VCS installations, symlinks, unqualified direct dependency artifacts, hidden patches, and dependency state received transitively from another extra fail the driver boundary. The remaining normalized dependency graph is recorded evidence, not a runtime authenticity claim. | +| Contract parsing and configuration shape | SDL uses the bounded RAES parser and compiler; experiment and runtime values use closed published models (`ContractModel`, generally `extra="forbid"`) and existing cross-artifact joins. No environment override, arbitrary import path, permissive mapping, duplicate schema, or duplicated enum list may bypass them. | +| Manifest and target shape | Backend manifest capability, contract-family, realization-envelope, digest, component-presence, and callable-signature validation run before target use. Unsupported facts are rejected or represented as limitations, never advertised optimistically. | +| Plan, transition, and result envelopes | `RuntimeManager` / `RuntimeControlPlane` validate planning identity, baseline snapshots, `ApplyResult`, changed addresses, workflow/evaluation/proposition state, participant history, time readback, and realization disclosure. Invalid portable output fails closed and does not become a success receipt. | +| Participant admission and disclosure | Action contract, implementation selection, target addresses, generation, evidence references, exposure policy, terminal result, sequence/linkage, and participant/episode scope use the published validators. Positive and negative tests must prove that another participant's data and evaluator-only/hidden references cannot enter views or observations. | +| Secret handling | The driver neither reads a repository/user secret store nor accepts credentials as runtime configuration. Source-native learned credentials remain opaque driver state. Tokens, secret values, environment dumps, and authentication material never enter argv, logs, evidence, snapshots, diagnostics, or portable histories. | +| OS/process exposure | Native code is imported lazily only when the extra-backed target is constructed, after the selected distribution and directly imported Gymnasium/NumPy artifacts have matched qualified versions, full installed import-root trees, applicable direct-archive identities, and module origins. Complete-root enumeration rejects unlisted source, bytecode, extension, native-library, data, and symlink entries before driver import. Base import/install does not import the simulator. Execution uses no shell, child process, inherited configuration dump, home-cache authority, checkout-relative import, or runtime network fetch. If a future qualified driver requires a subprocess, reuse the repository's confined temporary directory, explicit argument vector, small environment allowlist, cleared `PYTHONPATH`, safe-path, timeout, and bounded-output pattern. | +| Error and log envelopes | Base default-deny redaction and RAES diagnostics/results are the only portable failure path. Never use `str`/`repr` on native values; native exceptions contribute at most a safe type name, without args, causes, locals, paths, stdout/stderr, or traceback. Logs are limited to bounded published addresses, safe operation names, counts, and dispositions. | +| Authentication/authorization if later exposed | Reuse `create_control_plane_app()` with `ControlPlaneSecurityConfig.strict_defaults`, verified identity, role/target authorization, request-size limits, denial audit, and its redacted exception handler. Do not create an adapter-specific endpoint or weaker auth/config surface. | +| Persistence and idempotency | `ControlPlaneStore` and control-plane receipts remain the operation-persistence boundary. Backend-native state is lifecycle-scoped and is closed/verified through cleanup; it is not persisted as a second source of truth. | +| Distribution and workflow | Extend the existing `_distributions()` / `probe_installed_identity.py` clean-wheel proof and existing tests, docs, type, policy, and `verify` graph. The base environment and each extra remain isolated. Do not add a second lock, build probe authority, CI workflow, or required-check path. | + +## Extensibility seam + +The external parameter is an admitted immutable `EvidenceSelection` (or a +future selection with the same package-local role) plus a driver factory for +that selection. Action translation is keyed by published action-contract +address and validated portable arguments/targets; observation projection is +keyed by the published participant boundary; termination/evaluation mapping is +keyed by the selected source protocol. These are backend-private mappings, not +new base registries or portable schemas. + +A second qualified CyberBattleSim scenario, policy, or source artifact should +be addable as another immutable selection and driver configuration without +editing a cross-simulator registry, copying participant lifecycle logic, or +changing RAES contracts. Source-version-specific compatibility stays behind +the driver factory and remains tied to qualification evidence. Do not +generalize this seam into `raes_adapters.base` until a real cross-backend +mechanical pattern exists and RAES does not already own it. + +## Gotchas and anti-patterns + +- Do not turn an admitted selection into a deterministic-replay, dependency + installability, or outcome-equivalence claim without matching evidence. +- Do not treat source import/smoke success, finite conformance, or one clean run + as proof of deterministic replay, scientific validity, or outcome + equivalence. +- Do not conflate native `terminated`, Gym `truncated`, evaluator cutoff, + ownership, defender SLA failure, eviction, participant terminal status, and + cleanup outcome. Preserve precedence and the recorded off-by-one limitation. +- Do not turn reward into an action outcome, objective truth, evaluator + lifecycle state, evidence, or a derived measure. Project each fact through + its published owner and retain limitations. +- Do not expose native action ids, node ids, tuples, arrays, masks, + credential-cache contents, observations, `info`, logs, paths, hidden truth, + exceptions, arguments, environment values, credentials, or tracebacks as + portable data or “stable” references. Stable source references are + qualification/ledger identities and digests. +- Do not attribute an effect to the request's portable target when the private + source coordinate cannot be joined to that target. Preserve the request as + intent and leave realized effect target references empty. +- Do not duplicate manifest payloads, RAES schemas/validators, participant + views or histories, operation receipts, time state, cleanup receipts, + diagnostics, stores, conformance cases, or workflow/evaluator controllers. +- Do not override `BaseParticipantRuntime` lifecycle methods merely to mirror + the simulator API. Claim coordinated reset only if the native implementation + supplies real atomic multi-participant reset and rollback. +- Do not let Provisioner reset the participant episode, Orchestrator execute the + admitted action a second time, or Evaluator advance or mutate the simulator. +- Do not return portable success after a native mutation whose projection or + RAES validation failed. Treat compensation and cleanup failures as distinct + from the primary operation failure and preserve both safely. +- Do not claim a capability without executable evidence, including reset, + stochastic control, exposure, observation sealing/capture, objective truth, + cleanup verification, or concurrency. Reject unsupported requests and + disclose bounded limitations. +- Do not add process-global random seeding, working-directory dependence, + import-time simulator discovery, broad environment forwarding, unbounded + logging, a catch-all success fallback, or a backend-independent native + simulator abstraction. + +## Non-goals and implementation boundaries + +- This preflight does not select or approve a replacement artifact, patch, + repackaging route, or defect disposition; those belong to requalification. +- This guardrail record is not itself the optional extra, driver, target, + manifest, protocol surface, projection, cleanup result, test, or conformance + report; those remain executable repository artifacts. +- It does not change the authored RAES scenario to imitate native `size=10` + state or upgrade the recorded equivalence tier. Any mapping loss remains + explicit. +- It does not add semantics to `raes_adapters.base`, change published RAES + contracts, create a generic Gym adapter, or make CyberBattleSim concepts + shared authority. +- It adds no authentication mechanism, HTTP controller, CLI, daemon, + subprocess launcher, plugin discovery, environment-binding format, + repository/database/cache, second distribution/lockfile, or CI workflow. + +## Issue #27 acceptance evidence + +Qualification limitations bound the assertions in this table; none is an +admission gate. + +| Acceptance concern | Executable evidence | +| --- | --- | +| Selected scenario and target realization | `test_selected_scenario_realizes_and_runs_across_applicable_surfaces` compiles the checked-in size-10 scenario with `ReferenceProcessor` and exercises provisioning, orchestration, participant action, evaluation, and verified cleanup through one target. | +| Truthful manifest and published RAES contracts | `test_manifest_and_target_claim_exact_implemented_surfaces` validates manifest-v2 projection and constrained capability declarations; `test_target_passes_canonical_backend_conformance_probe` requires every canonical finite case to pass with no contract or capability gap. | +| Exactly one native transition per admitted action | `test_each_admitted_action_has_one_driver_operation_and_typed_terminal_result` proves the portable identity/result join without falsely attributing the private native coordinate to the requested portable target; `test_live_driver_is_lazy_seed_bounded_and_performs_exactly_one_native_step` verifies qualified complete artifact roots, critical source digests, selected Gymnasium/NumPy artifacts and module origins before source import, rejects an added extension artifact, then probes the live-driver seam, selected source configuration, and one native transition. | +| Participant/evaluator separation and no native leakage | `test_action_boundary_withholds_unrequested_evidence_and_native_failures` and `test_evaluator_reads_distinct_facts_without_advancing_or_leaking_to_participant` cover evidence admission, generic failure projection, reward ownership, non-mutating evaluation, and distinct internally joined artifact identities for successive projections. | +| Unsupported behavior fails before source mutation | `test_unsupported_action_is_rejected_before_native_mutation` and `test_unsupported_provisioning_is_rejected_before_construction` bind negative behavior to the declared capability surface. | +| Bounded stochastic controls, termination, and restart | `test_live_driver_is_lazy_seed_bounded_and_performs_exactly_one_native_step` records applied and unbound streams plus evaluator cutoff; `test_source_termination_closes_episode_and_restart_resets_same_seed` keeps source termination and participant restart distinct. | +| Idempotent, failure-preserving cleanup | `test_cleanup_is_idempotent_verified_and_preserves_bounded_failure` covers repeated cleanup, verification, required/best-effort outcomes, and native-error redaction; `test_provisioner_reconciles_live_driver_and_orchestrator_never_steps_driver` proves create → cleanup/destroy → create reuses the advertised target safely. | +| Optional dependency isolation | `test_backend_import_is_dependency_light_and_lazy` proves that importing the backend loads neither CyberBattleSim nor Gymnasium. The canonical distribution matrix verifies base, `cyberbattlesim`, and other extras independently from the single lock. | diff --git a/docs/decisions/cyberbattlesim-qualification-guardrails.md b/docs/decisions/cyberbattlesim-qualification-guardrails.md index 4491d5b..4ad4fc9 100644 --- a/docs/decisions/cyberbattlesim-qualification-guardrails.md +++ b/docs/decisions/cyberbattlesim-qualification-guardrails.md @@ -1,6 +1,6 @@ # CyberBattleSim qualification guardrails -Issue #25 is the authority for the qualification outcome. This note fixes the +Issue #25 is the authority for the qualification evidence. This note fixes the repository boundaries the qualification must respect; it is not an experiment selection, qualification record, or implementation plan. @@ -16,8 +16,9 @@ The implementation must not collapse these different artifacts into a generic - The **qualification record** identifies the immutable upstream source/runtime, legal and maintenance disposition, known defects, patches, and the - admissibility decision for `RAESystem/research#12`. It is backend-local - evidence, not a RAES schema or conformance claim. + maintainer's admission plus attainable claim strength for + `RAESystem/research#12`. It is backend-local evidence, not a RAES schema, + policy gate, or conformance claim. - The **public experiment protocol** identifies the selected scenario, baseline participant(s), basic defender, evaluator, reset semantics, stochastic controls, metrics, and termination rules. If it is emitted in machine-readable @@ -58,15 +59,16 @@ immutable source identity. The record must bind: - upstream and dependency licenses at the pinned revisions, attribution and notice duties, redistribution permission, retained-output permission, maintenance/archive status, and the evidence used for each conclusion; and -- a dated `admissible` or `not admissible` decision for - `RAESystem/research#12`, scoped to exactly the recorded source and protocol. +- the dated maintainer admission for `RAESystem/research#12`, scoped to exactly + the recorded source and protocol, plus explicit limitations on installability, + stochastic control, run attestation, and outcome reproduction. An immutable Git source and a publishable Python dependency are separate questions. Do not put a floating Git reference, local path, install-time clone, or unverified direct URL into the published extra. If the official source has no publishable installation route that reproduces the selected commit, the record -must expose that gap; an empty or Python-excluded extra is not a passing -qualification. +must expose that gap. The gap weakens apparatus-reconstruction claims; it does +not veto a maintainer-selected adapter or make a dependency-light extra false. Compatibility patches are part of the identity. Record the unmodified commit, patch digest, patched-tree or resulting-artifact digest, purpose, license, @@ -102,9 +104,10 @@ not setup conveniences. The extension seam is the selected qualification/protocol artifact identity passed to a module-local smoke runner. Scenario, participant, evaluator, seed, metric, or termination values must be read from that one canonical selection, -not repeated in tests, scripts, notebooks, and prose. A second admissible public -protocol should be addable as another immutable selection without changing the -runner or inventing a central simulator registry. +not repeated in tests, scripts, notebooks, and prose. A second +maintainer-selected public protocol should be addable as another immutable +selection without changing the runner or inventing a central simulator +registry. ## Security and observability boundary @@ -142,7 +145,7 @@ facts are sufficient to prove the smoke behavior. CyberBattleSim dependency transitively. - Do not treat a successful import, notebook execution, single episode, or CI pass as proof of reproducibility, determinism, legal usability, scientific - validity, or admissibility. + validity, or outcome equivalence. - Do not conflate simulator `done`/`terminated`/`truncated`, goal satisfaction, defender SLA failure, maximum steps, evaluator cutoff, and participant stop conditions. Record their identities, precedence, and off-by-one behavior. diff --git a/docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md b/docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md index 955132b..105f486 100644 --- a/docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md +++ b/docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md @@ -11,7 +11,7 @@ The deliverable is a package-local evidence set under - `qualification.json` remains the source of truth for the selected repository, commit, tree, archive and file digests, legal disposition, known defects, and - admissibility outcome. + admission and attainable claim strength. - `public-protocol.md` remains source-native evidence for the selected CyberBattleSim case. - The authored RAES SDL carries portable scenario meaning only. @@ -30,10 +30,10 @@ may be authored in SDL, while numeric reward, evaluator metrics, episode cutoffs, and random-stream controls belong in companion published experiment contracts. -The current qualification is `not-admissible`. Authoring and validating this -evidence set does not change that decision and must not be described as -installability, runnable adapter support, deterministic replay, backend -conformance, or scientific equivalence. +The maintainer-selected profile is admitted. Authoring and validating this +evidence set strengthens provenance but must not be described as dependency +installability, deterministic replay, backend conformance, or scientific +equivalence beyond the evidence actually captured. ## Reuse the published semantic owners diff --git a/docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md b/docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md index 8f1880a..66e55e2 100644 --- a/docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md +++ b/docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md @@ -70,13 +70,14 @@ root license is not by itself evidence that every nested or separately sourced asset can be redistributed. Dependency licenses and notice obligations remain part of the resolved profile. -If the `cyborg` extra is populated and advertised as an automatic installation -route, it must be installable from the published `raes-adapters` artifact on the -declared Python/platform boundary. A successful editable checkout, local path +The dependency-light `cyborg` adapter extra must be installable from the +published `raes-adapters` artifact on the declared Python/platform boundary. +That proves the adapter install surface, not automatic acquisition of the +native simulator. Keeping the extra empty does not prohibit the documented +source-install route. If the extra is later populated and advertised as an +automatic installation route, a successful editable checkout, local path dependency, floating branch or tag, install-time clone, or warm user cache is -not proof of that packaging claim. Keeping the extra empty does not prohibit -the documented source-install route. If official source cannot produce a -publishable immutable dependency, any governed repackaging or maintained fork +not proof of that packaging claim. Any governed repackaging or maintained fork must preserve notices and bind: - the unmodified upstream commit and tree; diff --git a/docs/decisions/cyborg-cage2-source-ledger-guardrails.md b/docs/decisions/cyborg-cage2-source-ledger-guardrails.md index e2d2309..f28ec9f 100644 --- a/docs/decisions/cyborg-cage2-source-ledger-guardrails.md +++ b/docs/decisions/cyborg-cage2-source-ledger-guardrails.md @@ -12,7 +12,7 @@ The deliverable is backend-local evidence under `raes_adapters.cyborg`: - `qualification.json` remains the only owner of the selected profile id, repository, commit and tree identities, selected-file digests, legal disposition, stochastic-source inventory, known defects, runtime evidence, - and admissibility decision. + maintainer admission, and attainable claim strength. - `cage2-source-ledger.jsonl` accounts for facts consumed from that selection and binds mapped facts only to published RAES SDL, contract, or evidence surfaces. @@ -206,7 +206,8 @@ contract catalog. Any affected mapped fact must retain a source row and an appropriately tiered loss. - Do not let a mapped row, a complete ledger, a green validator, or a source - digest imply an installable extra, implemented backend capability, + digest erase a recorded qualification limitation or imply an installable + extra, implemented backend capability, deterministic replay, or outcome equivalence. Backend admission comes from maintainer selection; each stronger claim still requires its own evidence. - Do not put semantic values into free-form metadata, `mapping_rule`, diff --git a/docs/index.md b/docs/index.md index efb2f9e..8e8c657 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,10 +3,13 @@ This repository contains a single distribution, `raes-adapters`, that connects concrete simulator backends to published RAES contracts. It ships shared base plumbing plus backend modules whose dependencies can live behind separate -extras in one lock once qualified. The reserved `cyborg` extra remains empty -because the selected upstream source needs an unpublished packaging fix. -The maintainer-admitted backend is supported through its documented source -installation without making the native simulator a base dependency. +extras in one lock. Maintainer-selected backends are admitted; backend-local +qualification evidence records attainable claim strength and limitations. The +`cyborg` and `cyberbattlesim` extras are dependency-light because their selected +native sources have no governed publishable artifact, so users acquire those +simulators separately and the base install remains independent. The admitted +CybORG backend is supported through its documented source installation despite +the empty automatic-installation extra. The shared `raes_adapters.base` module provides plumbing only. RAES remains the semantic and protocol authority. @@ -21,6 +24,7 @@ semantic and protocol authority. - [CybORG/CAGE-2 provisioner and backend-manifest guardrails](decisions/cyborg-cage2-provisioner-manifest-guardrails.md) - [CyberBattleSim qualification guardrails](decisions/cyberbattlesim-qualification-guardrails.md) - [CyberBattleSim scenario and source-ledger guardrails](decisions/cyberbattlesim-scenario-ledger-guardrails.md) +- [CyberBattleSim backend architecture guardrails](decisions/cyberbattlesim-backend-guardrails.md) - [Project services](maintainers/project-services.md) ## Verify a checkout diff --git a/mkdocs.yml b/mkdocs.yml index 9ae7073..03e07e9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -42,6 +42,7 @@ nav: - CybORG/CAGE-2 provisioner and manifest guardrails: decisions/cyborg-cage2-provisioner-manifest-guardrails.md - CyberBattleSim qualification guardrails: decisions/cyberbattlesim-qualification-guardrails.md - CyberBattleSim scenario and source-ledger guardrails: decisions/cyberbattlesim-scenario-ledger-guardrails.md + - CyberBattleSim backend architecture guardrails: decisions/cyberbattlesim-backend-guardrails.md - ADR 003 — Single distribution and Trusted Publishing: decisions/adrs/adr-003-single-distribution-and-trusted-publishing.md - ADR 002 — RAES authority and adapter boundaries: decisions/adrs/adr-002-raes-authority-and-adapter-boundaries.md - ADR 000 — Use ADRs (superseded): decisions/adrs/adr-000-use-adrs.md diff --git a/pyproject.toml b/pyproject.toml index f7d8e60..fd86913 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -36,10 +36,16 @@ classifiers = [ dependencies = ["raes==2.0.0"] [project.optional-dependencies] +# Microsoft publishes the selected CyberBattleSim source but no index or +# release artifact. The adapter extra is therefore dependency-light; users +# install the pinned simulator source separately and target construction +# reports its absence without weakening the base installation. +cyberbattlesim = [] # Reserved CybORG backend key. The selected backend is supported through the # documented source-install route. Issue #12 found the official wheel needs a # packaging-data patch that is not yet published, so the extra remains empty -# instead of hiding that installation limitation behind a runtime clone. +# instead of hiding that installation limitation behind a runtime clone or +# advertising an editable checkout, install-time clone, or ungoverned wheel. cyborg = [] [project.urls] @@ -89,6 +95,10 @@ select = ["E", "F", "I", "UP", "B", "SIM", "C4"] [tool.ruff.lint.per-file-ignores] "tests/**" = ["S101"] +"src/raes_adapters/cyberbattlesim/backend/driver.py" = ["UP004"] +"src/raes_adapters/cyberbattlesim/backend/evaluator.py" = ["UP004"] +"src/raes_adapters/cyberbattlesim/backend/orchestrator.py" = ["UP004"] +"src/raes_adapters/cyberbattlesim/backend/provisioner.py" = ["UP004"] [tool.mypy] python_version = "3.12" diff --git a/src/raes_adapters/__init__.py b/src/raes_adapters/__init__.py index eb99cf2..0600e18 100644 --- a/src/raes_adapters/__init__.py +++ b/src/raes_adapters/__init__.py @@ -3,10 +3,11 @@ A single distribution, ``raes-adapters``, that ships shared adapter plumbing (:mod:`raes_adapters.base`), one importable module per simulator backend (:mod:`raes_adapters.cyborg`, ...), and backend-local qualification evidence. -Implemented simulator dependencies are optional extras, so -``pip install raes-adapters`` gives the base plumbing and -``pip install raes-adapters[cyborg]`` adds the CybORG backend. A failed -qualification does not advertise a broken extra. +Simulator install surfaces are optional extras, so ``pip install +raes-adapters`` gives the base plumbing and ``pip install +raes-adapters[cyborg]`` selects the CybORG adapter surface. Maintainer selection +admits a simulator; qualification limitations bound the claims made for that +surface and may require native dependencies to be installed separately. Packaging boundary: RAES owns the *contracts* an adapter must honor; how this repository structures its packages, locks, and releases is a local decision diff --git a/src/raes_adapters/cyberbattlesim/backend/__init__.py b/src/raes_adapters/cyberbattlesim/backend/__init__.py new file mode 100644 index 0000000..9cb9a3e --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/__init__.py @@ -0,0 +1,48 @@ +"""RAES backend for the admitted CyberBattleSim chain profile.""" + +from __future__ import annotations + +from .cleanup import execute_cyberbattlesim_cleanup +from .driver import ( + CyberBattleSimDriver, + CyberBattleSimDriverProtocol, + DriverCleanupReport, + DriverEvaluation, + DriverResetReport, + DriverStep, +) +from .evaluator import EVALUATION_EVIDENCE_REF, CyberBattleSimEvaluator +from .manifest import ( + CYBERBATTLESIM_BACKEND_NAME, + create_cyberbattlesim_manifest, +) +from .orchestrator import CyberBattleSimOrchestrator +from .participant_runtime import ( + ACTION_EVIDENCE_REF, + CyberBattleSimParticipantRuntime, +) +from .provisioner import CyberBattleSimProvisioner +from .target import ( + create_cyberbattlesim_components, + create_cyberbattlesim_target, +) + +__all__ = [ + "ACTION_EVIDENCE_REF", + "CYBERBATTLESIM_BACKEND_NAME", + "CyberBattleSimDriver", + "CyberBattleSimDriverProtocol", + "CyberBattleSimEvaluator", + "CyberBattleSimOrchestrator", + "CyberBattleSimParticipantRuntime", + "CyberBattleSimProvisioner", + "DriverCleanupReport", + "DriverEvaluation", + "DriverResetReport", + "DriverStep", + "EVALUATION_EVIDENCE_REF", + "create_cyberbattlesim_components", + "create_cyberbattlesim_manifest", + "create_cyberbattlesim_target", + "execute_cyberbattlesim_cleanup", +] diff --git a/src/raes_adapters/cyberbattlesim/backend/cleanup.py b/src/raes_adapters/cyberbattlesim/backend/cleanup.py new file mode 100644 index 0000000..dbe03bd --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/cleanup.py @@ -0,0 +1,131 @@ +"""Verified cleanup execution for the in-process CyberBattleSim driver.""" + +from __future__ import annotations + +from functools import partial + +from raes_backend_protocols.backend_manifest import ( # type: ignore[import-untyped] + BackendManifest, +) +from raes_contracts.contracts import ( # type: ignore[import-untyped] + CleanStateClaimModel, + CleanupObligationModel, + CleanupObligationResultModel, + TrialCleanupPlanModel, + TrialCleanupReceiptModel, +) +from raes_contracts.contracts.trial_cleanup import ( # type: ignore[import-untyped] + TrialOutcome, +) + +from raes_adapters.base import execute_cleanup + +from .driver import CyberBattleSimDriverProtocol + + +def _evidence_ref(obligation_id: str, disposition: str) -> str: + """Build one portable cleanup evidence reference.""" + + return f"evidence.cyberbattlesim.cleanup.{obligation_id}.{disposition}" + + +def _result( + obligation: CleanupObligationModel, + status: str, + disposition: str | None = None, +) -> CleanupObligationResultModel: + """Build a bounded cleanup result without native details.""" + + evidence_refs = ( + [_evidence_ref(obligation.obligation_id, disposition)] if disposition is not None else [] + ) + return CleanupObligationResultModel( + obligation_id=obligation.obligation_id, + status=status, + evidence_refs=evidence_refs, + ) + + +def _destroy( + driver: CyberBattleSimDriverProtocol, + obligation: CleanupObligationModel, +) -> CleanupObligationResultModel: + """Close and verify the selected driver resource.""" + + report = driver.close() + status = "succeeded" if report.closed and report.verified else "unverified" + disposition = "closed" if status == "succeeded" else "unverified" + return _result(obligation, status, disposition) + + +def _verify( + driver: CyberBattleSimDriverProtocol, + obligation: CleanupObligationModel, +) -> CleanupObligationResultModel: + """Probe the selected driver clean-state boundary.""" + + verified = driver.verify_closed() + return _result( + obligation, + "succeeded" if verified else "failed", + "verified" if verified else "not-closed", + ) + + +def _reset( + driver: CyberBattleSimDriverProtocol, + obligation: CleanupObligationModel, +) -> CleanupObligationResultModel: + """Reset the selected driver without claiming deterministic replay.""" + + driver.reset(None) + return _result(obligation, "succeeded", "reset") + + +def _perform( + driver: CyberBattleSimDriverProtocol, + obligation: CleanupObligationModel, +) -> CleanupObligationResultModel: + """Dispatch one admitted cleanup obligation.""" + + handlers = { + "destroy": _destroy, + "verify": _verify, + "reset": _reset, + } + handler = handlers.get(obligation.action_kind) + return _result(obligation, "unsupported") if handler is None else handler(driver, obligation) + + +def _failed(obligation: CleanupObligationModel) -> CleanupObligationResultModel: + """Project any native cleanup exception as a generic failure.""" + + return _result(obligation, "failed", "failed") + + +def execute_cyberbattlesim_cleanup( + plan: TrialCleanupPlanModel, + manifest: BackendManifest, + driver: CyberBattleSimDriverProtocol, + *, + receipt_id: str, + execution_attempt_id: str, + trial_outcome: TrialOutcome, + clean_state_claim: CleanStateClaimModel | None = None, +) -> TrialCleanupReceiptModel: + """Execute one admitted cleanup plan without exposing native failures.""" + + operations = dict.fromkeys(plan.cleanup_obligations, partial(_perform, driver)) + return execute_cleanup( + plan, + manifest, + operations, + failure_result=_failed, + receipt_id=receipt_id, + execution_attempt_id=execution_attempt_id, + trial_outcome=trial_outcome, + clean_state_claim=clean_state_claim, + ) + + +__all__ = ["execute_cyberbattlesim_cleanup"] diff --git a/src/raes_adapters/cyberbattlesim/backend/driver.py b/src/raes_adapters/cyberbattlesim/backend/driver.py new file mode 100644 index 0000000..9a4c162 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/driver.py @@ -0,0 +1,942 @@ +"""Private CyberBattleSim lifecycle and source-transition boundary.""" + +from __future__ import annotations + +import hashlib +import importlib +import json +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from importlib.metadata import Distribution, PackageNotFoundError, distribution +from importlib.util import find_spec +from pathlib import Path, PurePosixPath +from threading import RLock +from typing import Protocol, TypedDict, cast + +from raes_adapters.cyberbattlesim import load_qualification + +_SUPPORTED_ACTION_KINDS = frozenset( + { + "connect", + "local-vulnerability", + "remote-vulnerability", + } +) +_NATIVE_ACTION_KIND = { + "connect": "connect", + "local-vulnerability": "local_vulnerability", + "remote-vulnerability": "remote_vulnerability", +} +_RUNTIME_SOURCE_PATHS = ( + "cyberbattle/__init__.py", + "cyberbattle/_env/cyberbattle_env.py", + "cyberbattle/_env/defender.py", + "cyberbattle/_env/cyberbattle_chain.py", + "cyberbattle/samples/chainpattern/chainpattern.py", +) +_QUALIFICATION_INVALID = "selected simulator qualification is invalid" +_SOURCE_IDENTITY_INVALID = "selected simulator source identity could not be verified" +_DEPENDENCY_IDENTITY_INVALID = "selected simulator dependency identity could not be verified" + + +class _ActionSpace(Protocol): + """Native action-space methods used by the driver.""" + + def seed(self, seed: int) -> None: ... + + +class _NativeEnvironment(Protocol): + """Selected native environment surface kept behind the driver.""" + + action_space: _ActionSpace + + def reset(self, *, seed: int | None) -> object: ... + + def step(self, action: dict[str, object]) -> object: ... + + def close(self) -> None: ... + + +class _EnvironmentWrapper(Protocol): + """Gymnasium wrapper surface used to obtain the native environment.""" + + unwrapped: _NativeEnvironment + + +class _GymnasiumModule(Protocol): + """Gymnasium module surface used by the selected construction path.""" + + def make(self, gym_id: str, **kwargs: object) -> _EnvironmentWrapper: ... + + +class _NumpyModule(Protocol): + """NumPy operations needed for private action-mask selection.""" + + int32: object + + def argwhere(self, mask: object) -> Sequence[object]: ... + + def asarray(self, value: object, *, dtype: object) -> object: ... + + +class _SourceEnvironmentModule(Protocol): + """Selected source environment constructors used by the driver.""" + + AttackerGoal: Callable[..., object] + DefenderConstraint: Callable[..., object] + + +class _DefenderModule(Protocol): + """Selected source defender constructor used by the driver.""" + + ScanAndReimageCompromisedMachines: Callable[..., object] + + +class _SelectedSource(TypedDict): + """Normalized selected source identity.""" + + package: str + version: str + + +class _ScenarioSelection(TypedDict): + """Normalized scenario construction values.""" + + gym_id: str + size: int + + +class _TerminationSelection(TypedDict): + """Normalized termination and evaluator-cutoff values.""" + + attacker_own_atleast: int + attacker_own_atleast_percent: int | float + defender_maintain_sla: int | float + evaluator_cutoff_steps: int + + +class _DefenderSelection(TypedDict): + """Normalized defender construction values.""" + + probability: int | float + scan_capacity: int + scan_frequency: int + + +class _Selection(TypedDict): + """Normalized selected native protocol configuration.""" + + scenario: _ScenarioSelection + termination: _TerminationSelection + defender: _DefenderSelection + + +@dataclass(frozen=True) +class DriverResetReport(object): + """Sanitized reset and stochastic-control dispositions.""" + + operation_ref: str + applied_streams: tuple[str, ...] + unbound_streams: tuple[str, ...] + + +@dataclass(frozen=True) +class DriverStep(object): + """Sanitized facts from at most one source transition.""" + + operation_ref: str + step_number: int + source_transition: bool + processed: bool + terminated: bool + truncated: bool + terminal_cause: str | None + portable_target_refs: tuple[str, ...] = () + + +@dataclass(frozen=True) +class DriverEvaluation(object): + """Evaluator-only source facts; never a participant result.""" + + step_count: int + cumulative_reward: float + terminated: bool + truncated: bool + terminal_cause: str | None + execution_ref: str = "driver.reset.injected" + projection_ref: str = "driver.reset.injected.evaluation.1" + + +@dataclass(frozen=True) +class DriverCleanupReport(object): + """Sanitized close and verification facts.""" + + operation_ref: str + closed: bool + verified: bool + already_closed: bool + + +class CyberBattleSimDriverProtocol(Protocol): + """Injectable boundary implemented by the live and test drivers.""" + + def construct(self) -> None: ... + + def reset(self, seed: int | None) -> DriverResetReport: ... + + def step(self, action_kind: str) -> DriverStep: ... + + def evaluate(self) -> DriverEvaluation: ... + + def close(self) -> DriverCleanupReport: ... + + def verify_closed(self) -> bool: ... + + +class CyberBattleSimDriver(object): + """Lazy in-process driver for the selected public chain profile. + + Native objects never leave this class. The portable action selects one + semantic action kind; the driver resolves the first currently available + native coordinate from the private source mask. This keeps discovery-order + indices and credential-cache positions out of RAES artifacts. + """ + + def __init__(self) -> None: + self._max_steps: int | None = None + self._lock = RLock() + self._environment: _NativeEnvironment | None = None + self._numpy: _NumpyModule | None = None + self._last_observation: object | None = None + self._step_count = 0 + self._operation_count = 0 + self._evaluation_count = 0 + self._execution_ref: str | None = None + self._cumulative_reward = 0.0 + self._terminated = False + self._truncated = False + self._terminal_cause: str | None = None + self._closed = True + self._artifacts_verified = False + + def construct(self) -> None: + """Construct the selected source environment once, on demand.""" + + with self._lock: + if self._environment is not None and not self._closed: + return + qualification, source, selection = self._selected_configuration() + try: + selected_distribution = distribution(source["package"]) + except PackageNotFoundError as exc: + raise RuntimeError("selected simulator source is not installed") from exc + if selected_distribution.version != source["version"]: + raise RuntimeError("installed simulator version does not match the selected source") + + if not self._artifacts_verified: + distributions = self._verify_runtime_artifacts( + qualification, + selected_distribution, + ) + self._verify_selected_source(qualification, selected_distribution) + self._verify_package_origin( + "cyberbattle", + selected_distribution, + "cyberbattle/__init__.py", + ) + self._verify_package_origin( + "gymnasium", + distributions["gymnasium"], + "gymnasium/__init__.py", + ) + self._verify_package_origin( + "numpy", + distributions["numpy"], + "numpy/__init__.py", + ) + # Importing ``cyberbattle`` registers CyberBattleChain-v0. + importlib.import_module("cyberbattle") + gymnasium = cast(_GymnasiumModule, importlib.import_module("gymnasium")) + numpy = cast(_NumpyModule, importlib.import_module("numpy")) + self._verify_package_origin( + "cyberbattle._env.cyberbattle_env", + selected_distribution, + "cyberbattle/_env/cyberbattle_env.py", + ) + self._verify_package_origin( + "cyberbattle._env.defender", + selected_distribution, + "cyberbattle/_env/defender.py", + ) + self._artifacts_verified = True + source_environment = cast( + _SourceEnvironmentModule, + importlib.import_module("cyberbattle._env.cyberbattle_env"), + ) + defender = cast( + _DefenderModule, + importlib.import_module("cyberbattle._env.defender"), + ) + termination = selection["termination"] + defender_selection = selection["defender"] + attacker_goal = source_environment.AttackerGoal( + own_atleast=termination["attacker_own_atleast"], + own_atleast_percent=termination["attacker_own_atleast_percent"], + ) + defender_constraint = source_environment.DefenderConstraint( + maintain_sla=termination["defender_maintain_sla"] + ) + defender_agent = defender.ScanAndReimageCompromisedMachines( + probability=defender_selection["probability"], + scan_capacity=defender_selection["scan_capacity"], + scan_frequency=defender_selection["scan_frequency"], + ) + environment = gymnasium.make( + selection["scenario"]["gym_id"], + size=selection["scenario"]["size"], + attacker_goal=attacker_goal, + defender_constraint=defender_constraint, + defender_agent=defender_agent, + ).unwrapped + self._environment = environment + self._numpy = numpy + self._max_steps = termination["evaluator_cutoff_steps"] + self._closed = False + + def reset(self, seed: int | None) -> DriverResetReport: + """Reset the source and report bounded random-stream coverage.""" + + with self._lock: + self.construct() + environment = self._require_environment() + reset_result = environment.reset(seed=seed) + if not isinstance(reset_result, tuple) or len(reset_result) != 2: + raise RuntimeError("selected simulator reset returned an unsupported shape") + self._last_observation = reset_result[0] + applied: tuple[str, ...] = () + unbound: tuple[str, ...] = ( + "gym-environment", + "gym-action-space", + "python-random", + "numpy-global", + ) + if seed is not None: + environment.action_space.seed(seed) + applied = ("gym-environment", "gym-action-space") + unbound = ("python-random", "numpy-global") + operation_ref = self._next_operation_ref("reset") + self._step_count = 0 + self._evaluation_count = 0 + self._execution_ref = operation_ref + self._cumulative_reward = 0.0 + self._terminated = False + self._truncated = False + self._terminal_cause = None + return DriverResetReport( + operation_ref=operation_ref, + applied_streams=applied, + unbound_streams=unbound, + ) + + def step(self, action_kind: str) -> DriverStep: + """Resolve one private native action and perform at most one source step.""" + + with self._lock: + if action_kind not in _SUPPORTED_ACTION_KINDS: + raise ValueError("unsupported CyberBattleSim action kind") + environment = self._require_environment() + if self._last_observation is None: + raise RuntimeError("selected simulator must be reset before action execution") + operation_ref = self._next_operation_ref("step") + native_action = self._resolve_native_action(action_kind) + if native_action is None: + return DriverStep( + operation_ref=operation_ref, + step_number=self._step_count, + source_transition=False, + processed=False, + terminated=False, + truncated=False, + terminal_cause=None, + ) + + step_result = environment.step(native_action) + if not isinstance(step_result, tuple) or len(step_result) != 5: + raise RuntimeError("selected simulator step returned an unsupported shape") + observation, reward, terminated, truncated, _source_info = step_result + self._last_observation = observation + self._step_count += 1 + self._cumulative_reward += float(reward) + self._terminated = bool(terminated) + self._truncated = bool(truncated) + if self._terminated: + self._terminal_cause = "source-terminated" + elif self._truncated: + self._terminal_cause = "source-truncated" + elif self._step_count >= self._require_max_steps(): + self._terminal_cause = "evaluator-cutoff" + else: + self._terminal_cause = None + return DriverStep( + operation_ref=operation_ref, + step_number=self._step_count, + source_transition=True, + processed=True, + terminated=self._terminated, + truncated=self._truncated, + terminal_cause=self._terminal_cause, + ) + + def evaluate(self) -> DriverEvaluation: + """Return evaluator-only facts without advancing the simulator.""" + + with self._lock: + self._require_environment() + execution_ref = self._execution_ref + if execution_ref is None: + raise RuntimeError("selected simulator must be reset before evaluation") + self._evaluation_count += 1 + return DriverEvaluation( + execution_ref=execution_ref, + projection_ref=f"{execution_ref}.evaluation.{self._evaluation_count}", + step_count=self._step_count, + cumulative_reward=self._cumulative_reward, + terminated=self._terminated, + truncated=self._truncated, + terminal_cause=self._terminal_cause, + ) + + def close(self) -> DriverCleanupReport: + """Close the source environment idempotently.""" + + with self._lock: + already_closed = self._closed + if self._environment is not None and not self._closed: + self._environment.close() + self._environment = None + self._last_observation = None + self._execution_ref = None + self._closed = True + return DriverCleanupReport( + operation_ref=self._next_operation_ref("close"), + closed=True, + verified=self.verify_closed(), + already_closed=already_closed, + ) + + def verify_closed(self) -> bool: + """Verify the bounded in-process ownership state.""" + + with self._lock: + return self._closed and self._environment is None + + def _resolve_native_action(self, action_kind: str) -> dict[str, object] | None: + if not isinstance(self._last_observation, Mapping): + raise RuntimeError("selected simulator observation has an unsupported shape") + masks = self._last_observation.get("action_mask") + if not isinstance(masks, Mapping): + raise RuntimeError("selected simulator action availability is unavailable") + native_kind = _NATIVE_ACTION_KIND[action_kind] + native_mask = masks.get(native_kind) + numpy = self._numpy + if numpy is None: + raise RuntimeError("selected simulator numeric runtime is unavailable") + coordinates = numpy.argwhere(native_mask) + if len(coordinates) == 0: + return None + selected = numpy.asarray(coordinates[0], dtype=numpy.int32) + return {native_kind: selected} + + def _require_environment(self) -> _NativeEnvironment: + if self._environment is None or self._closed: + raise RuntimeError("selected simulator environment is not constructed") + return self._environment + + def _require_max_steps(self) -> int: + if self._max_steps is None: + raise RuntimeError("selected simulator cutoff is unavailable") + return self._max_steps + + @staticmethod + def _selected_configuration() -> tuple[dict[str, object], _SelectedSource, _Selection]: + """Load and normalize the maintainer-selected native configuration.""" + + qualification = cast(dict[str, object], load_qualification()) + source = qualification.get("source") + protocol = qualification.get("protocol") + if not isinstance(source, dict) or not isinstance(protocol, dict): + raise RuntimeError(_QUALIFICATION_INVALID) + package = source.get("package") + version = source.get("version") + selection = protocol.get("selection") + if not isinstance(package, str) or not isinstance(version, str): + raise RuntimeError(_QUALIFICATION_INVALID) + if not isinstance(selection, dict): + raise RuntimeError(_QUALIFICATION_INVALID) + return ( + qualification, + {"package": package, "version": version}, + CyberBattleSimDriver._normalize_selection(selection), + ) + + @staticmethod + def _normalize_selection(selection: dict[object, object]) -> _Selection: + """Validate the three selected construction sections independently.""" + + scenario = selection.get("scenario") + termination = selection.get("termination") + defender = selection.get("defender") + if not all(isinstance(section, dict) for section in (scenario, termination, defender)): + raise RuntimeError(_QUALIFICATION_INVALID) + return { + "scenario": CyberBattleSimDriver._normalize_scenario( + cast(dict[object, object], scenario) + ), + "termination": CyberBattleSimDriver._normalize_termination( + cast(dict[object, object], termination) + ), + "defender": CyberBattleSimDriver._normalize_defender( + cast(dict[object, object], defender) + ), + } + + @staticmethod + def _normalize_scenario(section: dict[object, object]) -> _ScenarioSelection: + """Validate and normalize the selected scenario.""" + + gym_id = section.get("gym_id") + size = section.get("size") + if not isinstance(gym_id, str) or not isinstance(size, int): + raise RuntimeError(_QUALIFICATION_INVALID) + return {"gym_id": gym_id, "size": size} + + @staticmethod + def _normalize_termination(section: dict[object, object]) -> _TerminationSelection: + """Validate and normalize selected termination controls.""" + + own_atleast = section.get("attacker_own_atleast") + own_percent = section.get("attacker_own_atleast_percent") + maintain_sla = section.get("defender_maintain_sla") + cutoff = section.get("evaluator_cutoff_steps") + if not isinstance(own_atleast, int) or not isinstance(own_percent, (int, float)): + raise RuntimeError(_QUALIFICATION_INVALID) + if not isinstance(maintain_sla, (int, float)): + raise RuntimeError(_QUALIFICATION_INVALID) + if not isinstance(cutoff, int) or cutoff <= 0: + raise RuntimeError(_QUALIFICATION_INVALID) + return { + "attacker_own_atleast": own_atleast, + "attacker_own_atleast_percent": own_percent, + "defender_maintain_sla": maintain_sla, + "evaluator_cutoff_steps": cutoff, + } + + @staticmethod + def _normalize_defender(section: dict[object, object]) -> _DefenderSelection: + """Validate and normalize selected defender controls.""" + + probability = section.get("probability") + scan_capacity = section.get("scan_capacity") + scan_frequency = section.get("scan_frequency") + if not isinstance(probability, (int, float)): + raise RuntimeError(_QUALIFICATION_INVALID) + if not isinstance(scan_capacity, int) or not isinstance(scan_frequency, int): + raise RuntimeError(_QUALIFICATION_INVALID) + return { + "probability": probability, + "scan_capacity": scan_capacity, + "scan_frequency": scan_frequency, + } + + @staticmethod + def _verify_selected_source( + qualification: dict[str, object], + selected_distribution: Distribution, + ) -> None: + """Verify selected source records and the complete runtime tree.""" + + CyberBattleSimDriver._verify_runtime_source_tree( + qualification, + selected_distribution, + ) + source_files = qualification.get("source_files") + if not isinstance(source_files, list): + raise RuntimeError(_SOURCE_IDENTITY_INVALID) + expected: dict[str, str] = {} + for entry in source_files: + if not isinstance(entry, dict): + continue + path = entry.get("path") + digest = entry.get("sha256") + if isinstance(path, str) and isinstance(digest, str): + expected[path] = digest + try: + for source_path in _RUNTIME_SOURCE_PATHS: + expected_digest = expected[source_path] + installed_source = cast( + Path, + selected_distribution.locate_file(source_path), + ) + content = installed_source.read_bytes() + if hashlib.sha256(content).hexdigest() != expected_digest: + raise RuntimeError(_SOURCE_IDENTITY_INVALID) + except (KeyError, OSError) as exc: + raise RuntimeError(_SOURCE_IDENTITY_INVALID) from exc + + @staticmethod + def _verify_runtime_source_tree( + qualification: dict[str, object], + selected_distribution: Distribution, + ) -> None: + """Verify every installed file under the selected source import root.""" + + tree = qualification.get("runtime_source_tree") + distribution_files = selected_distribution.files + if not isinstance(tree, dict) or distribution_files is None: + raise RuntimeError(_SOURCE_IDENTITY_INVALID) + expected_digest = tree.get("sha256") + expected_count = tree.get("file_count") + if not isinstance(expected_digest, str) or not isinstance(expected_count, int): + raise RuntimeError(_SOURCE_IDENTITY_INVALID) + paths = CyberBattleSimDriver._runtime_root_paths( + selected_distribution, + "cyberbattle", + _SOURCE_IDENTITY_INVALID, + ) + if len(paths) != expected_count: + raise RuntimeError(_SOURCE_IDENTITY_INVALID) + package_root = cast( + Path, + selected_distribution.locate_file("cyberbattle"), + ).resolve() + observed_digest = CyberBattleSimDriver._installed_tree_digest( + selected_distribution, + paths, + _SOURCE_IDENTITY_INVALID, + package_root=package_root, + ) + if observed_digest != expected_digest: + raise RuntimeError(_SOURCE_IDENTITY_INVALID) + + @staticmethod + def _installed_tree_digest( + selected_distribution: Distribution, + paths: Sequence[str], + identity_error: str, + *, + package_root: Path | None = None, + ) -> str: + """Hash sorted installed paths and contents into a portable tree identity.""" + + digest = hashlib.sha256() + try: + for artifact_path in paths: + portable_path = PurePosixPath(artifact_path) + if portable_path.is_absolute() or ".." in portable_path.parts: + raise RuntimeError(identity_error) + installed_path = cast( + Path, + selected_distribution.locate_file(artifact_path), + ).resolve() + if package_root is not None and not installed_path.is_relative_to(package_root): + raise RuntimeError(identity_error) + content_digest = hashlib.sha256(installed_path.read_bytes()).hexdigest() + digest.update(artifact_path.encode("utf-8")) + digest.update(b"\0") + digest.update(content_digest.encode("ascii")) + digest.update(b"\n") + except OSError as exc: + raise RuntimeError(identity_error) from exc + return digest.hexdigest() + + @staticmethod + def _verify_runtime_artifacts( + qualification: dict[str, object], + selected_distribution: Distribution, + ) -> dict[str, Distribution]: + """Verify the complete selected runtime artifact set.""" + + selected_versions, records = CyberBattleSimDriver._artifact_maps(qualification) + verified: dict[str, Distribution] = {"cyberbattlesim": selected_distribution} + for package_name, record in records.items(): + dependency = CyberBattleSimDriver._verify_runtime_artifact( + package_name, + record, + selected_versions, + verified, + ) + verified[package_name] = dependency + return verified + + @staticmethod + def _artifact_maps( + qualification: dict[str, object], + ) -> tuple[dict[str, str], dict[str, dict[object, object]]]: + """Index selected dependency versions and runtime artifact records.""" + + artifacts = qualification.get("runtime_artifacts") + dependencies = qualification.get("dependencies") + if not isinstance(artifacts, list) or not isinstance(dependencies, list): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + selected_versions = { + str(entry["name"]).casefold(): entry["version"] + for entry in dependencies + if isinstance(entry, dict) + and isinstance(entry.get("name"), str) + and isinstance(entry.get("version"), str) + } + records = { + str(entry["name"]).casefold(): entry + for entry in artifacts + if isinstance(entry, dict) and isinstance(entry.get("name"), str) + } + if set(records) != {"cyberbattlesim", "gymnasium", "numpy"}: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + return selected_versions, records + + @staticmethod + def _verify_runtime_artifact( + package_name: str, + record: dict[object, object], + selected_versions: dict[str, str], + verified: dict[str, Distribution], + ) -> Distribution: + """Verify one selected artifact record and installed distribution.""" + + expected_version = CyberBattleSimDriver._validate_artifact_identity( + package_name, + record, + selected_versions, + ) + dependency = CyberBattleSimDriver._resolve_distribution(package_name, verified) + if dependency.version != expected_version: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + CyberBattleSimDriver._verify_direct_installation(record, dependency) + CyberBattleSimDriver._verify_artifact_roots(record, dependency) + return dependency + + @staticmethod + def _validate_artifact_identity( + package_name: str, + record: dict[object, object], + selected_versions: dict[str, str], + ) -> str: + """Validate one artifact's declared version and identity policy.""" + + expected_version = record.get("version") + artifact = record.get("artifact") + if not isinstance(expected_version, str) or not isinstance(artifact, dict): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + valid = ( + isinstance(artifact.get("filename"), str) + and CyberBattleSimDriver._is_sha256(artifact.get("sha256")) + and isinstance(artifact.get("require_direct_archive_sha256"), bool) + and artifact.get("runtime_identity") == "complete-root-tree" + and selected_versions.get(package_name) == expected_version + ) + if not valid: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + return expected_version + + @staticmethod + def _resolve_distribution( + package_name: str, + verified: dict[str, Distribution], + ) -> Distribution: + """Resolve an installed distribution without leaking package metadata.""" + + dependency = verified.get(package_name) + if dependency is not None: + return dependency + try: + return distribution(package_name) + except PackageNotFoundError as exc: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) from exc + + @staticmethod + def _verify_direct_installation( + record: dict[object, object], + selected_distribution: Distribution, + ) -> None: + """Reject unverifiable editable/VCS installs and check archive identity.""" + + direct_url_text = CyberBattleSimDriver._direct_url_text(selected_distribution) + if direct_url_text is None: + return + expected_digest, require_archive_digest = CyberBattleSimDriver._direct_artifact_policy( + record + ) + direct_url = CyberBattleSimDriver._load_direct_url(direct_url_text) + if not {"dir_info", "vcs_info"}.isdisjoint(direct_url): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + observed_digest = CyberBattleSimDriver._archive_sha256(direct_url) + if require_archive_digest and observed_digest != expected_digest: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + + @staticmethod + def _direct_url_text(selected_distribution: Distribution) -> str | None: + """Read optional direct-install provenance from a distribution.""" + + read_text = getattr(selected_distribution, "read_text", None) + if not callable(read_text): + return None + direct_url_text = read_text("direct_url.json") + if direct_url_text is not None and not isinstance(direct_url_text, str): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + return direct_url_text + + @staticmethod + def _direct_artifact_policy(record: dict[object, object]) -> tuple[str, bool]: + """Read the selected archive digest policy from an artifact record.""" + + artifact = record.get("artifact") + if not isinstance(artifact, dict): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + expected_digest = artifact.get("sha256") + require_archive_digest = artifact.get("require_direct_archive_sha256") + if not isinstance(expected_digest, str): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + if not isinstance(require_archive_digest, bool): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + return expected_digest, require_archive_digest + + @staticmethod + def _load_direct_url(direct_url_text: str) -> dict[object, object]: + """Decode a direct-install provenance record as a JSON object.""" + + try: + direct_url = json.loads(direct_url_text) + except (TypeError, json.JSONDecodeError) as exc: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) from exc + if not isinstance(direct_url, dict): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + return direct_url + + @staticmethod + def _archive_sha256(direct_url: dict[object, object]) -> object: + """Read modern or legacy SHA-256 metadata from a direct archive.""" + + archive_info = direct_url.get("archive_info") + hashes = archive_info.get("hashes") if isinstance(archive_info, dict) else None + legacy_hash = archive_info.get("hash") if isinstance(archive_info, dict) else None + observed_digest = hashes.get("sha256") if isinstance(hashes, dict) else None + if observed_digest is None and isinstance(legacy_hash, str): + algorithm, separator, digest = legacy_hash.partition("=") + if algorithm == "sha256" and separator: + observed_digest = digest + return observed_digest + + @staticmethod + def _verify_artifact_roots( + record: dict[object, object], + selected_distribution: Distribution, + ) -> None: + """Verify every declared import root for one runtime artifact.""" + + roots = record.get("roots") + if not isinstance(roots, list) or not roots: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + for root in roots: + if not isinstance(root, dict): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + CyberBattleSimDriver._verify_artifact_root(root, selected_distribution) + + @staticmethod + def _verify_artifact_root( + root: dict[object, object], + selected_distribution: Distribution, + ) -> None: + """Verify one complete installed import-root tree.""" + + root_path = root.get("path") + expected_count = root.get("file_count") + expected_digest = root.get("sha256") + if not isinstance(root_path, str) or not isinstance(expected_count, int): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + if not CyberBattleSimDriver._is_sha256(expected_digest): + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + paths = CyberBattleSimDriver._runtime_root_paths( + selected_distribution, + root_path, + _DEPENDENCY_IDENTITY_INVALID, + ) + if len(paths) != expected_count: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + observed_digest = CyberBattleSimDriver._installed_tree_digest( + selected_distribution, + paths, + _DEPENDENCY_IDENTITY_INVALID, + ) + if observed_digest != expected_digest: + raise RuntimeError(_DEPENDENCY_IDENTITY_INVALID) + + @staticmethod + def _runtime_root_paths( + selected_distribution: Distribution, + root_path: str, + identity_error: str = _DEPENDENCY_IDENTITY_INVALID, + ) -> list[str]: + """Enumerate a real, symlink-free installed import-root tree.""" + + portable_root = PurePosixPath(root_path) + if portable_root.is_absolute() or ".." in portable_root.parts: + raise RuntimeError(identity_error) + installed_root = cast( + Path, + selected_distribution.locate_file(root_path), + ) + if installed_root.is_symlink() or not installed_root.is_dir(): + raise RuntimeError(identity_error) + paths: list[str] = [] + try: + for installed_path in installed_root.rglob("*"): + if installed_path.is_symlink(): + raise RuntimeError(identity_error) + if installed_path.is_dir(): + continue + if not installed_path.is_file(): + raise RuntimeError(identity_error) + relative_path = installed_path.relative_to(installed_root) + paths.append(str(portable_root / PurePosixPath(relative_path.as_posix()))) + except OSError as exc: + raise RuntimeError(identity_error) from exc + return sorted(paths) + + @staticmethod + def _is_sha256(value: object) -> bool: + """Return whether a value is a lowercase SHA-256 digest.""" + + return ( + isinstance(value, str) + and len(value) == 64 + and all(character in "0123456789abcdef" for character in value) + ) + + @staticmethod + def _verify_package_origin( + module_name: str, + selected_distribution: Distribution, + expected_relative_path: str, + ) -> None: + module_spec = find_spec(module_name) + origin = module_spec.origin if module_spec is not None else None + expected = cast( + Path, + selected_distribution.locate_file(expected_relative_path), + ).resolve() + if not isinstance(origin, str) or Path(origin).resolve() != expected: + raise RuntimeError("selected simulator module origin could not be verified") + + def _next_operation_ref(self, operation: str) -> str: + self._operation_count += 1 + return f"driver.{operation}.{self._operation_count}" + + +__all__ = [ + "CyberBattleSimDriver", + "CyberBattleSimDriverProtocol", + "DriverCleanupReport", + "DriverEvaluation", + "DriverResetReport", + "DriverStep", +] diff --git a/src/raes_adapters/cyberbattlesim/backend/evaluator.py b/src/raes_adapters/cyberbattlesim/backend/evaluator.py new file mode 100644 index 0000000..277fbfe --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/evaluator.py @@ -0,0 +1,808 @@ +"""Evaluator-only projection of sanitized CyberBattleSim run facts.""" + +from __future__ import annotations + +import hashlib +from datetime import UTC, datetime + +from raes_contracts.contracts import ( # type: ignore[import-untyped] + EvaluationHistoryEventModel, + EvaluationResultStateModel, + ExperimentCaptureRequirementModel, + ExperimentCaptureSpecModel, + ExperimentCaptureSpecReferenceModel, + ExperimentCaptureWindowModel, + ExperimentChecksumModel, + ExperimentDerivedMeasureMethodModel, + ExperimentDerivedMeasureModel, + ExperimentEvidenceRecordModel, + ExperimentEvidenceRecordReferenceModel, + ExperimentMeasurementChannelReferenceModel, + ExperimentReferenceModel, + ExperimentTaskReferenceModel, + ExperimentValidityNoteModel, + PropositionLossDisclosureModel, + PropositionTruthResultModel, +) +from raes_contracts.contracts.experiment_capture import ( # type: ignore[import-untyped] + ExperimentRawEvidenceContentModel, +) +from raes_contracts.diagnostics import Diagnostic # type: ignore[import-untyped] +from raes_contracts.evaluation import ( # type: ignore[import-untyped] + EvaluationHistoryEventType, + EvaluationResultContract, + EvaluationResultStatus, +) +from raes_contracts.planning import ( # type: ignore[import-untyped] + ChangeAction, + EvaluationOp, + EvaluationPlan, + RuntimeDomain, +) +from raes_contracts.runtime_state import ( # type: ignore[import-untyped] + ApplyResult, + RuntimeSnapshot, + SnapshotEntry, +) + +from raes_adapters.cyberbattlesim import load_qualification +from raes_adapters.cyberbattlesim.scenario_ledger import CYBERBATTLE_CHAIN + +from .driver import CyberBattleSimDriverProtocol, DriverEvaluation + +EVALUATION_EVIDENCE_REF = "evidence-record.cyberbattlesim.evaluator-summary" + +_EVIDENCE_RECORD_VERSION = "1.0.0" +_CAPTURE_SPEC_ID = "cyberbattlesim-evaluator-capture" +_CAPTURE_REQUIREMENT_ID = "cyberbattlesim-evaluator-summary" +_CAPTURE_WINDOW_ID = "cyberbattlesim-selected-episode" +_TASK_ID = CYBERBATTLE_CHAIN.task_id +_SUPPORTED_RESOURCE_TYPES = frozenset( + {"condition-binding", "proposition", "assertion", "objective"} +) + + +def _now_iso() -> str: + """Return a portable UTC timestamp.""" + + return datetime.now(UTC).isoformat().replace("+00:00", "Z") + + +def _scoped_id(base: str, projection_ref: str) -> str: + """Scope an artifact identity to one evaluator projection.""" + + if not projection_ref: + raise RuntimeError("CyberBattleSim evaluation projection identity is unavailable") + identity_digest = hashlib.sha256(projection_ref.encode("utf-8")).hexdigest() + return f"{base}.{identity_digest}" + + +class CyberBattleSimEvaluator(object): + """Read evaluator-owned facts without advancing the source environment.""" + + def __init__(self, driver: CyberBattleSimDriverProtocol) -> None: + self._driver = driver + self._running = False + self._startup_order: list[str] = [] + self._results: dict[str, dict[str, object]] = {} + self._history: dict[str, list[dict[str, object]]] = {} + self._capture_spec: ExperimentCaptureSpecModel | None = None + self._evidence_records: tuple[ExperimentEvidenceRecordModel, ...] = () + self._derived_measures: tuple[ExperimentDerivedMeasureModel, ...] = () + + def start( + self, + plan: EvaluationPlan, + snapshot: RuntimeSnapshot, + ) -> ApplyResult: + """Apply an evaluation plan and project evaluator-owned source facts.""" + + terminal_result = self._terminal_result(plan, snapshot) + if terminal_result is not None: + return terminal_result + evaluation, projection_failure = self._evaluation_projection(plan, snapshot) + if projection_failure is not None: + return projection_failure + return self._apply_projection(plan, snapshot, evaluation) + + @staticmethod + def _terminal_result( + plan: EvaluationPlan, + snapshot: RuntimeSnapshot, + ) -> ApplyResult | None: + """Return an unsupported failure or unchanged-plan success when terminal.""" + + unsupported = CyberBattleSimEvaluator._unsupported_result(plan, snapshot) + if unsupported is not None: + return unsupported + if not CyberBattleSimEvaluator._has_mutations(plan): + return ApplyResult(success=True, snapshot=snapshot) + return None + + @staticmethod + def _unsupported_result( + plan: EvaluationPlan, + snapshot: RuntimeSnapshot, + ) -> ApplyResult | None: + """Return a portable failure for the first unsupported resource.""" + + operation = next( + ( + item + for item in plan.operations + if item.resource_type not in _SUPPORTED_RESOURCE_TYPES + ), + None, + ) + if operation is None: + return None + return ApplyResult( + success=False, + snapshot=snapshot, + diagnostics=[ + Diagnostic( + code="cyberbattlesim.evaluation.unsupported-resource", + domain="evaluation", + address=operation.address, + message=( + "The selected CyberBattleSim profile does not support " + "this evaluation resource type." + ), + ) + ], + ) + + @staticmethod + def _has_mutations(plan: EvaluationPlan) -> bool: + """Return whether a plan changes evaluator-owned state.""" + + return any(operation.action != ChangeAction.UNCHANGED for operation in plan.operations) + + @staticmethod + def _needs_projection(plan: EvaluationPlan) -> bool: + """Return whether source facts are required by this plan.""" + + return any( + operation.action in {ChangeAction.CREATE, ChangeAction.UPDATE} + and operation.resource_type in {"condition-binding", "objective"} + for operation in plan.operations + ) + + def _evaluation_projection( + self, + plan: EvaluationPlan, + snapshot: RuntimeSnapshot, + ) -> tuple[DriverEvaluation, ApplyResult | None]: + """Read and record one evaluator projection when the plan requires it.""" + + if not self._needs_projection(plan): + self._clear_evidence() + return self._empty_evaluation(), None + try: + evaluation = self._driver.evaluate() + except Exception: + return self._empty_evaluation(), self._projection_failure(snapshot) + captured_at = _now_iso() + self._capture_spec = self._capture_specification(evaluation, captured_at) + evidence_record, derived_measure = self._experiment_evidence( + evaluation, + captured_at, + ) + self._evidence_records = (evidence_record,) + self._derived_measures = (derived_measure,) + return evaluation, None + + def _clear_evidence(self) -> None: + """Clear evaluator evidence when no source projection is needed.""" + + self._capture_spec = None + self._evidence_records = () + self._derived_measures = () + + @staticmethod + def _empty_evaluation() -> DriverEvaluation: + """Return the non-source placeholder used for metadata-only changes.""" + + return DriverEvaluation( + execution_ref="cyberbattlesim.no-execution", + projection_ref="cyberbattlesim.no-projection", + step_count=0, + cumulative_reward=0.0, + terminated=False, + truncated=False, + terminal_cause=None, + ) + + @staticmethod + def _projection_failure(snapshot: RuntimeSnapshot) -> ApplyResult: + """Return a bounded source-projection failure.""" + + return ApplyResult( + success=False, + snapshot=snapshot, + diagnostics=[ + Diagnostic( + code="cyberbattlesim.evaluation.projection-failed", + domain="evaluation", + address="evaluation.cyberbattlesim.selected-profile", + message=( + "The CyberBattleSim evaluator could not project the selected run facts." + ), + ) + ], + ) + + def _apply_projection( + self, + plan: EvaluationPlan, + snapshot: RuntimeSnapshot, + evaluation: DriverEvaluation, + ) -> ApplyResult: + """Apply normalized evaluator state and truth projections.""" + + needs_projection = self._needs_projection(plan) + evidence_ref = ( + _scoped_id(EVALUATION_EVIDENCE_REF, evaluation.projection_ref) + if needs_projection + else None + ) + entries, results, history, truth_results, changed = self._apply_operations( + plan, + snapshot, + evaluation, + evidence_ref, + ) + self._apply_truth_results(plan, truth_results, evidence_ref, needs_projection) + self._running = bool(plan.resources or plan.operations) + self._startup_order = list(plan.startup_order) + self._results = results + self._history = history + return ApplyResult( + success=True, + snapshot=snapshot.with_entries( + entries, + evaluation_results=results, + evaluation_history=history, + proposition_truth_results=truth_results, + ), + changed_addresses=changed, + ) + + def _apply_operations( + self, + plan: EvaluationPlan, + snapshot: RuntimeSnapshot, + evaluation: DriverEvaluation, + evidence_ref: str | None, + ) -> tuple[ + dict[str, SnapshotEntry], + dict[str, dict[str, object]], + dict[str, list[dict[str, object]]], + dict[str, dict[str, object]], + list[str], + ]: + """Apply create, update, and delete operations to copied state maps.""" + + entries = dict(snapshot.entries) + results = dict(snapshot.evaluation_results) + history = {address: list(events) for address, events in snapshot.evaluation_history.items()} + truth_results = dict(snapshot.proposition_truth_results) + changed: list[str] = [] + now = _now_iso() + for operation in plan.operations: + if operation.action == ChangeAction.UNCHANGED: + continue + if operation.action == ChangeAction.DELETE: + self._delete_operation(operation, entries, results, history, truth_results) + else: + self._upsert_operation( + operation, + entries, + results, + history, + evaluation, + now, + evidence_ref, + ) + changed.append(operation.address) + return entries, results, history, truth_results, changed + + @staticmethod + def _delete_operation( + operation: EvaluationOp, + entries: dict[str, SnapshotEntry], + results: dict[str, dict[str, object]], + history: dict[str, list[dict[str, object]]], + truth_results: dict[str, dict[str, object]], + ) -> None: + """Delete one evaluator resource and all owned result state.""" + + entries.pop(operation.address, None) + results.pop(operation.address, None) + history.pop(operation.address, None) + truth_results.pop(operation.address, None) + + def _upsert_operation( + self, + operation: EvaluationOp, + entries: dict[str, SnapshotEntry], + results: dict[str, dict[str, object]], + history: dict[str, list[dict[str, object]]], + evaluation: DriverEvaluation, + now: str, + evidence_ref: str | None, + ) -> None: + """Create or update one evaluator resource and typed result state.""" + + is_truth_resource = operation.resource_type in {"proposition", "assertion"} + entries[operation.address] = SnapshotEntry( + address=operation.address, + domain=RuntimeDomain.EVALUATION, + resource_type=operation.resource_type, + payload=operation.payload, + ordering_dependencies=operation.ordering_dependencies, + refresh_dependencies=operation.refresh_dependencies, + status="admitted" if is_truth_resource else "evaluating", + ) + if is_truth_resource: + return + result_state = self._result_state(operation, evaluation, now, evidence_ref) + results[operation.address] = result_state.model_dump(mode="json") + history[operation.address] = [ + event.model_dump(mode="json") for event in self._history_events(result_state, now) + ] + + @staticmethod + def _apply_truth_results( + plan: EvaluationPlan, + truth_results: dict[str, dict[str, object]], + evidence_ref: str | None, + needs_projection: bool, + ) -> None: + """Project admitted assertions as bounded unknown truth outcomes.""" + + proposition_bases = { + operation.address: operation.payload.get("evaluation_basis") + for operation in plan.operations + if operation.resource_type == "proposition" + } + for operation in plan.operations: + truth_result = CyberBattleSimEvaluator._truth_result( + operation, + proposition_bases, + evidence_ref, + needs_projection, + ) + if truth_result is not None: + truth_results[operation.address] = truth_result.model_dump(mode="json") + + @staticmethod + def _truth_result( + operation: EvaluationOp, + proposition_bases: dict[str, object], + evidence_ref: str | None, + needs_projection: bool, + ) -> PropositionTruthResultModel | None: + """Build one bounded truth result when an assertion is well formed.""" + + if ( + operation.action in {ChangeAction.DELETE, ChangeAction.UNCHANGED} + or operation.resource_type != "assertion" + ): + return None + proposition_address = operation.payload.get("proposition_address") + polarity = operation.payload.get("polarity") + evaluation_basis = proposition_bases.get(str(proposition_address)) + valid = ( + isinstance(proposition_address, str) + and polarity in {"positive", "negative"} + and evaluation_basis in {"declared_state", "observed_state"} + ) + if not valid: + return None + return PropositionTruthResultModel( + result_id=f"truth.{operation.address}", + proposition_address=proposition_address, + assertion_address=operation.address, + assertion_polarity=polarity, + proposition_outcome="unknown", + assertion_outcome="unknown", + evaluation_basis=evaluation_basis, + indeterminacy_reason=("lossy_evidence" if needs_projection else "missing_evidence"), + evidence_refs=[evidence_ref] if evidence_ref is not None else [], + loss_disclosures=[ + PropositionLossDisclosureModel( + kind="lossy", + within_admissible_bound=True, + ) + ], + ) + + @staticmethod + def _result_state( + operation: EvaluationOp, + evaluation: DriverEvaluation, + now: str, + evidence_ref: str | None, + ) -> EvaluationResultStateModel: + result_contract = operation.payload.get("result_contract", {}) + if not isinstance(result_contract, dict): + raise RuntimeError("compiled evaluation result contract is invalid") + contract = EvaluationResultContract.from_mapping(result_contract) + reports_score = contract.supports_score and not contract.supports_passed + if evidence_ref is None: + raise RuntimeError("CyberBattleSim evaluation evidence identity is unavailable") + return EvaluationResultStateModel( + resource_type=contract.resource_type, + run_id=evaluation.execution_ref, + status=( + EvaluationResultStatus.READY.value + if reports_score + else EvaluationResultStatus.RUNNING.value + ), + observed_at=now, + updated_at=now, + passed=None, + score=evaluation.cumulative_reward if reports_score else None, + max_score=contract.fixed_max_score if reports_score else None, + detail=( + "CyberBattleSim evaluator projection after " + f"{evaluation.step_count} source transitions." + ), + evidence_refs=[evidence_ref], + ) + + @staticmethod + def _history_events( + result_state: EvaluationResultStateModel, + now: str, + ) -> list[EvaluationHistoryEventModel]: + terminal = result_state.status == EvaluationResultStatus.READY.value + return [ + EvaluationHistoryEventModel( + event_type=EvaluationHistoryEventType.EVALUATION_STARTED.value, + timestamp=now, + status=EvaluationResultStatus.RUNNING.value, + passed=None, + score=None, + max_score=None, + detail=None, + evidence_refs=[], + details={}, + ), + EvaluationHistoryEventModel( + event_type=( + EvaluationHistoryEventType.EVALUATION_READY.value + if terminal + else EvaluationHistoryEventType.EVALUATION_UPDATED.value + ), + timestamp=now, + status=result_state.status, + passed=result_state.passed, + score=result_state.score, + max_score=result_state.max_score, + detail=result_state.detail, + evidence_refs=result_state.evidence_refs, + details={}, + ), + ] + + def status(self) -> dict[str, object]: + return { + "running": self._running, + "startup_order": list(self._startup_order), + "results": len(self._results), + "capture_spec": self._capture_spec is not None, + "evidence_records": len(self._evidence_records), + "derived_measures": len(self._derived_measures), + } + + def results(self) -> dict[str, dict[str, object]]: + return {address: dict(result) for address, result in self._results.items()} + + def history(self) -> dict[str, list[dict[str, object]]]: + return { + address: [dict(event) for event in events] for address, events in self._history.items() + } + + def evidence_records(self) -> tuple[ExperimentEvidenceRecordModel, ...]: + """Return typed, redacted evaluator evidence for the selected run.""" + + return self._evidence_records + + def capture_spec(self) -> ExperimentCaptureSpecModel | None: + """Return the typed capture boundary used for evaluator evidence.""" + + return self._capture_spec + + def derived_measures(self) -> tuple[ExperimentDerivedMeasureModel, ...]: + """Return typed measures whose limits prevent replay overclaiming.""" + + return self._derived_measures + + def stop(self, snapshot: RuntimeSnapshot) -> ApplyResult: + entries = { + address: entry + for address, entry in snapshot.entries.items() + if entry.domain != RuntimeDomain.EVALUATION + } + removed = [ + address + for address, entry in snapshot.entries.items() + if entry.domain == RuntimeDomain.EVALUATION + ] + self._running = False + self._startup_order = [] + self._results = {} + self._history = {} + self._capture_spec = None + self._evidence_records = () + self._derived_measures = () + return ApplyResult( + success=True, + snapshot=snapshot.with_entries( + entries, + evaluation_results={}, + evaluation_history={}, + proposition_truth_results={}, + ), + changed_addresses=removed, + ) + + @staticmethod + def _experiment_evidence( + evaluation: DriverEvaluation, + now: str, + ) -> tuple[ExperimentEvidenceRecordModel, ExperimentDerivedMeasureModel]: + """Build projection-scoped evidence and its derived reward measure.""" + + source_revision = CyberBattleSimEvaluator._source_revision() + evidence_record_id = _scoped_id( + EVALUATION_EVIDENCE_REF, + evaluation.projection_ref, + ) + evidence_record = CyberBattleSimEvaluator._evidence_record( + evaluation, + now, + source_revision, + evidence_record_id, + ) + derived_measure = CyberBattleSimEvaluator._derived_measure( + evaluation, + now, + evidence_record_id, + ) + return evidence_record, derived_measure + + @staticmethod + def _source_revision() -> str: + """Read the selected source revision used in evidence provenance.""" + + qualification = load_qualification() + source = qualification.get("source") + source_revision = source.get("commit") if isinstance(source, dict) else None + if not isinstance(source_revision, str): + raise RuntimeError("selected simulator qualification is invalid") + return source_revision + + @staticmethod + def _evidence_record( + evaluation: DriverEvaluation, + now: str, + source_revision: str, + evidence_record_id: str, + ) -> ExperimentEvidenceRecordModel: + """Build one redacted, checksummed evaluator evidence record.""" + + capture_spec_id = _scoped_id(_CAPTURE_SPEC_ID, evaluation.projection_ref) + capture_requirement_id = _scoped_id( + _CAPTURE_REQUIREMENT_ID, + evaluation.projection_ref, + ) + capture_window_id = _scoped_id(_CAPTURE_WINDOW_ID, evaluation.projection_ref) + payload_summary = ( + "Sanitized evaluator summary: " + f"{evaluation.step_count} source transitions and cumulative " + f"reward {evaluation.cumulative_reward:.17g}." + ) + return ExperimentEvidenceRecordModel( + schema_version="experiment-evidence-record/v1", + evidence_record_id=evidence_record_id, + record_version=_EVIDENCE_RECORD_VERSION, + capture_spec_ref=ExperimentCaptureSpecReferenceModel( + ref_kind="capture-spec", + ref_id=capture_spec_id, + ref_version="1.0.0", + ), + capture_requirement_ref=capture_requirement_id, + run_ref=ExperimentReferenceModel( + ref_kind="run", + ref_id=evaluation.execution_ref, + ref_version="1.0.0", + ), + task_ref=ExperimentTaskReferenceModel( + ref_kind="task", + ref_id=_TASK_ID, + ref_version="1.0.0", + ), + source_refs=[ + ExperimentReferenceModel( + ref_kind="protocol", + ref_id="cyberbattlesim-chain-public", + ref_version=source_revision, + ) + ], + evidence_kind="telemetry", + captured_at=now, + capture_window_ref=capture_window_id, + raw_content=ExperimentRawEvidenceContentModel( + content_uri=(f"urn:raes:{evidence_record_id}:payload-summary"), + content_checksum=ExperimentChecksumModel( + algorithm="sha256", + value=hashlib.sha256(payload_summary.encode("utf-8")).hexdigest(), + ), + payload_summary=payload_summary, + loss_disclosure=( + "Source-native observations, action availability, credentials, " + "reward components, hidden state, and info are withheld." + ), + ), + sensitivity="redacted", + redaction_state="redacted", + provenance_refs=[ + ExperimentReferenceModel( + ref_kind="other", + ref_id="qualification.cyberbattlesim.selected-source", + ref_version=source_revision, + ) + ], + ) + + @staticmethod + def _derived_measure( + evaluation: DriverEvaluation, + now: str, + evidence_record_id: str, + ) -> ExperimentDerivedMeasureModel: + """Build the bounded cumulative-reward derived measure.""" + + derived_measure_id = _scoped_id( + "measure.cyberbattlesim.cumulative-attacker-reward", + evaluation.projection_ref, + ) + return ExperimentDerivedMeasureModel( + schema_version="experiment-derived-measure/v1", + derived_measure_id=derived_measure_id, + measure_version="1.0.0", + measure_kind="score", + metric_ref=ExperimentReferenceModel( + ref_kind="metric-definition", + ref_id="cumulative_attacker_reward", + ref_version="1.0.0", + ), + method=ExperimentDerivedMeasureMethodModel( + method_id="cyberbattlesim-cumulative-reward", + method_version="1.0.0", + name="Source cumulative attacker reward", + description=( + "Report the sanitized cumulative reward maintained by the " + "serialized source driver." + ), + ), + source_evidence_refs=[ + ExperimentEvidenceRecordReferenceModel( + ref_kind="evidence-record", + ref_id=evidence_record_id, + ref_version=_EVIDENCE_RECORD_VERSION, + ) + ], + generated_at=now, + value_status="reported", + value=evaluation.cumulative_reward, + uncertainty="No statistical uncertainty is estimated from this single run.", + limitations=[ + ( + "Single-run cumulative source reward; no deterministic replay " + "or outcome equivalence is claimed." + ), + ( + "Python-global and NumPy-global random streams remain unbound " + "by the selected reset protocol." + ), + ], + provenance_refs=[ + ExperimentReferenceModel( + ref_kind="task", + ref_id=_TASK_ID, + ref_version="1.0.0", + ) + ], + ) + + @staticmethod + def _capture_specification( + evaluation: DriverEvaluation, + now: str, + ) -> ExperimentCaptureSpecModel: + """Build the projection-scoped evaluator capture specification.""" + + capture_spec_id = _scoped_id(_CAPTURE_SPEC_ID, evaluation.projection_ref) + capture_requirement_id = _scoped_id( + _CAPTURE_REQUIREMENT_ID, + evaluation.projection_ref, + ) + capture_window_id = _scoped_id(_CAPTURE_WINDOW_ID, evaluation.projection_ref) + return ExperimentCaptureSpecModel( + schema_version="experiment-capture-spec/v1", + capture_spec_id=capture_spec_id, + spec_version="1.0.0", + title="CyberBattleSim evaluator summary capture", + description=( + "Capture the sanitized evaluator-owned summary used to derive " + "the selected cumulative-reward measure." + ), + scope_refs=[ + ExperimentReferenceModel( + ref_kind="run", + ref_id=evaluation.execution_ref, + ref_version="1.0.0", + ), + ExperimentReferenceModel( + ref_kind="task", + ref_id=_TASK_ID, + ref_version="1.0.0", + ), + ], + capture_windows=[ + ExperimentCaptureWindowModel( + window_id=capture_window_id, + window_kind="event", + starts_at=now, + ends_at=now, + description=( + "One read-only evaluator projection after the current " + "serialized source transition." + ), + ) + ], + capture_requirements={ + capture_requirement_id: ExperimentCaptureRequirementModel( + requirement_id=capture_requirement_id, + title="Sanitized evaluator summary", + capture_kind="telemetry", + capture_scope="run", + channel_ref=ExperimentMeasurementChannelReferenceModel( + ref_kind="measurement-channel", + ref_id="cyberbattlesim-evaluation-history", + ref_version="1.0.0", + ), + window_refs=[capture_window_id], + expected_media_types=["application/json"], + sensitivity="redacted", + redaction_policy="redaction.cyberbattlesim.evaluator-summary", + integrity_requirements=["sha256"], + retention_policy="run-lifetime", + loss_disclosure_required=True, + notes=[ + "Native observations, masks, credentials, reward components, " + "hidden state, and info remain driver-private." + ], + ) + }, + validity_notes=[ + ExperimentValidityNoteModel( + category="reproducibility", + note=( + "The capture attests one observed summary; it does not " + "establish deterministic replay." + ), + mitigation=("Record applied and unbound stochastic streams with the run."), + ) + ], + ) + + +__all__ = [ + "CyberBattleSimEvaluator", + "EVALUATION_EVIDENCE_REF", +] diff --git a/src/raes_adapters/cyberbattlesim/backend/manifest.py b/src/raes_adapters/cyberbattlesim/backend/manifest.py new file mode 100644 index 0000000..972dec6 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/manifest.py @@ -0,0 +1,340 @@ +"""Evidence-bounded CyberBattleSim backend manifest.""" + +from __future__ import annotations + +from importlib.metadata import PackageNotFoundError +from importlib.metadata import version as distribution_version + +from raes_backend_protocols.backend_manifest import ( # type: ignore[import-untyped] + BackendManifest, +) +from raes_backend_protocols.capabilities import ( # type: ignore[import-untyped] # type: ignore[import-untyped] + CLEANUP_CAPABILITY_REQUIRED_CONTRACTS, + PARTICIPANT_RUNTIME_CAPABILITY_REQUIRED_CONTRACTS, + BackendCapabilitySet, + CleanupCapabilities, + EvaluatorCapabilities, + OrchestratorCapabilities, + ParticipantFeatureSupport, + ParticipantRuntimeCapabilities, + ProvisionerCapabilities, + WorkflowFeature, +) +from raes_contracts.apparatus import ( # type: ignore[import-untyped] + ConceptBinding, + RealizationSupportDeclaration, +) +from raes_contracts.backend_profiles import ( # type: ignore[import-untyped] + load_backend_profile, +) +from raes_contracts.contracts import ( # type: ignore[import-untyped] + ExperimentCaptureSpecModel, + ExperimentDerivedMeasureModel, + ExperimentEvidenceRecordModel, + PropositionTruthResultModel, +) +from raes_contracts.contracts.base import ( # type: ignore[import-untyped] + ContractModel, +) +from raes_contracts.vocabulary import ( # type: ignore[import-untyped] + ParticipantFeatureSupportLevel, + RealizationSupportMode, +) + +from raes_adapters.cyberbattlesim import load_qualification + +CYBERBATTLESIM_BACKEND_NAME = "cyberbattlesim" + + +def _adapter_version() -> str: + """Return the installed adapter version or a local-development marker.""" + + try: + return distribution_version("raes-adapters") + except PackageNotFoundError: + return "0.0.0+unknown" + + +def _model_contract_id(model: type[ContractModel]) -> str: + """Resolve a published RAES model's contract identity.""" + + schema_version = model.model_fields["schema_version"].default + if not isinstance(schema_version, str): + schema_property = model.model_json_schema()["properties"]["schema_version"] + schema_version = schema_property.get("const") + if not isinstance(schema_version, str): + raise RuntimeError("published RAES contract model has no schema identity") + return schema_version.replace("/", "-") + + +def _supported_contracts(capabilities: BackendCapabilitySet) -> frozenset[str]: + """Compose this declaration from RAES-owned profiles and model identities.""" + + supported = set(load_backend_profile("full-remote-control-plane").required_contracts) + supported.update(CLEANUP_CAPABILITY_REQUIRED_CONTRACTS) + participant = capabilities.participant_runtime + if participant is not None: + declared_terms = ( + participant.supported_participant_roles + | participant.supported_behavior_features + | participant.supported_interaction_features + ) + for required_by_term in PARTICIPANT_RUNTIME_CAPABILITY_REQUIRED_CONTRACTS.values(): + for term in declared_terms: + supported.update(required_by_term.get(term, ())) + supported.update( + _model_contract_id(model) + for model in ( + PropositionTruthResultModel, + ExperimentCaptureSpecModel, + ExperimentEvidenceRecordModel, + ExperimentDerivedMeasureModel, + ) + ) + return frozenset(supported) + + +def _concept_bindings() -> tuple[ConceptBinding, ...]: + """Bind declared capability scopes to RAES-owned concept families.""" + + return ( + ConceptBinding( + scope="capabilities.provisioner.supported_node_types", + family="assets", + ), + ConceptBinding( + scope="capabilities.provisioner.supported_os_families", + family="assets", + ), + ConceptBinding( + scope="capabilities.provisioner.supported_content_types", + family="tools-and-artifacts", + ), + ConceptBinding( + scope="capabilities.provisioner.supported_account_features", + family="identities", + ), + ConceptBinding( + scope="capabilities.provisioner.supported_domain_profiles", + family="identities", + ), + ConceptBinding( + scope="capabilities.provisioner.supported_service_materialization_profiles", + family="tools-and-artifacts", + ), + ConceptBinding( + scope="capabilities.orchestrator.supported_sections", + family="actions-and-events", + ), + ConceptBinding( + scope="capabilities.evaluator.supported_sections", + family="observables", + ), + ConceptBinding( + scope="capabilities.participant_runtime.supported_participant_roles", + family="identities", + ), + ConceptBinding( + scope="capabilities.participant_runtime.supported_behavior_features", + family="actions-and-events", + ), + ConceptBinding( + scope="capabilities.participant_runtime.supported_interaction_features", + family="relationships", + ), + ) + + +def _provisioner_capabilities() -> ProvisionerCapabilities: + """Declare selected generated-chain provisioning support.""" + + return ProvisionerCapabilities( + name="cyberbattlesim-provisioner", + supported_node_types=frozenset({"switch", "vm"}), + supported_os_families=frozenset({"linux", "windows"}), + supported_content_types=frozenset(), + supported_account_features=frozenset({"groups", "auth_method"}), + supported_domain_profiles=frozenset(), + supported_service_materialization_profiles=frozenset(), + max_total_nodes=10, + supports_accounts=True, + constraints={ + "topology_realization": ( + "selected generated chain; authored topology is representative" + ), + "account_realization": ( + "auth method and group intent map to source-generated credentials; " + "no source identity equivalence" + ), + }, + ) + + +def _orchestrator_capabilities() -> OrchestratorCapabilities: + """Declare portable episode-orchestration support.""" + + return OrchestratorCapabilities( + name="cyberbattlesim-orchestrator", + supported_sections=frozenset({"events", "workflows"}), + supports_workflows=True, + supports_assertion_refs=False, + supports_inject_bindings=False, + supported_workflow_features=frozenset({WorkflowFeature.CALL}), + constraints={ + "native_transition_owner": "participant-runtime", + "workflow_scope": "episode lifecycle and serialized participant steps", + }, + ) + + +def _evaluator_capabilities() -> EvaluatorCapabilities: + """Declare evaluator-owned stochastic result projection support.""" + + return EvaluatorCapabilities( + name="cyberbattlesim-evaluator", + supported_sections=frozenset({"conditions", "propositions", "assertions", "objectives"}), + supports_scoring=True, + supports_objectives=True, + supported_predicate_families=frozenset({"presence", "boolean", "string", "number"}), + supported_quantifiers=frozenset({"all", "any", "at_least"}), + supported_truth_outcomes=frozenset({"true", "false", "unknown", "unsupported"}), + supported_evidence_channels=frozenset({"api_response"}), + supported_time_domains=frozenset({"wall_clock"}), + preserves_binding_provenance=True, + constraints={ + "reward_owner": "evaluator", + "outcome_reproduction": "stochastic-bounded", + "proposition_projection": "binding-preserving unknown under lossy source evidence", + "objective_terminal_state": ( + "running until a distinct mapped terminal cause is available" + ), + }, + ) + + +def _participant_capabilities() -> ParticipantRuntimeCapabilities: + """Declare bounded red-participant runtime support.""" + + return ParticipantRuntimeCapabilities( + name="cyberbattlesim-participant-runtime", + supported_participant_roles=frozenset({"red"}), + supported_behavior_features=frozenset( + { + "action_contracts", + "attribution_support", + "behavior_history", + "effects", + "failure_classes", + "observation_boundaries", + "outcome_interpretation", + "preconditions", + "state_transitions", + "temporal_contracts", + } + ), + supported_interaction_features=frozenset({"interference"}), + feature_support=( + ParticipantFeatureSupport( + feature="interference", + support_level=ParticipantFeatureSupportLevel.DISCLOSED_WEAK, + limitation_refs=("limitation:cyberbattlesim:source-internal-defender",), + disclosure_refs=("docs/decisions/cyberbattlesim-backend-guardrails.md",), + ), + ), + supports_autonomous_execution=False, + supports_bounded_concurrency=False, + constraints={ + "max_in_flight_native_transitions": "1", + "native_action_coordinates": "driver-private", + "native_target_attribution": ( + "unavailable; request targets remain intent and are not echoed " + "as realized effect targets" + ), + "participant_action_scope": ( + "connect, local-vulnerability, and remote-vulnerability attacker contracts" + ), + "source_internal_defender": ( + "scan-and-reimage executes inside the selected source step " + "and is not a participant-admitted action" + ), + }, + ) + + +def _capabilities() -> BackendCapabilitySet: + """Compose the complete selected backend capability declaration.""" + + return BackendCapabilitySet( + provisioner=_provisioner_capabilities(), + orchestrator=_orchestrator_capabilities(), + evaluator=_evaluator_capabilities(), + participant_runtime=_participant_capabilities(), + cleanup=CleanupCapabilities( + name="cyberbattlesim-cleanup", + supported_contract_versions=CLEANUP_CAPABILITY_REQUIRED_CONTRACTS, + supported_action_kinds=frozenset({"destroy", "reset", "verify"}), + supported_verification_methods=frozenset({"probe", "receipt"}), + supports_reusable_state=True, + supports_residual_state_disclosure=True, + ), + ) + + +def create_cyberbattlesim_manifest() -> BackendManifest: + """Return the manifest for the admitted selected source profile.""" + + qualification = load_qualification() + source = qualification.get("source") + source_revision = source.get("commit") if isinstance(source, dict) else None + if not isinstance(source_revision, str): + raise RuntimeError("selected simulator qualification is invalid") + capabilities = _capabilities() + return BackendManifest( + name=CYBERBATTLESIM_BACKEND_NAME, + version=_adapter_version(), + supported_contract_versions=_supported_contracts(capabilities), + compatible_processors=frozenset({"raes-reference-processor"}), + concept_bindings=_concept_bindings(), + realization_support=( + RealizationSupportDeclaration( + domain="runtime-realization", + support_mode=RealizationSupportMode.CONSTRAINED, + supported_constraint_kinds=frozenset( + { + "node-type", + "os-family", + "account-feature", + "workflow-feature", + "workflow-state-predicate", + } + ), + supported_exact_requirement_kinds=frozenset({"declared-capability-match"}), + disclosure_kinds=frozenset( + { + "backend-manifest-v2", + "runtime-snapshot-v1", + "operation-status-v1", + } + ), + constraints={ + "source_profile": f"cyberbattlesim-chain-public-{source_revision[:7]}", + "topology": "representative authored topology maps to generated size-10 chain", + }, + ), + ), + constraints={ + "source_identity": "qualified-complete-runtime-artifact-roots-attested", + "protocol_configuration": "attested", + "execution_controls": "partial", + "run_evidence": "attestable", + "outcome_reproduction": "stochastic-bounded", + "dependency_installation": "separately-installed-pinned-source", + }, + capabilities=capabilities, + ) + + +__all__ = [ + "CYBERBATTLESIM_BACKEND_NAME", + "create_cyberbattlesim_manifest", +] diff --git a/src/raes_adapters/cyberbattlesim/backend/orchestrator.py b/src/raes_adapters/cyberbattlesim/backend/orchestrator.py new file mode 100644 index 0000000..7ebc514 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/orchestrator.py @@ -0,0 +1,222 @@ +"""Portable orchestration state for CyberBattleSim episodes.""" + +from __future__ import annotations + +from datetime import UTC, datetime + +from raes_contracts.contracts import ( # type: ignore[import-untyped] + WorkflowExecutionStateModel, + WorkflowHistoryEventModel, + WorkflowStepStateModel, +) +from raes_contracts.diagnostics import Diagnostic # type: ignore[import-untyped] +from raes_contracts.planning import ( # type: ignore[import-untyped] + ChangeAction, + OrchestrationPlan, + RuntimeDomain, +) +from raes_contracts.runtime_state import ( # type: ignore[import-untyped] + ApplyResult, + RuntimeSnapshot, + SnapshotEntry, +) +from raes_contracts.workflow import ( # type: ignore[import-untyped] + WorkflowCompensationStatus, + WorkflowHistoryEventType, + WorkflowStatus, + WorkflowStepLifecycle, +) + +_SUPPORTED_RESOURCE_TYPES = frozenset({"event", "workflow"}) + + +def _now_iso() -> str: + """Return a portable UTC timestamp.""" + + return datetime.now(UTC).isoformat().replace("+00:00", "Z") + + +class CyberBattleSimOrchestrator(object): + """Own workflow lifecycle only; participant runtime owns source steps.""" + + def __init__(self) -> None: + self._running = False + self._startup_order: list[str] = [] + self._results: dict[str, dict[str, object]] = {} + self._history: dict[str, list[dict[str, object]]] = {} + + def start( + self, + plan: OrchestrationPlan, + snapshot: RuntimeSnapshot, + ) -> ApplyResult: + unsupported = [ + operation + for operation in plan.operations + if operation.resource_type not in _SUPPORTED_RESOURCE_TYPES + ] + if unsupported: + operation = unsupported[0] + return ApplyResult( + success=False, + snapshot=snapshot, + diagnostics=[ + Diagnostic( + code="cyberbattlesim.orchestration.unsupported-resource", + domain="orchestration", + address=operation.address, + message=( + "The CyberBattleSim orchestrator does not support " + "this orchestration resource type." + ), + ) + ], + ) + mutating_operations = [ + operation for operation in plan.operations if operation.action != ChangeAction.UNCHANGED + ] + if not mutating_operations: + return ApplyResult(success=True, snapshot=snapshot) + + entries = dict(snapshot.entries) + results = dict(snapshot.orchestration_results) + history = { + address: list(events) for address, events in snapshot.orchestration_history.items() + } + changed_addresses: list[str] = [] + now = _now_iso() + for operation in plan.operations: + if operation.action == ChangeAction.UNCHANGED: + continue + if operation.action == ChangeAction.DELETE: + entries.pop(operation.address, None) + results.pop(operation.address, None) + history.pop(operation.address, None) + changed_addresses.append(operation.address) + continue + entries[operation.address] = SnapshotEntry( + address=operation.address, + domain=RuntimeDomain.ORCHESTRATION, + resource_type=operation.resource_type, + payload=operation.payload, + ordering_dependencies=operation.ordering_dependencies, + refresh_dependencies=operation.refresh_dependencies, + status="queued", + ) + if operation.resource_type == "workflow": + results[operation.address] = self._workflow_result( + operation.payload, now + ).model_dump(mode="json") + history[operation.address] = [ + self._workflow_started_event(operation.payload, now).model_dump(mode="json") + ] + changed_addresses.append(operation.address) + + self._running = bool(results or changed_addresses) + self._startup_order = list(plan.startup_order) + self._results = results + self._history = history + return ApplyResult( + success=True, + snapshot=snapshot.with_entries( + entries, + orchestration_results=results, + orchestration_history=history, + ), + changed_addresses=changed_addresses, + ) + + @staticmethod + def _workflow_result( + payload: dict[str, object], + now: str, + ) -> WorkflowExecutionStateModel: + result_contract = payload.get("result_contract", {}) + if not isinstance(result_contract, dict): + result_contract = {} + observable_steps_raw = result_contract.get("observable_steps", {}) + observable_steps = { + step_name: WorkflowStepStateModel( + lifecycle=WorkflowStepLifecycle.PENDING.value, + outcome=None, + attempts=0, + ) + for step_name, step_payload in ( + observable_steps_raw.items() if isinstance(observable_steps_raw, dict) else [] + ) + if isinstance(step_payload, dict) + } + return WorkflowExecutionStateModel( + workflow_status=WorkflowStatus.RUNNING.value, + run_id=f"{payload.get('name', 'cyberbattlesim-episode')}-run", + started_at=now, + updated_at=now, + terminal_reason=None, + compensation_status=WorkflowCompensationStatus.NOT_REQUIRED.value, + compensation_started_at=None, + compensation_updated_at=None, + compensation_failures=[], + steps=observable_steps, + ) + + @staticmethod + def _workflow_started_event( + payload: dict[str, object], + now: str, + ) -> WorkflowHistoryEventModel: + execution_contract = payload.get("execution_contract", {}) + start_step = ( + execution_contract.get("start_step") if isinstance(execution_contract, dict) else None + ) + return WorkflowHistoryEventModel( + event_type=WorkflowHistoryEventType.WORKFLOW_STARTED.value, + timestamp=now, + step_name=start_step if isinstance(start_step, str) else None, + branch_name=None, + join_step=None, + outcome=None, + details={}, + ) + + def status(self) -> dict[str, object]: + return { + "running": self._running, + "startup_order": list(self._startup_order), + "results": len(self._results), + } + + def results(self) -> dict[str, dict[str, object]]: + return {address: dict(result) for address, result in self._results.items()} + + def history(self) -> dict[str, list[dict[str, object]]]: + return { + address: [dict(event) for event in events] for address, events in self._history.items() + } + + def stop(self, snapshot: RuntimeSnapshot) -> ApplyResult: + entries = { + address: entry + for address, entry in snapshot.entries.items() + if entry.domain != RuntimeDomain.ORCHESTRATION + } + removed = [ + address + for address, entry in snapshot.entries.items() + if entry.domain == RuntimeDomain.ORCHESTRATION + ] + self._running = False + self._startup_order = [] + self._results = {} + self._history = {} + return ApplyResult( + success=True, + snapshot=snapshot.with_entries( + entries, + orchestration_results={}, + orchestration_history={}, + ), + changed_addresses=removed, + ) + + +__all__ = ["CyberBattleSimOrchestrator"] diff --git a/src/raes_adapters/cyberbattlesim/backend/participant_runtime.py b/src/raes_adapters/cyberbattlesim/backend/participant_runtime.py new file mode 100644 index 0000000..bb59d81 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/participant_runtime.py @@ -0,0 +1,461 @@ +"""CyberBattleSim participant lifecycle and one-step action execution.""" + +from __future__ import annotations + +from datetime import UTC, datetime + +from raes_backend_protocols.participant_runtime_base import ( # type: ignore[import-untyped] + BaseParticipantRuntime, +) +from raes_contracts.contracts import ( # type: ignore[import-untyped] + ParticipantActionEffectResultModel, + ParticipantActionResultModel, + ParticipantObservationEnvelopeModel, + ParticipantObservationLossDescriptorModel, + ParticipantObservationStochasticContextModel, + SourceStatusModel, +) +from raes_contracts.diagnostics import ( # type: ignore[import-untyped] + Diagnostic, + Severity, +) +from raes_contracts.participant_binding import ( # type: ignore[import-untyped] + ParticipantActionAdmissionRequest, + ParticipantNativeActionExecution, +) +from raes_contracts.participant_episode import ( # type: ignore[import-untyped] + ParticipantEpisodeInitializeRequest, + ParticipantEpisodeResetRequest, + ParticipantEpisodeRestartRequest, + ParticipantEpisodeTerminalReason, + ParticipantEpisodeTerminateRequest, +) +from raes_contracts.runtime_state import ( # type: ignore[import-untyped] + ApplyResult, + RuntimeSnapshot, +) + +from .driver import ( + CyberBattleSimDriverProtocol, + DriverResetReport, + DriverStep, +) + +ACTION_EVIDENCE_REF = "evidence.cyberbattlesim.attacker-action" + +_ACTION_KIND_BY_CONTRACT = { + "participant.action-contract.connect": "connect", + "participant.action-contract.local-vulnerability": "local-vulnerability", + "participant.action-contract.remote-vulnerability": "remote-vulnerability", +} + + +def _now_iso() -> str: + """Return a portable UTC timestamp.""" + + return datetime.now(UTC).isoformat().replace("+00:00", "Z") + + +class CyberBattleSimParticipantRuntime(BaseParticipantRuntime): # type: ignore[misc] + """Bind RAES participant lifecycle to exactly one serialized source action.""" + + def __init__( + self, + driver: CyberBattleSimDriverProtocol, + *, + seed: int | None = None, + ) -> None: + super().__init__() + self._driver = driver + self._seed = seed + self._observations: dict[ + str, + list[ParticipantObservationEnvelopeModel], + ] = {} + self._driver_operation_refs: dict[str, str] = {} + + def initialize( + self, + request: ParticipantEpisodeInitializeRequest, + snapshot: RuntimeSnapshot, + ) -> ApplyResult: + portable = super().initialize(request, snapshot) + return self._reset_source_after_portable( + request.participant_address, + portable, + snapshot, + ) + + def reset( + self, + request: ParticipantEpisodeResetRequest, + snapshot: RuntimeSnapshot, + ) -> ApplyResult: + portable = super().reset(request, snapshot) + return self._reset_source_after_portable( + request.participant_address, + portable, + snapshot, + ) + + def restart( + self, + request: ParticipantEpisodeRestartRequest, + snapshot: RuntimeSnapshot, + ) -> ApplyResult: + portable = super().restart(request, snapshot) + return self._reset_source_after_portable( + request.participant_address, + portable, + snapshot, + ) + + def _reset_source_after_portable( + self, + participant_address: str, + portable: ApplyResult, + predecessor: RuntimeSnapshot, + ) -> ApplyResult: + if not portable.success: + return portable + try: + report = self._driver.reset(self._seed) + except Exception: + self._restore_portable_mirror(predecessor) + return ApplyResult( + success=False, + snapshot=predecessor, + diagnostics=[ + Diagnostic( + code="cyberbattlesim.participant.reset-failed", + domain="participant", + address=participant_address, + message=("The selected CyberBattleSim episode could not be reset."), + ) + ], + ) + self._observations[participant_address] = [] + return ApplyResult( + success=True, + snapshot=portable.snapshot, + diagnostics=[ + *portable.diagnostics, + *self._reset_diagnostics(participant_address, report), + ], + changed_addresses=list(portable.changed_addresses), + details=dict(portable.details), + ) + + def _restore_portable_mirror(self, snapshot: RuntimeSnapshot) -> None: + self._results = { + address: dict(result) + for address, result in snapshot.participant_episode_results.items() + } + self._history = { + address: [dict(event) for event in events] + for address, events in snapshot.participant_episode_history.items() + } + + @staticmethod + def _reset_diagnostics( + participant_address: str, + report: DriverResetReport, + ) -> list[Diagnostic]: + diagnostics = [ + Diagnostic( + code="cyberbattlesim.seed.applied", + domain="participant", + address=participant_address, + message=("The adapter applied a selected stochastic-control binding."), + severity=Severity.INFO, + ) + for _stream in report.applied_streams + ] + diagnostics.extend( + Diagnostic( + code="cyberbattlesim.seed.unbound", + domain="participant", + address=participant_address, + message=("A selected CyberBattleSim random stream remains unbound."), + severity=Severity.WARNING, + ) + for _stream in report.unbound_streams + ) + return diagnostics + + def _model_action( + self, + request: ParticipantActionAdmissionRequest, + snapshot: RuntimeSnapshot, + *, + episode_id: str, + ) -> ParticipantNativeActionExecution: + """Validate the portable action kind before native execution.""" + + action_kind = _ACTION_KIND_BY_CONTRACT.get(request.action_contract_address) + if action_kind is None or request.validated_selection is None: + return self._rejected_action( + request, + snapshot, + episode_id=episode_id, + failure_class="unsupported_action", + code="cyberbattlesim.participant.unsupported-action", + message=( + "The selected CyberBattleSim profile does not support this participant action." + ), + ) + return self._execute_supported_action( + request, + snapshot, + episode_id=episode_id, + action_kind=action_kind, + ) + + def _execute_supported_action( + self, + request: ParticipantActionAdmissionRequest, + snapshot: RuntimeSnapshot, + *, + episode_id: str, + action_kind: str, + ) -> ParticipantNativeActionExecution: + """Execute one supported action and bound native failures.""" + + try: + step = self._driver.step(action_kind) + except Exception: + return self._rejected_action( + request, + snapshot, + episode_id=episode_id, + failure_class="backend_error", + code="cyberbattlesim.participant.action-failed", + message=( + "The CyberBattleSim driver could not complete the admitted participant action." + ), + status="failed", + ) + + self._driver_operation_refs[request.action_instance_id] = step.operation_ref + if not step.source_transition: + return self._rejected_action( + request, + snapshot, + episode_id=episode_id, + failure_class="target_unavailable", + code="cyberbattlesim.participant.action-unavailable", + message=("No source transition was available for the admitted participant action."), + ) + return self._accepted_action( + request, + snapshot, + episode_id=episode_id, + step=step, + ) + + def _accepted_action( + self, + request: ParticipantActionAdmissionRequest, + snapshot: RuntimeSnapshot, + *, + episode_id: str, + step: DriverStep, + ) -> ParticipantNativeActionExecution: + """Build portable results for one completed source transition.""" + + observation = self._observation( + request, + episode_id=episode_id, + step=step, + ) + self._observations.setdefault(request.participant_address, []).append(observation) + status = "succeeded" if step.processed else "failed" + failure_class = None if step.processed else "unknown" + evidence_refs = ( + [ACTION_EVIDENCE_REF] + if ACTION_EVIDENCE_REF in request.observation_boundary_evidence_refs + else [] + ) + action_result = ParticipantActionResultModel( + status=status, + participant_address=request.participant_address, + episode_id=episode_id, + action_instance_id=request.action_instance_id, + action_contract_address=request.action_contract_address, + observation_point=observation.observation_ref, + effects=[ + ParticipantActionEffectResultModel( + effect_id=f"{request.action_instance_id}.source-transition", + effect_class="unknown_effect", + description=( + "The selected source processed one participant action; " + "the native effect remains outside this projection." + ), + target_refs=list(step.portable_target_refs), + evidence_refs=evidence_refs, + ) + ], + failure_class=failure_class, + observations=[observation.observation_ref], + evidence_refs=evidence_refs, + ) + + modeled = ApplyResult(success=True, snapshot=snapshot) + if step.terminal_cause is not None: + terminal_reason = ( + ParticipantEpisodeTerminalReason.TRUNCATED + if step.truncated or step.terminal_cause == "evaluator-cutoff" + else ParticipantEpisodeTerminalReason.COMPLETED + ) + modeled = BaseParticipantRuntime.terminate( + self, + ParticipantEpisodeTerminateRequest( + participant_address=request.participant_address, + terminal_reason=terminal_reason, + detail=("The selected source reached a terminal episode condition."), + ), + snapshot, + ) + return ParticipantNativeActionExecution( + apply_result=modeled, + action_result=action_result, + ) + + def _rejected_action( + self, + request: ParticipantActionAdmissionRequest, + snapshot: RuntimeSnapshot, + *, + episode_id: str, + failure_class: str, + code: str, + message: str, + status: str = "rejected", + ) -> ParticipantNativeActionExecution: + observation_point = ( + f"observation.cyberbattlesim.{episode_id}.{request.action_instance_id}.withheld" + ) + action_result = ParticipantActionResultModel( + status=status, + participant_address=request.participant_address, + episode_id=episode_id, + action_instance_id=request.action_instance_id, + action_contract_address=request.action_contract_address, + observation_point=observation_point, + failure_class=failure_class, + ) + return ParticipantNativeActionExecution( + apply_result=ApplyResult( + success=False, + snapshot=snapshot, + diagnostics=[ + Diagnostic( + code=code, + domain="participant", + address=request.participant_address, + message=message, + ) + ], + ), + action_result=action_result, + ) + + def _observation( + self, + request: ParticipantActionAdmissionRequest, + *, + episode_id: str, + step: DriverStep, + ) -> ParticipantObservationEnvelopeModel: + now = _now_iso() + observation_ref = ( + f"observation.cyberbattlesim.{episode_id}." + f"{step.step_number}.{request.action_instance_id}" + ) + evidence_refs = ( + [ACTION_EVIDENCE_REF] + if ACTION_EVIDENCE_REF in request.observation_boundary_evidence_refs + else [] + ) + return ParticipantObservationEnvelopeModel( + event_id=f"event.{observation_ref}", + schema_name="raes.participant_runtime.observation", + schema_version="1.0.0", + event_type="observation_emission", + extension_policy="reject_unknown_required", + source_status=SourceStatusModel( + status_id=1 if step.processed else 2, + status="success" if step.processed else "failure", + status_code=( + "participant_action_processed" + if step.processed + else "participant_action_failed" + ), + status_detail=("The selected source processed one participant action."), + source_status_label="selected-source-action", + source_status_mapping="raes.participant-action.terminal", + ), + participant_address=request.participant_address, + episode_id=episode_id, + sequence_number=step.step_number, + occurred_at=now, + recorded_at=now, + ingested_at=now, + clock_authority="cyberbattlesim-serialized-source-order", + ordering_basis="serialized_backend_order", + actor_ref=request.participant_address, + producer_ref="backend.cyberbattlesim.participant-runtime", + source_system_ref="qualification.cyberbattlesim.selected-source", + provenance_refs=[ + "qualification.cyberbattlesim.selected-source", + "mapping.cyberbattlesim.loss-disclosures", + ], + evidence_refs=evidence_refs, + redaction_policy_ref="redaction.cyberbattlesim.participant-view", + authorization_scope=f"participant:{request.participant_address}", + observation_ref=observation_ref, + visibility_projection_ref=request.observation_boundary_address, + information_guarantee="lossy_projection", + delivery_basis="emission_is_delivery", + delivered_at=now, + hidden_state_refs=[], + centralized_state_refs=[], + loss_descriptor=ParticipantObservationLossDescriptorModel( + kind="bounded-source-projection", + fields_redacted=[ + "native-state", + "native-action-availability", + "native-credential-material", + "evaluator-only-facts", + ], + ), + stochastic_context=ParticipantObservationStochasticContextModel( + seed_ref=("seed.cyberbattlesim.public" if self._seed is not None else None), + randomization_policy_ref=( + "qualification.cyberbattlesim.partial-stochastic-control" + ), + ), + redacted_field_refs=[ + "native-state", + "evaluator-only-facts", + ], + ) + + def observations( + self, + participant_address: str, + ) -> tuple[ParticipantObservationEnvelopeModel, ...]: + """Return only the requested participant's sealed observation models.""" + + return tuple(self._observations.get(participant_address, ())) + + def driver_operation_ref(self, action_instance_id: str) -> str | None: + """Return the adapter-generated join for one action, never native data.""" + + return self._driver_operation_refs.get(action_instance_id) + + +__all__ = [ + "ACTION_EVIDENCE_REF", + "CyberBattleSimParticipantRuntime", +] diff --git a/src/raes_adapters/cyberbattlesim/backend/provisioner.py b/src/raes_adapters/cyberbattlesim/backend/provisioner.py new file mode 100644 index 0000000..760a84f --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/provisioner.py @@ -0,0 +1,142 @@ +"""CyberBattleSim provisioner over the selected source profile.""" + +from __future__ import annotations + +from raes_contracts.diagnostics import Diagnostic # type: ignore[import-untyped] +from raes_contracts.planning import ( # type: ignore[import-untyped] + ChangeAction, + ProvisioningPlan, + RuntimeDomain, +) +from raes_contracts.runtime_state import ( # type: ignore[import-untyped] + ApplyResult, + RuntimeSnapshot, + SnapshotEntry, +) + +from .driver import CyberBattleSimDriverProtocol + +_SUPPORTED_RESOURCE_TYPES = frozenset( + { + "network", + "node", + "account-placement", + } +) + + +class CyberBattleSimProvisioner(object): + """Realize the selected generated chain while preserving portable intent.""" + + def __init__(self, driver: CyberBattleSimDriverProtocol) -> None: + self._driver = driver + + @staticmethod + def validate(plan: ProvisioningPlan) -> list[Diagnostic]: + diagnostics = list(plan.diagnostics) + for operation in plan.operations: + if operation.resource_type not in _SUPPORTED_RESOURCE_TYPES: + diagnostics.append( + Diagnostic( + code="cyberbattlesim.provisioning.unsupported-resource", + domain="provisioning", + address=operation.address, + message=( + "The selected CyberBattleSim profile does not realize " + "this provisioning resource type." + ), + ) + ) + return diagnostics + + def apply( + self, + plan: ProvisioningPlan, + snapshot: RuntimeSnapshot, + ) -> ApplyResult: + """Apply selected-source construction and portable state changes.""" + + diagnostics = self.validate(plan) + preflight = self._preflight(plan, snapshot, diagnostics) + if preflight is not None: + return preflight + + entries = dict(snapshot.entries) + changed_addresses: list[str] = [] + for operation in plan.operations: + if operation.action == ChangeAction.UNCHANGED: + continue + if operation.action == ChangeAction.DELETE: + entries.pop(operation.address, None) + changed_addresses.append(operation.address) + continue + entries[operation.address] = SnapshotEntry( + address=operation.address, + domain=RuntimeDomain.PROVISIONING, + resource_type=operation.resource_type, + payload=operation.payload, + ordering_dependencies=operation.ordering_dependencies, + refresh_dependencies=operation.refresh_dependencies, + status="applied", + ) + changed_addresses.append(operation.address) + + return ApplyResult( + success=True, + snapshot=snapshot.with_entries(entries), + diagnostics=diagnostics, + changed_addresses=changed_addresses, + ) + + def _preflight( + self, + plan: ProvisioningPlan, + snapshot: RuntimeSnapshot, + diagnostics: list[Diagnostic], + ) -> ApplyResult | None: + """Return a terminal validation, no-op, or construction result.""" + + if any(diagnostic.is_error for diagnostic in diagnostics): + return ApplyResult( + success=False, + snapshot=snapshot, + diagnostics=diagnostics, + ) + mutating_operations = [ + operation for operation in plan.operations if operation.action != ChangeAction.UNCHANGED + ] + if not mutating_operations: + return ApplyResult(success=True, snapshot=snapshot, diagnostics=diagnostics) + should_construct = any( + operation.action in {ChangeAction.CREATE, ChangeAction.UPDATE} + for operation in plan.operations + ) + return self._construct_failure(snapshot, diagnostics) if should_construct else None + + def _construct_failure( + self, + snapshot: RuntimeSnapshot, + diagnostics: list[Diagnostic], + ) -> ApplyResult | None: + """Construct the selected source and bound any native failure.""" + + try: + self._driver.construct() + except Exception: + diagnostics.append( + Diagnostic( + code="cyberbattlesim.provisioning.construct-failed", + domain="provisioning", + address="provision.cyberbattlesim.selected-profile", + message=("The selected CyberBattleSim source could not be constructed."), + ) + ) + return ApplyResult( + success=False, + snapshot=snapshot, + diagnostics=diagnostics, + ) + return None + + +__all__ = ["CyberBattleSimProvisioner"] diff --git a/src/raes_adapters/cyberbattlesim/backend/target.py b/src/raes_adapters/cyberbattlesim/backend/target.py new file mode 100644 index 0000000..c6cb14b --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/backend/target.py @@ -0,0 +1,64 @@ +"""Runtime target composition for the CyberBattleSim backend.""" + +from __future__ import annotations + +from raes_runtime.registry import ( # type: ignore[import-untyped] + RuntimeTarget, + RuntimeTargetComponents, +) + +from raes_adapters.base import build_runtime_target + +from .driver import CyberBattleSimDriver, CyberBattleSimDriverProtocol +from .evaluator import CyberBattleSimEvaluator +from .manifest import ( + CYBERBATTLESIM_BACKEND_NAME, + create_cyberbattlesim_manifest, +) +from .orchestrator import CyberBattleSimOrchestrator +from .participant_runtime import CyberBattleSimParticipantRuntime +from .provisioner import CyberBattleSimProvisioner + + +def create_cyberbattlesim_components( + *, + driver: CyberBattleSimDriverProtocol, + seed: int | None = None, +) -> RuntimeTargetComponents: + """Create components sharing one driver-owned source lifecycle.""" + + return RuntimeTargetComponents( + provisioner=CyberBattleSimProvisioner(driver), + orchestrator=CyberBattleSimOrchestrator(), + evaluator=CyberBattleSimEvaluator(driver), + participant_runtime=CyberBattleSimParticipantRuntime( + driver, + seed=seed, + ), + ) + + +def create_cyberbattlesim_target( + *, + driver: CyberBattleSimDriverProtocol | None = None, + seed: int | None = None, +) -> RuntimeTarget: + """Return a validated RAES target for the selected source profile.""" + + selected_driver = driver if driver is not None else CyberBattleSimDriver() + manifest = create_cyberbattlesim_manifest() + components = create_cyberbattlesim_components( + driver=selected_driver, + seed=seed, + ) + return build_runtime_target( + CYBERBATTLESIM_BACKEND_NAME, + manifest, + components, + ) + + +__all__ = [ + "create_cyberbattlesim_components", + "create_cyberbattlesim_target", +] diff --git a/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.task.exp.yaml b/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.task.exp.yaml index 835b8e1..71aa35a 100644 --- a/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.task.exp.yaml +++ b/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.task.exp.yaml @@ -19,7 +19,8 @@ evaluation_protocol: protocol_version: 1.0.0 intent: >- Measure attacker progress, reward, availability, and terminal cause for the - selected public protocol without asserting reproducibility or admissibility. + selected admitted public protocol without asserting deterministic replay or + outcome equivalence. unit_of_analysis: episode metric_definitions: steps_to_termination: @@ -83,7 +84,7 @@ evaluation_protocol: ref_id: availability-series intended_use: >- Define the portable evaluation protocol for the selected public CyberBattleSim - case so a future admitted run can be scored consistently. + case so an attested run can be scored consistently. population_or_construct: >- A single credential-cache baseline attacker against a basic scan-and-reimage defender on the fixed CyberBattleChain-v0 topology. @@ -94,9 +95,9 @@ split_and_leakage_controls: apparatus_constraints: notes: - >- - No CyberBattleSim adapter, backend, or participant manifest is implemented. - The case is not-admissible to RAESystem/research#12 (no index/release - artifact, incomplete random-stream binding, unresolved benchmark defects). + The maintainer-selected profile is admitted. No index/release artifact, + incomplete random-stream binding, and unresolved benchmark defects limit + installability, deterministic-replay, and outcome claims. validity_notes: - category: construct note: >- @@ -118,8 +119,8 @@ artifact_refs: uri: file://public-protocol.md checksum: algorithm: sha256 - value: e20b197f48100f0c33b4b7ae55d02bbace8fc2da5afe6552b14566648371e4c6 - size_bytes: 6804 + value: 0de0d8585ee9e3bb0901bd2d71b71a9504774b38bdab38f6f5509d0a9e34d26f + size_bytes: 6864 created_at: "2026-07-29T00:00:00Z" source: RAESystem/adapters#25 qualification sensitivity: public diff --git a/src/raes_adapters/cyberbattlesim/mapping/README.md b/src/raes_adapters/cyberbattlesim/mapping/README.md index 24cf6a5..3abf883 100644 --- a/src/raes_adapters/cyberbattlesim/mapping/README.md +++ b/src/raes_adapters/cyberbattlesim/mapping/README.md @@ -55,7 +55,7 @@ field to carry raw native arrays, reward vectors, or action ids (a structural exclusion), and a deterministic scan rejects known native identifiers (grounded in the qualification record's recorded observation keys) and native object/array representations in portable content. No claim rests on CI success, a single -cumulative score, or a top-level seed. The current -qualification is `not-admissible`; authoring and validating this evidence set -does not change that and asserts no installability, replay, conformance, or -equivalence. +cumulative score, or a top-level seed. The selected profile is admitted; +authoring and validating this evidence set strengthens its provenance without +asserting dependency installability, deterministic replay, conformance, or +outcome equivalence. diff --git a/src/raes_adapters/cyberbattlesim/mapping/loss-disclosures.md b/src/raes_adapters/cyberbattlesim/mapping/loss-disclosures.md index 99c674d..51ee13c 100644 --- a/src/raes_adapters/cyberbattlesim/mapping/loss-disclosures.md +++ b/src/raes_adapters/cyberbattlesim/mapping/loss-disclosures.md @@ -21,12 +21,12 @@ names no loss, or a loss declares no tier. The official Microsoft source declares `cyberbattlesim==0.1.0` but publishes neither a public index distribution nor a release artifact, and PyPI does not accept direct-URL dependencies. The exact selected commit therefore cannot be -reinstalled through a published, hash-locked route, so no `cyberbattlesim` extra -is advertised. This defeats the base reproducibility precondition on which every -higher equivalence tier depends: an apparatus that cannot be reconstructed from a -published artifact cannot ground a portable replication claim. The gap is a -standing admissibility blocker for RAESystem/research#12, not a packaging -convenience to be papered over with a floating Git reference or an empty extra. +reinstalled through a published, hash-locked dependency route. The +dependency-light `cyberbattlesim` extra therefore carries the adapter without +pretending to install upstream. This weakens apparatus reconstruction and every +higher equivalence tier. It does not make the maintainer-selected simulator +unavailable to the adapter, and the limitation must not be hidden with a +floating Git reference. ## loss-unbound-random-streams diff --git a/src/raes_adapters/cyberbattlesim/mapping/source-ledger.jsonl b/src/raes_adapters/cyberbattlesim/mapping/source-ledger.jsonl index f262b6f..53c4358 100644 --- a/src/raes_adapters/cyberbattlesim/mapping/source-ledger.jsonl +++ b/src/raes_adapters/cyberbattlesim/mapping/source-ledger.jsonl @@ -26,4 +26,4 @@ {"source_id": "provenance-source-identity", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "setup.py", "source_selector": "setup name and version", "source_digest": "69af36bcb04450d6dd8f71f23b9340288cbb461021128915eacd28408647c949", "category": "provenance-licensing", "disposition": "mapped", "raes_target": "resource:qualification.json", "mapping_rule": "The distribution name and version (cyberbattlesim 0.1.0) and immutable commit/tree/archive identity are owned by the qualification record.", "verification": "digest joins qualification.source_files; resource:qualification.json present in package."} {"source_id": "provenance-license", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "LICENSE", "source_selector": "MIT license text", "source_digest": "3d4ada4e04d153d74f5bc4e5e6aebd12ef20077716529c36089dc07f14fd0dcf", "category": "provenance-licensing", "disposition": "mapped", "raes_target": "resource:qualification.json", "mapping_rule": "The MIT license and redistribution/attribution duties are owned by the qualification record's legal disposition.", "verification": "digest joins qualification.source_files; resource:qualification.json present in package."} {"source_id": "provenance-notice", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/NOTICE", "source_selector": "NOTICE third-party attributions", "source_digest": "5fc64ab06837e47cb22c6fbbad556734947ec0c4af6eb0d2700668447b63d01b", "category": "provenance-licensing", "disposition": "mapped", "raes_target": "resource:qualification.json", "mapping_rule": "The bundled NOTICE and its third-party attribution components are owned by the qualification record's legal disposition.", "verification": "digest joins qualification.source_files; resource:qualification.json present in package."} -{"source_id": "admissibility-no-artifact", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "requirements.txt", "source_selector": "install requirements", "source_digest": "76f9e396a2c21af090844e756b2aa965f5eda3f15537daa70bc135e365d9c494", "category": "provenance-licensing", "disposition": "loss-disclosed", "loss_ref": "loss-no-source-artifact", "equivalence_tier": "reproducibility", "mapping_rule": "The official source declares cyberbattlesim 0.1.0 but publishes no index distribution or release artifact, so the exact source cannot be portably reinstalled and no cyberbattlesim extra is published.", "verification": "digest joins qualification.source_files; loss-no-source-artifact declared with tier reproducibility."} +{"source_id": "packaging-no-artifact", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "requirements.txt", "source_selector": "install requirements", "source_digest": "76f9e396a2c21af090844e756b2aa965f5eda3f15537daa70bc135e365d9c494", "category": "provenance-licensing", "disposition": "loss-disclosed", "loss_ref": "loss-no-source-artifact", "equivalence_tier": "reproducibility", "mapping_rule": "The official source declares cyberbattlesim 0.1.0 but publishes no index distribution or release artifact, so the selected source cannot be portably reinstalled as a hash-locked dependency; the dependency-light adapter extra does not pretend otherwise.", "verification": "digest joins qualification.source_files; loss-no-source-artifact declared with tier reproducibility."} diff --git a/src/raes_adapters/cyberbattlesim/public-protocol.md b/src/raes_adapters/cyberbattlesim/public-protocol.md index 915274e..db20308 100644 --- a/src/raes_adapters/cyberbattlesim/public-protocol.md +++ b/src/raes_adapters/cyberbattlesim/public-protocol.md @@ -5,9 +5,10 @@ This is the one selected source-native protocol for the CyberBattleSim qualification in RAESystem/adapters issue #25. It is backend-local evidence, not a RAES-authored scenario, adapter manifest, conformance profile, or claim -of equivalence. The associated qualification record currently classifies the -case as **not admissible** to RAESystem/research#12; this protocol fixes what -would be run after the admission blockers are resolved. +of equivalence. The maintainer-selected profile is admitted for adapter +realization. The associated qualification record fixes the source identity and +bounds the attainable claims; its packaging, stochastic-control, and upstream +defect limitations do not veto execution. The source is Microsoft's official [`microsoft/CyberBattleSim`](https://github.com/microsoft/CyberBattleSim) diff --git a/src/raes_adapters/cyberbattlesim/qualification.json b/src/raes_adapters/cyberbattlesim/qualification.json index a71bca0..83d925e 100644 --- a/src/raes_adapters/cyberbattlesim/qualification.json +++ b/src/raes_adapters/cyberbattlesim/qualification.json @@ -27,6 +27,10 @@ "path": "cyberbattle/NOTICE", "sha256": "5fc64ab06837e47cb22c6fbbad556734947ec0c4af6eb0d2700668447b63d01b" }, + { + "path": "cyberbattle/__init__.py", + "sha256": "146a6297d4361b7cd550e293e050c98b6795c4fc5416a1e72f62a8dc142668c4" + }, { "path": "cyberbattle/_env/cyberbattle_env.py", "sha256": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9" @@ -60,6 +64,71 @@ "sha256": "0462976356adbaad02bfc0ef125ec286d56065986d887e389b8b9419d40be6d1" } ], + "runtime_source_tree": { + "root": "cyberbattle", + "include": "all regular files; symlinks and unlisted files are rejected", + "file_count": 48, + "canonicalization": "Sorted POSIX path, NUL, lowercase SHA-256 of raw file bytes, LF.", + "sha256": "aa97624b86b2f5c9ad80ea52a345d5fd6f8fa89cad31070c5e4748c55510ba02" + }, + "runtime_artifacts": [ + { + "name": "cyberbattlesim", + "version": "0.1.0", + "artifact": { + "filename": "cyberbattlesim-0.1.0-py3-none-any.whl", + "sha256": "6e5f855a999ccfcb93f643dda9cbf7ac679640c67df246f4750c344acf2e2138", + "require_direct_archive_sha256": false, + "runtime_identity": "complete-root-tree" + }, + "roots": [ + { + "path": "cyberbattle", + "file_count": 48, + "sha256": "aa97624b86b2f5c9ad80ea52a345d5fd6f8fa89cad31070c5e4748c55510ba02" + } + ] + }, + { + "name": "gymnasium", + "version": "0.29.1", + "artifact": { + "filename": "gymnasium-0.29.1-py3-none-any.whl", + "sha256": "61c3384b5575985bb7f85e43213bcb40f36fcdff388cae6bc229304c71f2843e", + "require_direct_archive_sha256": true, + "runtime_identity": "complete-root-tree" + }, + "roots": [ + { + "path": "gymnasium", + "file_count": 252, + "sha256": "c165ab8c58c65f7481aab52f532eeafc754c1cbe3ab369c6ee7d5d41291c467a" + } + ] + }, + { + "name": "numpy", + "version": "1.26.4", + "artifact": { + "filename": "numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", + "sha256": "675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed", + "require_direct_archive_sha256": true, + "runtime_identity": "complete-root-tree" + }, + "roots": [ + { + "path": "numpy", + "file_count": 781, + "sha256": "799d65585b710293c6b526ea2968e437743b2a53228cd2c7aad9b6e358bffe37" + }, + { + "path": "numpy.libs", + "file_count": 3, + "sha256": "cf040ac02020f0dbb7f41d0bb78050b176b681ed1c9868ea4953de9ce1b8de4c" + } + ] + } + ], "benchmark_snapshot": { "tag": "benchmark-2024-08-08", "commit": "2cbe240a760bdf15e16277110e3065d19eb6bdbe", @@ -72,7 +141,7 @@ }, "protocol": { "resource": "public-protocol.md", - "sha256": "e20b197f48100f0c33b4b7ae55d02bbace8fc2da5afe6552b14566648371e4c6", + "sha256": "0de0d8585ee9e3bb0901bd2d71b71a9504774b38bdab38f6f5509d0a9e34d26f", "selection": { "scenario": { "gym_id": "CyberBattleChain-v0", @@ -164,7 +233,7 @@ "normalized_environment_sha256": "7683481b42452df410d7ebd6dc1b2f3f0f4019c393f63677e707d106267b1b81", "resolver": "uv", "python": "3.12.3", - "artifact_hashes": "The selected upstream wheel is hashed; the normalized installed graph is hashed as a set. Upstream does not publish a hash-locked dependency graph.", + "artifact_hashes": "The selected simulator wheel and the directly imported Gymnasium and NumPy wheels are SHA-256 pinned. Complete import-root tree digests cover all regular files, including native libraries, and reject symlinks or unlisted files before driver import. The remaining normalized installed graph is evidence rather than a runtime authenticity claim.", "lock_disposition": "Evidence only. No second repository lockfile is introduced." }, "dependencies": [ @@ -479,24 +548,28 @@ } ], "packaging": { - "repository_extra": null, + "repository_extra": "cyberbattlesim", "base_and_cyborg_unchanged": true, - "decision": "Do not publish an empty, direct-URL, or knowingly uninstallable cyberbattlesim extra.", - "reason": "The official source declares cyberbattlesim 0.1.0 but publishes neither an index distribution nor a release artifact. PyPI does not accept direct-URL dependencies." + "decision": "Publish a dependency-light adapter extra and keep simulator acquisition outside the wheel metadata.", + "reason": "The official source declares cyberbattlesim 0.1.0 but publishes neither an index distribution nor a release artifact. The adapter therefore checks for the separately installed pinned source at target construction and reports an explicit bounded failure when it is absent." }, - "admissibility": { + "admission": { "target": "RAESystem/research#12", - "decision": "not-admissible", - "decided_at": "2026-07-29", - "blockers": [ + "decision": "admitted", + "authority": "maintainer-selection", + "decided_at": "2026-07-31", + "limitations": [ "no-index-or-release-artifact", "incomplete-random-stream-binding", "unresolved-upstream-benchmark-defects" ], - "conditions_for_requalification": [ - "Select an official index-publishable artifact or approve and qualify a governed repackaging with complete notices and provenance.", - "Bind every observed random stream through reset and evaluation, then repeat the clean smoke and experiment run.", - "Resolve or explicitly patch and qualify the material benchmark/evaluation defects against unmodified source behavior." - ] + "claim_strength": { + "source_identity": "attested", + "protocol_configuration": "attested", + "execution_controls": "partial", + "run_evidence": "attestable", + "outcome_reproduction": "stochastic-bounded" + }, + "interpretation": "The maintainer-selected profile is admitted for adapter realization. Packaging gaps, uncontrolled random streams, and upstream defects bound the claims that RAES may attest; they do not veto the simulator." } } diff --git a/src/raes_adapters/cyborg/__init__.py b/src/raes_adapters/cyborg/__init__.py index 1a055c6..a815482 100644 --- a/src/raes_adapters/cyborg/__init__.py +++ b/src/raes_adapters/cyborg/__init__.py @@ -7,6 +7,9 @@ The native backend is deliberately imported only when the default construction driver is used. Base-only installations therefore remain independent, while a user-installed selected CybORG source checkout can be driven through RAES. +The selected profile is admitted; the absence of a governed public artifact +containing the packaging fix limits automatic-installation claims rather than +backend use. """ from __future__ import annotations diff --git a/src/raes_adapters/cyborg/mapping/README.md b/src/raes_adapters/cyborg/mapping/README.md index f06280d..f48c5b3 100644 --- a/src/raes_adapters/cyborg/mapping/README.md +++ b/src/raes_adapters/cyborg/mapping/README.md @@ -2,9 +2,10 @@ This backend-local evidence bridges the source closure qualified by issue #12 to the published RAES surfaces that can carry each CAGE-2 fact. It does not -author an SDL scenario or experiment or establish equivalence. Maintainer -selection admits the backend; this ledger grades what RAES can record and what -remains lossy. +author an SDL scenario or experiment, change the qualification's source +evidence or claim strength, or establish equivalence. Maintainer selection +admits the backend; this ledger grades what RAES can record and what remains +lossy. Per RAES ADR-069 §2 and `docs/decisions/cage-2-replication-design.md`, the mapping from upstream CAGE-2/CybORG source facts to portable RAES artifacts is a diff --git a/src/raes_adapters/cyborg/mapping/cage2-loss-disclosures.md b/src/raes_adapters/cyborg/mapping/cage2-loss-disclosures.md index 793defd..eac6cb0 100644 --- a/src/raes_adapters/cyborg/mapping/cage2-loss-disclosures.md +++ b/src/raes_adapters/cyborg/mapping/cage2-loss-disclosures.md @@ -4,7 +4,8 @@ These disclosures accompany the source selection `cage2-cyborg-2.1-source-26ce1c1`. Each heading is a stable ledger reference; the machine-readable tier line binds the precise ADR-069 claim tiers weakened. A disclosure permits a bounded claim with the stated weakness. It does not -prevent use of the selected backend and does not establish any equivalence tier. +erase a qualification limitation, prevent use of the selected backend, or +establish any equivalence tier. ## loss-scenario-user3-port-mismatch diff --git a/src/raes_adapters/cyborg/qualification.json b/src/raes_adapters/cyborg/qualification.json index e2f6421..0c10d38 100644 --- a/src/raes_adapters/cyborg/qualification.json +++ b/src/raes_adapters/cyborg/qualification.json @@ -315,7 +315,7 @@ "python": "3.12.3", "requirements_sha256": "a930508faee1766fb685ffa2fdd924121fdd0c056c92144850d20383951ba301", "normalized_environment_sha256": "56e2c189d6fd4b195584f34d93f66500b30eba8eedff6fca869ed61a8ab4079a", - "uv_lock_sha256": "7f815a2ed3d699432eb2bbfe5b3a2d6d09eae41be36ee794ee53cdb36d28b31b", + "uv_lock_sha256": "2ee83cdc249ba289b818c89f2c1c5ddd3a03f7587652ad17bfb8c7065847082e", "candidate_resolution": "passed", "canonical_lock_disposition": "The current single uv.lock intentionally excludes the unpublished candidate CybORG artifact; the resolved environment is qualification evidence, not a second lockfile." }, @@ -459,7 +459,7 @@ { "id": "wheel-omits-runtime-data", "evidence": "The unmodified wheel omits CybORG/version.txt and Scenario2 YAML resources, so importing CybORG fails after a normal wheel install.", - "disposition": "blocking for the published extra" + "disposition": "prevents declaring the ungoverned native artifact as an extra dependency; the adapter extra remains dependency-light" }, { "id": "legacy-gym-unmaintained", @@ -482,16 +482,24 @@ "disposition": "no outcome-reproducibility claim" } ], - "admissibility": { + "admission": { "decision": "admitted", + "authority": "maintainer-selection", "scope": "maintainer-selected-cyborg-backend", - "installation_limitations": [ + "limitations": [ "no-governed-public-patched-artifact", "declared-python-range-not-qualified", "declared-platform-range-not-qualified" ], - "extra_disposition": "leave the cyborg extra empty rather than publish a local path, editable checkout, install-time clone, or ungoverned wheel", - "qualification_effect": "installation, replay, and equivalence claims are limited; the maintainer-selected source-installed backend remains admitted for adapter use", + "claim_strength": { + "source_identity": "attested", + "protocol_configuration": "attested", + "execution_controls": "partial", + "run_evidence": "attestable", + "outcome_reproduction": "stochastic-bounded" + }, + "extra_disposition": "keep the cyborg extra dependency-light rather than publish a local path, editable checkout, install-time clone, or ungoverned wheel", + "interpretation": "The maintainer-selected profile is admitted for adapter realization. Artifact and platform coverage gaps bound installability and reproducibility claims; they do not veto the simulator.", "requalification_trigger": "an immutable governed artifact that contains the packaging-only fix and is qualified across the declared Python and platform support boundary" } } diff --git a/tests/test_cyberbattlesim_backend.py b/tests/test_cyberbattlesim_backend.py new file mode 100644 index 0000000..95d14e2 --- /dev/null +++ b/tests/test_cyberbattlesim_backend.py @@ -0,0 +1,1370 @@ +"""Contract-level behavior for the CyberBattleSim backend.""" + +from __future__ import annotations + +import hashlib +import subprocess +import sys +from dataclasses import dataclass, field, replace +from pathlib import Path +from types import SimpleNamespace + +import pytest +from raes_backend_protocols.backend_manifest import BackendManifest +from raes_backend_protocols.manifest import backend_manifest_v2_model +from raes_contracts.contracts import ( + ApparatusIdentityModel, + CleanStateRequirementModel, + CleanupObligationModel, + CleanupResourceBoundaryModel, + ExecutionRetryPolicyModel, + ExperimentCaptureSpecModel, + ExperimentDerivedMeasureModel, + ExperimentEvidenceRecordModel, + ParticipantActionResultModel, + ParticipantExposurePolicyModel, + ParticipantImplementationCapabilitiesModel, + ParticipantImplementationCompatibilityModel, + ParticipantImplementationManifestModel, + ParticipantImplementationSelectionModel, + TrialCleanupPlanModel, +) +from raes_contracts.participant_action_arguments import ParticipantValidatedActionSelection +from raes_contracts.participant_binding import ParticipantActionAdmissionRequest +from raes_contracts.participant_episode import ( + ParticipantEpisodeInitializeRequest, + ParticipantEpisodeRestartRequest, +) +from raes_contracts.planning import ( + ChangeAction, + EvaluationOp, + EvaluationPlan, + OrchestrationOp, + OrchestrationPlan, + ProvisioningPlan, + ProvisionOp, +) +from raes_contracts.runtime_state import OperationState, RuntimeSnapshot +from raes_processor.reference import ReferenceProcessor +from raes_runtime import RuntimeControlPlane +from raes_runtime.registry import RuntimeTarget + +from raes_adapters.base import run_conformance_probe +from raes_adapters.cyberbattlesim.backend import ( + ACTION_EVIDENCE_REF, + CYBERBATTLESIM_BACKEND_NAME, + CyberBattleSimDriver, + CyberBattleSimEvaluator, + CyberBattleSimParticipantRuntime, + CyberBattleSimProvisioner, + DriverCleanupReport, + DriverEvaluation, + DriverResetReport, + DriverStep, + create_cyberbattlesim_manifest, + create_cyberbattlesim_target, + execute_cyberbattlesim_cleanup, +) + +PARTICIPANT = "participant.behavior.attacker" +OBSERVATION_BOUNDARY = "participant.observation-boundary.attacker-view" +LOCAL_ACTION = "participant.action-contract.local-vulnerability" + + +@dataclass +class FakeDriver: + """Deterministic injected driver that retains deliberately private native state.""" + + steps: list[DriverStep] = field(default_factory=list) + evaluation: DriverEvaluation = field( + default_factory=lambda: DriverEvaluation( + step_count=0, + cumulative_reward=0.0, + terminated=False, + truncated=False, + terminal_cause=None, + ) + ) + construct_calls: int = 0 + reset_calls: list[int | None] = field(default_factory=list) + step_calls: list[str] = field(default_factory=list) + evaluate_calls: int = 0 + close_calls: int = 0 + closed: bool = False + fail_close: bool = False + fail_step: bool = False + native_observation: object = field( + default_factory=lambda: { + "_explored_network": "must-not-cross", + "credential_cache": "must-not-cross", + "action_mask": "must-not-cross", + "reward_vector": [14.0, -1.0], + } + ) + + def construct(self) -> None: + self.construct_calls += 1 + self.closed = False + + def reset(self, seed: int | None) -> DriverResetReport: + self.reset_calls.append(seed) + self.closed = False + return DriverResetReport( + operation_ref=f"driver.reset.{len(self.reset_calls)}", + applied_streams=("gym-environment", "gym-action-space"), + unbound_streams=("python-random", "numpy-global"), + ) + + def step(self, action_kind: str) -> DriverStep: + self.step_calls.append(action_kind) + if self.fail_step: + raise RuntimeError("private native step failure with secret-token") + if self.steps: + return self.steps.pop(0) + return DriverStep( + operation_ref=f"driver.step.{len(self.step_calls)}", + step_number=len(self.step_calls), + source_transition=True, + processed=True, + terminated=False, + truncated=False, + terminal_cause=None, + ) + + def evaluate(self) -> DriverEvaluation: + self.evaluate_calls += 1 + return replace( + self.evaluation, + projection_ref=(f"{self.evaluation.execution_ref}.evaluation.{self.evaluate_calls}"), + ) + + def close(self) -> DriverCleanupReport: + self.close_calls += 1 + if self.fail_close: + raise RuntimeError("private native close failure with secret-token") + already_closed = self.closed + self.closed = True + return DriverCleanupReport( + operation_ref=f"driver.close.{self.close_calls}", + closed=True, + verified=True, + already_closed=already_closed, + ) + + def verify_closed(self) -> bool: + return self.closed + + +def _participant_manifest() -> ParticipantImplementationManifestModel: + contracts = [ + "participant-implementation-manifest-v1", + "participant-implementation-provenance-v1", + "participant-episode-state-envelope-v1", + "participant-episode-history-event-stream-v1", + "participant-behavior-history-event-stream-v1", + "participant-decision-surface-v2", + ] + return ParticipantImplementationManifestModel( + identity=ApparatusIdentityModel( + name="cyberbattlesim-credential-cache-policy", + version="1.0.0", + ), + implementation_kind="policy", + supported_contract_versions=contracts, + compatibility=ParticipantImplementationCompatibilityModel( + participant_runtimes=["cyberbattlesim-participant-runtime"], + backends=[CYBERBATTLESIM_BACKEND_NAME], + ), + concept_bindings=[ + {"scope": "implementation_kind", "family": "apparatus-declarations"}, + { + "scope": "capabilities.supported_participant_contracts", + "family": "apparatus-declarations", + }, + { + "scope": "capabilities.supported_decision_surface_modes", + "family": "apparatus-declarations", + }, + { + "scope": "capabilities.tool_affordance_expectations", + "family": "tools-and-artifacts", + }, + { + "scope": "capabilities.exposure_policy_kinds", + "family": "provenance-and-evidence", + }, + ], + capabilities=ParticipantImplementationCapabilitiesModel( + supported_participant_contracts=[ + "participant-episode-state-envelope-v1", + "participant-episode-history-event-stream-v1", + "participant-behavior-history-event-stream-v1", + "participant-decision-surface-v2", + ], + supported_decision_surface_modes=["policy-directed"], + tool_affordance_expectations=["credential-store"], + exposure_policy_kinds=["hidden-truth", "observation-stream"], + ), + ) + + +def _action_request( + action_contract_address: str = LOCAL_ACTION, + *, + action_instance_id: str = "action-1", + disclose_action_evidence: bool = True, +) -> ParticipantActionAdmissionRequest: + manifest = _participant_manifest() + exposure_policy = ParticipantExposurePolicyModel( + policy_id="cyberbattlesim-attacker-view", + exposure_policy_kinds=["hidden-truth", "observation-stream"], + disclosed_refs=[OBSERVATION_BOUNDARY], + withheld_refs=["evidence.cyberbattlesim.hidden-world"], + visibility_scope_refs=[PARTICIPANT], + ) + selection = ParticipantImplementationSelectionModel( + participant_address=PARTICIPANT, + implementation_identity=manifest.identity, + manifest_ref="manifest.cyberbattlesim.credential-cache-policy", + manifest_digest="sha256:" + "1" * 64, + selected_decision_surface_mode="policy-directed", + participant_contract_versions=[ + "participant-episode-state-envelope-v1", + "participant-behavior-history-event-stream-v1", + "participant-decision-surface-v2", + ], + exposure_policy=exposure_policy, + ) + return ParticipantActionAdmissionRequest( + participant_address=PARTICIPANT, + action_contract_address=action_contract_address, + observation_boundary_address=OBSERVATION_BOUNDARY, + action_instance_id=action_instance_id, + implementation_manifest=manifest, + implementation_selection=selection, + visible_refs=(OBSERVATION_BOUNDARY,), + disclosed_refs=(OBSERVATION_BOUNDARY,), + observation_boundary_evidence_refs=( + (ACTION_EVIDENCE_REF,) if disclose_action_evidence else () + ), + validated_selection=ParticipantValidatedActionSelection( + action_contract_address=action_contract_address, + argument_shape_ref="argument-shape.cyberbattlesim.selected-action", + proposal_ref=f"proposal.{action_instance_id}", + normalized_arguments=(), + loss_disclosure_refs=("loss-abstracted-topology",), + ), + target_addresses=("provision.node.entry-client",), + requires_terminal_outcome=True, + ) + + +def _cleanup_plan(*, required: bool = True) -> TrialCleanupPlanModel: + boundary = CleanupResourceBoundaryModel( + boundary_id="cyberbattlesim-process", + resource_kind="in-process-simulator", + owner_ref=CYBERBATTLESIM_BACKEND_NAME, + resource_refs=["runtime.cyberbattlesim.selected-environment"], + ) + destroy = CleanupObligationModel( + obligation_id="destroy-environment", + boundary_refs=[boundary.boundary_id], + action_kind="destroy", + triggers=["success", "failure"], + requirement="required" if required else "best-effort", + idempotency="idempotent", + verification_probe_refs=["probe:cyberbattlesim-closed"], + timeout_seconds=5, + ) + verify = CleanupObligationModel( + obligation_id="verify-environment-closed", + boundary_refs=[boundary.boundary_id], + action_kind="verify", + triggers=["success", "failure"], + requirement="best-effort", + depends_on=[destroy.obligation_id], + idempotency="idempotent", + verification_probe_refs=["probe:cyberbattlesim-closed"], + timeout_seconds=5, + ) + return TrialCleanupPlanModel( + plan_id="cyberbattlesim-cleanup", + plan_entry_id="cyberbattlesim-cleanup-entry", + run_id="cyberbattlesim-run", + clean_state=CleanStateRequirementModel( + mode="verified-reset", + boundary_refs=[boundary.boundary_id], + verification_probe_refs=["probe:cyberbattlesim-closed"], + ), + resource_boundaries={boundary.boundary_id: boundary}, + cleanup_obligations={ + destroy.obligation_id: destroy, + verify.obligation_id: verify, + }, + retry_policy=ExecutionRetryPolicyModel( + max_attempts=1, + after_effect_policy="disallow", + ), + ) + + +def test_backend_import_is_dependency_light_and_lazy() -> None: + completed = subprocess.run( + [ + sys.executable, + "-I", + "-c", + ( + "import sys; import raes_adapters.cyberbattlesim.backend; " + "assert 'cyberbattle' not in sys.modules; " + "assert 'gymnasium' not in sys.modules" + ), + ], + check=False, + capture_output=True, + text=True, + ) + assert completed.returncode == 0, completed.stderr + + +def test_manifest_and_target_claim_exact_implemented_surfaces() -> None: + driver = FakeDriver() + manifest = create_cyberbattlesim_manifest() + target = create_cyberbattlesim_target(driver=driver, seed=20260729) + + assert isinstance(manifest, BackendManifest) + assert manifest.name == CYBERBATTLESIM_BACKEND_NAME + assert manifest.has_orchestrator + assert manifest.has_evaluator + assert manifest.has_participant_runtime + assert manifest.has_cleanup + assert not manifest.has_observation + assert not manifest.has_time + assert manifest.participant_runtime is not None + assert manifest.participant_runtime.supports_bounded_concurrency is False + assert manifest.participant_runtime.supported_interaction_features == frozenset( + {"interference"} + ) + assert manifest.participant_runtime.feature_support[0].support_level.value == ("disclosed_weak") + assert manifest.constraints["outcome_reproduction"] == "stochastic-bounded" + assert ( + manifest.constraints["source_identity"] + == "qualified-complete-runtime-artifact-roots-attested" + ) + assert manifest.provisioner.supported_content_types == frozenset() + assert manifest.evaluator.supported_evidence_channels == frozenset({"api_response"}) + assert manifest.evaluator.supported_time_domains == frozenset({"wall_clock"}) + assert { + "participant-control-occurrence-v1", + "participant-crossing-occurrence-v1", + "participant-lifecycle-event-v1", + } <= manifest.supported_contract_versions + assert backend_manifest_v2_model(manifest).identity.name == CYBERBATTLESIM_BACKEND_NAME + assert isinstance(target, RuntimeTarget) + assert target.name == CYBERBATTLESIM_BACKEND_NAME + + +def test_provisioner_reconciles_live_driver_and_orchestrator_never_steps_driver() -> None: + driver = FakeDriver() + provisioner = CyberBattleSimProvisioner(driver) + snapshot = RuntimeSnapshot() + plan = ProvisioningPlan( + operations=[ + ProvisionOp( + action=ChangeAction.CREATE, + address="provision.node.entry-client", + resource_type="node", + payload={"node_type": "vm", "os_family": "linux"}, + ) + ] + ) + + applied = provisioner.apply(plan, snapshot) + repeated = provisioner.apply(plan, applied.snapshot) + + assert applied.success + assert repeated.success + assert driver.construct_calls == 2 + assert applied.snapshot.entries["provision.node.entry-client"].status == "applied" + + target = create_cyberbattlesim_target(driver=driver) + orchestration = OrchestrationPlan( + operations=[ + OrchestrationOp( + action=ChangeAction.CREATE, + address="orchestration.workflow.episode", + resource_type="workflow", + payload={ + "name": "episode", + "execution_contract": {"start_step": "participant-action"}, + "result_contract": { + "state_schema_version": "workflow-step-state/v1", + "observable_steps": {"participant-action": {}}, + }, + }, + ) + ], + startup_order=["orchestration.workflow.episode"], + ) + started = target.orchestrator.start(orchestration, repeated.snapshot) + + assert started.success + assert ( + started.snapshot.orchestration_results["orchestration.workflow.episode"]["workflow_status"] + == "running" + ) + assert driver.step_calls == [] + + cleaned = execute_cyberbattlesim_cleanup( + _cleanup_plan(), + create_cyberbattlesim_manifest(), + driver, + receipt_id="cleanup-receipt-reuse", + execution_attempt_id="cleanup-attempt-reuse", + trial_outcome="succeeded", + ) + recreated = provisioner.apply(plan, started.snapshot) + + assert cleaned.cleanup_status == "succeeded" + assert recreated.success + assert driver.construct_calls == 3 + assert not driver.closed + + +def test_unchanged_operations_preserve_snapshots_and_component_state() -> None: + driver = FakeDriver( + evaluation=DriverEvaluation( + step_count=1, + cumulative_reward=7.0, + terminated=False, + truncated=False, + terminal_cause=None, + ) + ) + provisioner = CyberBattleSimProvisioner(driver) + created_provisioning = provisioner.apply( + ProvisioningPlan( + operations=[ + ProvisionOp( + action=ChangeAction.CREATE, + address="provision.node.entry-client", + resource_type="node", + payload={"node_type": "vm", "os_family": "linux"}, + ) + ] + ), + RuntimeSnapshot(), + ) + unchanged_provisioning = provisioner.apply( + ProvisioningPlan( + operations=[ + ProvisionOp( + action=ChangeAction.UNCHANGED, + address="provision.node.entry-client", + resource_type="node", + payload={"node_type": "vm", "os_family": "windows"}, + ) + ] + ), + created_provisioning.snapshot, + ) + assert unchanged_provisioning.snapshot is created_provisioning.snapshot + assert unchanged_provisioning.changed_addresses == [] + assert driver.construct_calls == 1 + + orchestrator = create_cyberbattlesim_target(driver=driver).orchestrator + created_orchestration = orchestrator.start( + OrchestrationPlan( + operations=[ + OrchestrationOp( + action=ChangeAction.CREATE, + address="orchestration.workflow.episode", + resource_type="workflow", + payload={ + "name": "episode", + "result_contract": { + "observable_steps": {"participant-action": {}}, + }, + }, + ) + ] + ), + created_provisioning.snapshot, + ) + orchestration_status = orchestrator.status() + unchanged_orchestration = orchestrator.start( + OrchestrationPlan( + operations=[ + OrchestrationOp( + action=ChangeAction.UNCHANGED, + address="orchestration.workflow.episode", + resource_type="workflow", + payload={"name": "replacement-must-not-apply"}, + ) + ] + ), + created_orchestration.snapshot, + ) + assert unchanged_orchestration.snapshot is created_orchestration.snapshot + assert unchanged_orchestration.changed_addresses == [] + assert orchestrator.status() == orchestration_status + + evaluator = CyberBattleSimEvaluator(driver) + created_evaluation = evaluator.start( + EvaluationPlan( + operations=[ + EvaluationOp( + action=ChangeAction.CREATE, + address="evaluation.objective.own-network", + resource_type="objective", + payload={ + "result_contract": { + "resource_type": "objective", + "supports_score": True, + "fixed_max_score": 5000, + } + }, + ) + ] + ), + created_orchestration.snapshot, + ) + evaluation_status = evaluator.status() + unchanged_evaluation = evaluator.start( + EvaluationPlan( + operations=[ + EvaluationOp( + action=ChangeAction.UNCHANGED, + address="evaluation.objective.own-network", + resource_type="objective", + payload={ + "result_contract": { + "resource_type": "objective", + "supports_score": True, + "fixed_max_score": 1, + } + }, + ) + ] + ), + created_evaluation.snapshot, + ) + assert unchanged_evaluation.snapshot is created_evaluation.snapshot + assert unchanged_evaluation.changed_addresses == [] + assert evaluator.status() == evaluation_status + assert driver.evaluate_calls == 1 + + +def test_each_admitted_action_has_one_driver_operation_and_typed_terminal_result() -> None: + driver = FakeDriver() + runtime = CyberBattleSimParticipantRuntime(driver, seed=20260729) + initialized = runtime.initialize( + ParticipantEpisodeInitializeRequest( + participant_address=PARTICIPANT, + episode_id="episode-1", + ), + RuntimeSnapshot(), + ) + + result = runtime.admit_action(_action_request(), initialized.snapshot) + + assert result.success + assert driver.step_calls == ["local-vulnerability"] + assert isinstance(result.action_result, ParticipantActionResultModel) + assert result.action_result.status == "succeeded" + assert result.action_result.participant_address == PARTICIPANT + assert result.action_result.episode_id == "episode-1" + assert result.action_result.action_instance_id == "action-1" + assert result.action_result.action_contract_address == LOCAL_ACTION + assert result.action_result.evidence_refs == [ACTION_EVIDENCE_REF] + assert result.action_result.effects[0].target_refs == [] + assert runtime.driver_operation_ref("action-1") == "driver.step.1" + observations = runtime.observations(PARTICIPANT) + assert len(observations) == 1 + assert observations[0].participant_address == PARTICIPANT + assert observations[0].hidden_state_refs == [] + assert observations[0].centralized_state_refs == [] + portable = str( + { + "result": result.action_result.model_dump(mode="json"), + "observation": observations[0].model_dump(mode="json"), + "snapshot": result.snapshot.participant_behavior_history, + } + ) + for forbidden in ( + "_explored_network", + "credential_cache", + "action_mask", + "reward_vector", + "must-not-cross", + "driver.step.1", + ): + assert forbidden not in portable + assert runtime.observations("participant.agent.defender") == () + + +def test_unsupported_action_is_rejected_before_native_mutation() -> None: + driver = FakeDriver() + runtime = CyberBattleSimParticipantRuntime(driver) + initialized = runtime.initialize( + ParticipantEpisodeInitializeRequest( + participant_address=PARTICIPANT, + episode_id="episode-unsupported", + ), + RuntimeSnapshot(), + ) + + result = runtime.admit_action( + _action_request( + "participant.action-contract.scan-and-reimage", + action_instance_id="action-unsupported", + ), + initialized.snapshot, + ) + + assert not result.success + assert driver.step_calls == [] + assert result.action_result is not None + assert result.action_result.status == "rejected" + assert result.action_result.failure_class == "unsupported_action" + + +def test_action_boundary_withholds_unrequested_evidence_and_native_failures() -> None: + driver = FakeDriver() + runtime = CyberBattleSimParticipantRuntime(driver) + initialized = runtime.initialize( + ParticipantEpisodeInitializeRequest( + participant_address=PARTICIPANT, + episode_id="episode-boundary", + ), + RuntimeSnapshot(), + ) + + undisclosed = runtime.admit_action( + _action_request(disclose_action_evidence=False), + initialized.snapshot, + ) + + assert undisclosed.success + assert undisclosed.action_result is not None + assert undisclosed.action_result.evidence_refs == [] + assert undisclosed.action_result.effects[0].evidence_refs == [] + assert runtime.observations(PARTICIPANT)[0].evidence_refs == [] + + failing_runtime = CyberBattleSimParticipantRuntime(FakeDriver(fail_step=True)) + failing_initialized = failing_runtime.initialize( + ParticipantEpisodeInitializeRequest( + participant_address=PARTICIPANT, + episode_id="episode-failure", + ), + RuntimeSnapshot(), + ) + failed = failing_runtime.admit_action( + _action_request(action_instance_id="action-failure"), + failing_initialized.snapshot, + ) + portable = str( + { + "diagnostics": [diagnostic.message for diagnostic in failed.diagnostics], + "action_result": failed.action_result.model_dump(mode="json") + if failed.action_result is not None + else None, + } + ) + + assert not failed.success + assert failed.action_result is not None + assert failed.action_result.status == "failed" + assert "secret-token" not in portable + + +def test_unsupported_provisioning_is_rejected_before_construction() -> None: + driver = FakeDriver() + plan = ProvisioningPlan( + operations=[ + ProvisionOp( + action=ChangeAction.CREATE, + address="provision.persistent-volume.unsupported", + resource_type="persistent-volume", + payload={}, + ) + ] + ) + + result = CyberBattleSimProvisioner(driver).apply(plan, RuntimeSnapshot()) + + assert not result.success + assert driver.construct_calls == 0 + assert result.diagnostics[0].code == "cyberbattlesim.provisioning.unsupported-resource" + + +def test_evaluator_reads_distinct_facts_without_advancing_or_leaking_to_participant() -> None: + driver = FakeDriver( + evaluation=DriverEvaluation( + step_count=1, + cumulative_reward=14.0, + terminated=False, + truncated=False, + terminal_cause=None, + ) + ) + runtime = CyberBattleSimParticipantRuntime(driver) + initialized = runtime.initialize( + ParticipantEpisodeInitializeRequest( + participant_address=PARTICIPANT, + episode_id="episode-evaluation", + ), + RuntimeSnapshot(), + ) + action = runtime.admit_action(_action_request(), initialized.snapshot) + evaluator = CyberBattleSimEvaluator(driver) + evaluation_plan = EvaluationPlan( + operations=[ + EvaluationOp( + action=ChangeAction.CREATE, + address="evaluation.objective.own-network", + resource_type="objective", + payload={ + "result_contract": { + "resource_type": "objective", + "supports_score": True, + "fixed_max_score": 5000, + } + }, + ) + ], + startup_order=["evaluation.objective.own-network"], + ) + + evaluated = evaluator.start(evaluation_plan, action.snapshot) + + assert evaluated.success + assert driver.step_calls == ["local-vulnerability"] + assert driver.evaluate_calls == 1 + assert evaluator.results()["evaluation.objective.own-network"]["score"] == 14.0 + assert "14.0" not in str(action.action_result.model_dump(mode="json")) + assert ACTION_EVIDENCE_REF not in str(evaluator.results()) + evidence_records = evaluator.evidence_records() + derived_measures = evaluator.derived_measures() + capture_spec = evaluator.capture_spec() + assert isinstance(capture_spec, ExperimentCaptureSpecModel) + assert len(evidence_records) == len(derived_measures) == 1 + assert isinstance(evidence_records[0], ExperimentEvidenceRecordModel) + assert isinstance(derived_measures[0], ExperimentDerivedMeasureModel) + assert evidence_records[0].capture_spec_ref.ref_id == capture_spec.capture_spec_id + assert derived_measures[0].value == 14.0 + assert derived_measures[0].limitations + assert "deterministic replay" in derived_measures[0].limitations[0] + raw_content = evidence_records[0].raw_content + assert raw_content.payload_summary is not None + assert raw_content.content_checksum is not None + assert raw_content.content_checksum.algorithm == "sha256" + assert ( + raw_content.content_checksum.value + == hashlib.sha256(raw_content.payload_summary.encode("utf-8")).hexdigest() + ) + portable_evaluation = str( + { + "evidence": evidence_records[0].model_dump(mode="json"), + "measure": derived_measures[0].model_dump(mode="json"), + } + ) + for forbidden in ( + "_explored_network", + "credential_cache", + "action_mask", + "reward_vector", + "must-not-cross", + ): + assert forbidden not in portable_evaluation + + first_evidence = evidence_records[0] + first_measure = derived_measures[0] + first_capture_spec = capture_spec + driver.evaluation = replace(driver.evaluation, cumulative_reward=21.0) + reevaluated = evaluator.start(evaluation_plan, evaluated.snapshot) + second_evidence = evaluator.evidence_records()[0] + second_measure = evaluator.derived_measures()[0] + second_capture_spec = evaluator.capture_spec() + + assert reevaluated.success + assert isinstance(second_capture_spec, ExperimentCaptureSpecModel) + assert first_evidence.evidence_record_id != second_evidence.evidence_record_id + assert first_measure.derived_measure_id != second_measure.derived_measure_id + assert first_capture_spec.capture_spec_id != second_capture_spec.capture_spec_id + assert ( + first_capture_spec.capture_windows[0].window_id + != second_capture_spec.capture_windows[0].window_id + ) + assert second_measure.value == 21.0 + assert second_evidence.run_ref.ref_id == driver.evaluation.execution_ref + assert ( + reevaluated.snapshot.evaluation_results["evaluation.objective.own-network"]["run_id"] + == second_evidence.run_ref.ref_id + ) + assert second_evidence.capture_spec_ref.ref_id == second_capture_spec.capture_spec_id + assert second_evidence.capture_window_ref == second_capture_spec.capture_windows[0].window_id + assert second_measure.source_evidence_refs[0].ref_id == second_evidence.evidence_record_id + assert reevaluated.snapshot.evaluation_results["evaluation.objective.own-network"][ + "evidence_refs" + ] == [second_evidence.evidence_record_id] + + +def test_source_termination_closes_episode_and_restart_resets_same_seed() -> None: + driver = FakeDriver( + steps=[ + DriverStep( + operation_ref="driver.step.terminal", + step_number=1, + source_transition=True, + processed=True, + terminated=True, + truncated=False, + terminal_cause="source-terminated", + ) + ] + ) + runtime = CyberBattleSimParticipantRuntime(driver, seed=20260729) + initialized = runtime.initialize( + ParticipantEpisodeInitializeRequest( + participant_address=PARTICIPANT, + episode_id="episode-terminal", + ), + RuntimeSnapshot(), + ) + terminal = runtime.admit_action(_action_request(), initialized.snapshot) + + assert terminal.success + assert terminal.snapshot.participant_episode_results[PARTICIPANT]["status"] == "terminated" + restarted = runtime.restart( + ParticipantEpisodeRestartRequest( + participant_address=PARTICIPANT, + episode_id="episode-restarted", + ), + terminal.snapshot, + ) + assert restarted.success + assert driver.reset_calls == [20260729, 20260729] + assert restarted.snapshot.participant_episode_results[PARTICIPANT]["previous_episode_id"] == ( + "episode-terminal" + ) + + +def test_cleanup_is_idempotent_verified_and_preserves_bounded_failure() -> None: + driver = FakeDriver() + manifest = create_cyberbattlesim_manifest() + plan = _cleanup_plan() + + first = execute_cyberbattlesim_cleanup( + plan, + manifest, + driver, + receipt_id="cleanup-receipt-1", + execution_attempt_id="cleanup-attempt-1", + trial_outcome="succeeded", + ) + second = execute_cyberbattlesim_cleanup( + plan, + manifest, + driver, + receipt_id="cleanup-receipt-2", + execution_attempt_id="cleanup-attempt-2", + trial_outcome="succeeded", + ) + + assert first.cleanup_status == second.cleanup_status == "succeeded" + assert driver.close_calls == 2 + assert driver.closed + assert all(result.status == "succeeded" for result in second.obligation_results.values()) + + failing_driver = FakeDriver(fail_close=True) + failed = execute_cyberbattlesim_cleanup( + _cleanup_plan(required=False), + manifest, + failing_driver, + receipt_id="cleanup-receipt-failed", + execution_attempt_id="cleanup-attempt-failed", + trial_outcome="failed", + ) + assert failed.cleanup_status == "failed" + assert failed.obligation_results["destroy-environment"].status == "failed" + assert "secret-token" not in str(failed.model_dump(mode="json")) + + +def test_selected_scenario_realizes_and_runs_across_applicable_surfaces() -> None: + scenario = ( + Path(__file__).parents[1] + / "src" + / "raes_adapters" + / "cyberbattlesim" + / "scenario" + / "cyberbattle-chain.sdl.yaml" + ) + driver = FakeDriver( + evaluation=DriverEvaluation( + step_count=1, + cumulative_reward=7.0, + terminated=False, + truncated=False, + terminal_cause=None, + ) + ) + target = create_cyberbattlesim_target(driver=driver, seed=20260729) + realized = ReferenceProcessor.realize( + scenario, + target.manifest, + parameters={"chain_size": 10}, + target_name=CYBERBATTLESIM_BACKEND_NAME, + ) + + assert realized.is_valid + assert realized.diagnostics == () + assert len(realized.execution_plan.provisioning.operations) == 7 + assert len(realized.execution_plan.evaluation.operations) == 6 + + control_plane = RuntimeControlPlane(target) + provisioning_receipt = control_plane.submit_provisioning( + realized.execution_plan.provisioning, + idempotency_key="selected-scenario-provisioning", + request_fingerprint="selected-scenario-provisioning-v1", + ) + initialize_receipt = control_plane.initialize_participant_episode( + PARTICIPANT, + episode_id="episode-selected-scenario", + idempotency_key="selected-scenario-initialize", + request_fingerprint="selected-scenario-initialize-v1", + ) + participant_behavior = realized.runtime_model.participant_behaviors[PARTICIPANT] + action_receipt = control_plane.admit_participant_action( + participant_behavior, + _action_request(), + idempotency_key="selected-scenario-action", + request_fingerprint="selected-scenario-action-v1", + ) + repeated_action_receipt = control_plane.admit_participant_action( + participant_behavior, + _action_request(), + idempotency_key="selected-scenario-action", + request_fingerprint="selected-scenario-action-v1", + ) + evaluation_receipt = control_plane.submit_evaluation( + realized.execution_plan.evaluation, + idempotency_key="selected-scenario-evaluation", + request_fingerprint="selected-scenario-evaluation-v1", + ) + orchestration_receipt = control_plane.submit_orchestration( + realized.execution_plan.orchestration, + idempotency_key="selected-scenario-orchestration", + request_fingerprint="selected-scenario-orchestration-v1", + ) + + receipts = ( + provisioning_receipt, + initialize_receipt, + action_receipt, + evaluation_receipt, + orchestration_receipt, + ) + assert all(receipt.accepted for receipt in receipts) + assert all( + control_plane.get_operation(receipt.operation_id).state == OperationState.SUCCEEDED + for receipt in receipts + ) + assert repeated_action_receipt.operation_id == action_receipt.operation_id + truth_results = control_plane.snapshot.proposition_truth_results + assert set(truth_results) == { + "evaluation.assertion.ownership-postcondition", + "evaluation.assertion.sla-invariant", + } + assert {result["proposition_outcome"] for result in truth_results.values()} == {"unknown"} + objective_results = control_plane.snapshot.evaluation_results + assert { + objective_results["evaluation.objective.own-network"]["status"], + objective_results["evaluation.objective.maintain-availability"]["status"], + } == {"running"} + assert all(result["passed"] is None for result in objective_results.values()) + assert driver.step_calls == ["local-vulnerability"] + assert driver.evaluate_calls == 1 + + cleaned = execute_cyberbattlesim_cleanup( + _cleanup_plan(), + target.manifest, + driver, + receipt_id="cleanup-selected-scenario", + execution_attempt_id="cleanup-selected-scenario-attempt", + trial_outcome="succeeded", + ) + assert cleaned.cleanup_status == "succeeded" + assert driver.closed + + +def test_target_passes_canonical_backend_conformance_probe() -> None: + report = run_conformance_probe( + create_cyberbattlesim_target(driver=FakeDriver()), + ) + + assert report.passed + assert report.unsupported_contract_gaps == () + assert report.unsupported_capability_gaps == () + assert all(case.passed for case in report.cases) + + +def test_live_driver_is_lazy_seed_bounded_and_performs_exactly_one_native_step( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + class NumericRuntime: + int32 = "int32" + + @staticmethod + def argwhere(mask: list[tuple[int, int]]) -> list[tuple[int, int]]: + return mask + + @staticmethod + def asarray( + value: tuple[int, int], + *, + dtype: object, + ) -> tuple[int, int]: + assert dtype == NumericRuntime.int32 + return value + + class NativeEnvironment: + def __init__(self) -> None: + self.action_space = SimpleNamespace(seed=self.seed_action_space) + self.action_space_seeds: list[int] = [] + self.reset_seeds: list[int | None] = [] + self.native_steps: list[dict[str, object]] = [] + self.close_calls = 0 + + def seed_action_space(self, seed: int) -> None: + self.action_space_seeds.append(seed) + + def reset(self, *, seed: int | None) -> tuple[dict[str, object], dict[str, object]]: + self.reset_seeds.append(seed) + return ( + { + "action_mask": { + "connect": [(0, 0)], + "local_vulnerability": [(0, 1)], + "remote_vulnerability": [(0, 0)], + }, + "credential_cache": ["native-secret"], + }, + {"native-info": "must-not-cross"}, + ) + + def step( + self, + action: dict[str, object], + ) -> tuple[dict[str, object], float, bool, bool, dict[str, object]]: + self.native_steps.append(action) + observation, _ = self.reset(seed=None) + return observation, -3.5, False, False, {"native-info": "must-not-cross"} + + def close(self) -> None: + self.close_calls += 1 + + environment = NativeEnvironment() + selected_configuration: dict[str, object] = {} + source_content = { + "cyberbattle/__init__.py": b"selected cyberbattle package", + "cyberbattle/_env/cyberbattle_env.py": b"selected cyberbattle environment", + "cyberbattle/_env/defender.py": b"selected defender", + "cyberbattle/_env/cyberbattle_chain.py": b"selected chain registration", + "cyberbattle/samples/chainpattern/chainpattern.py": b"selected chain generator", + "cyberbattle/simulation/model.py": b"selected transitive runtime module", + } + source_root = tmp_path / "selected-distribution" + for source_path, content in source_content.items(): + installed_source = source_root / source_path + installed_source.parent.mkdir(parents=True, exist_ok=True) + installed_source.write_bytes(content) + + class SelectedDistribution: + def __init__( + self, + version: str, + root: Path, + files: tuple[str, ...], + ) -> None: + self.version = version + self.root = root + self.files = files + self.direct_url_text: str | None = None + + def locate_file(self, path: str) -> Path: + return self.root / path + + def read_text(self, filename: str) -> str | None: + if filename == "direct_url.json": + return self.direct_url_text + return None + + dependency_roots = { + "gymnasium": tmp_path / "gymnasium-distribution", + "numpy": tmp_path / "numpy-distribution", + } + for package_name, dependency_root in dependency_roots.items(): + package_init = dependency_root / package_name / "__init__.py" + package_init.parent.mkdir(parents=True) + package_init.write_bytes(f"selected {package_name}".encode()) + distributions = { + "cyberbattlesim": SelectedDistribution( + "0.1.0", + source_root, + tuple(source_content), + ), + "gymnasium": SelectedDistribution( + "0.29.1", + dependency_roots["gymnasium"], + ("gymnasium/__init__.py",), + ), + "numpy": SelectedDistribution( + "1.26.4", + dependency_roots["numpy"], + ("numpy/__init__.py",), + ), + } + module_origins = { + "cyberbattle": source_root / "cyberbattle/__init__.py", + "cyberbattle._env.cyberbattle_env": (source_root / "cyberbattle/_env/cyberbattle_env.py"), + "cyberbattle._env.defender": source_root / "cyberbattle/_env/defender.py", + "gymnasium": dependency_roots["gymnasium"] / "gymnasium/__init__.py", + "numpy": dependency_roots["numpy"] / "numpy/__init__.py", + } + + def make_environment(name: str, **kwargs: object) -> SimpleNamespace: + selected_configuration["name"] = name + selected_configuration.update(kwargs) + return SimpleNamespace(unwrapped=environment) + + class Configuration: + def __init__(self, **kwargs: object) -> None: + self.values = kwargs + + modules = { + "cyberbattle": SimpleNamespace(), + "gymnasium": SimpleNamespace(make=make_environment), + "numpy": NumericRuntime, + "cyberbattle._env.cyberbattle_env": SimpleNamespace( + AttackerGoal=Configuration, + DefenderConstraint=Configuration, + ), + "cyberbattle._env.defender": SimpleNamespace( + ScanAndReimageCompromisedMachines=Configuration, + ), + } + imported_modules: list[str] = [] + + def import_selected_module(name: str) -> object: + imported_modules.append(name) + return modules[name] + + runtime_tree_digest = hashlib.sha256() + for source_path, content in sorted(source_content.items()): + runtime_tree_digest.update(source_path.encode()) + runtime_tree_digest.update(b"\0") + runtime_tree_digest.update(hashlib.sha256(content).hexdigest().encode()) + runtime_tree_digest.update(b"\n") + + def runtime_root_record( + root_path: str, + content_by_path: dict[str, bytes], + ) -> dict[str, object]: + digest = hashlib.sha256() + selected_paths = sorted( + path + for path in content_by_path + if path == root_path or path.startswith(f"{root_path}/") + ) + for selected_path in selected_paths: + digest.update(selected_path.encode()) + digest.update(b"\0") + digest.update(hashlib.sha256(content_by_path[selected_path]).hexdigest().encode()) + digest.update(b"\n") + return { + "path": root_path, + "file_count": len(selected_paths), + "sha256": digest.hexdigest(), + } + + dependency_content = { + package_name: {f"{package_name}/__init__.py": f"selected {package_name}".encode()} + for package_name in dependency_roots + } + monkeypatch.setattr( + "raes_adapters.cyberbattlesim.backend.driver.distribution", + lambda name: distributions[name], + ) + monkeypatch.setattr( + "raes_adapters.cyberbattlesim.backend.driver.find_spec", + lambda name: SimpleNamespace(origin=str(module_origins[name])), + ) + selected_qualification = { + "source": { + "package": "cyberbattlesim", + "version": "0.1.0", + }, + "source_files": [ + { + "path": path, + "sha256": hashlib.sha256(content).hexdigest(), + } + for path, content in source_content.items() + ], + "runtime_source_tree": { + "file_count": len(source_content), + "sha256": runtime_tree_digest.hexdigest(), + }, + "runtime_artifacts": [ + { + "name": "cyberbattlesim", + "version": "0.1.0", + "artifact": { + "filename": "cyberbattlesim-0.1.0-py3-none-any.whl", + "sha256": "1" * 64, + "require_direct_archive_sha256": False, + "runtime_identity": "complete-root-tree", + }, + "roots": [runtime_root_record("cyberbattle", source_content)], + }, + { + "name": "gymnasium", + "version": "0.29.1", + "artifact": { + "filename": "gymnasium-0.29.1-py3-none-any.whl", + "sha256": "2" * 64, + "require_direct_archive_sha256": True, + "runtime_identity": "complete-root-tree", + }, + "roots": [ + runtime_root_record( + "gymnasium", + dependency_content["gymnasium"], + ) + ], + }, + { + "name": "numpy", + "version": "1.26.4", + "artifact": { + "filename": "numpy-1.26.4-cp312-manylinux.whl", + "sha256": "3" * 64, + "require_direct_archive_sha256": True, + "runtime_identity": "complete-root-tree", + }, + "roots": [ + runtime_root_record( + "numpy", + dependency_content["numpy"], + ) + ], + }, + ], + "dependencies": [ + {"name": "cyberbattlesim", "version": "0.1.0"}, + {"name": "gymnasium", "version": "0.29.1"}, + {"name": "numpy", "version": "1.26.4"}, + ], + "protocol": { + "selection": { + "scenario": { + "gym_id": "CyberBattleChain-v0", + "size": 10, + }, + "termination": { + "attacker_own_atleast": 0, + "attacker_own_atleast_percent": 1.0, + "defender_maintain_sla": 0.8, + "evaluator_cutoff_steps": 1, + }, + "defender": { + "probability": 0.6, + "scan_capacity": 2, + "scan_frequency": 5, + }, + } + }, + } + monkeypatch.setattr( + "raes_adapters.cyberbattlesim.backend.driver.load_qualification", + lambda: selected_qualification, + ) + monkeypatch.setattr( + "raes_adapters.cyberbattlesim.backend.driver.importlib.import_module", + import_selected_module, + ) + driver = CyberBattleSimDriver() + + reset = driver.reset(20260729) + step = driver.step("local-vulnerability") + evaluation = driver.evaluate() + first_close = driver.close() + second_close = driver.close() + + assert selected_configuration["name"] == "CyberBattleChain-v0" + assert selected_configuration["size"] == 10 + assert reset.applied_streams == ("gym-environment", "gym-action-space") + assert reset.unbound_streams == ("python-random", "numpy-global") + assert environment.reset_seeds[0] == 20260729 + assert environment.action_space_seeds == [20260729] + assert len(environment.native_steps) == 1 + assert tuple(environment.native_steps[0]) == ("local_vulnerability",) + assert step.source_transition + assert step.processed + assert step.step_number == 1 + assert step.terminal_cause == "evaluator-cutoff" + assert evaluation.cumulative_reward == -3.5 + assert evaluation.terminal_cause == "evaluator-cutoff" + assert evaluation.execution_ref == reset.operation_ref + assert evaluation.projection_ref == f"{reset.operation_ref}.evaluation.1" + assert first_close.verified + assert not first_close.already_closed + assert second_close.verified + assert second_close.already_closed + assert environment.close_calls == 1 + + imported_modules.clear() + transitive_source = source_root / "cyberbattle/simulation/model.py" + transitive_source.write_bytes(b"tampered transitive runtime module") + tampered_driver = CyberBattleSimDriver() + with pytest.raises( + RuntimeError, + match="selected simulator dependency identity could not be verified", + ): + tampered_driver.construct() + assert imported_modules == [] + + transitive_source.write_bytes(source_content["cyberbattle/simulation/model.py"]) + injected_extension = source_root / "cyberbattle/_env/cyberbattle_env.so" + injected_extension.write_bytes(b"unqualified native extension") + injected_driver = CyberBattleSimDriver() + with pytest.raises( + RuntimeError, + match="selected simulator dependency identity could not be verified", + ): + injected_driver.construct() + assert imported_modules == [] + + injected_extension.unlink() + distributions[ + "cyberbattlesim" + ].direct_url_text = '{"dir_info":{"editable":true},"url":"file:///unqualified/source"}' + editable_driver = CyberBattleSimDriver() + with pytest.raises( + RuntimeError, + match="selected simulator dependency identity could not be verified", + ): + editable_driver.construct() + assert imported_modules == [] + + distributions["cyberbattlesim"].direct_url_text = None + module_origins["cyberbattle"] = tmp_path / "unselected/cyberbattle/__init__.py" + unselected_driver = CyberBattleSimDriver() + with pytest.raises( + RuntimeError, + match="selected simulator module origin could not be verified", + ): + unselected_driver.construct() + assert imported_modules == [] diff --git a/tests/test_cyberbattlesim_qualification.py b/tests/test_cyberbattlesim_qualification.py index e4b7602..143082f 100644 --- a/tests/test_cyberbattlesim_qualification.py +++ b/tests/test_cyberbattlesim_qualification.py @@ -59,6 +59,45 @@ def test_clean_install_and_source_native_smoke_are_bounded_and_complete() -> Non assert "credential_cache" not in smoke +def test_runtime_artifact_attestation_covers_complete_import_roots() -> None: + record = cyberbattlesim.load_qualification() + runtime_tree = record["runtime_source_tree"] + + assert runtime_tree == { + "root": "cyberbattle", + "include": "all regular files; symlinks and unlisted files are rejected", + "file_count": 48, + "canonicalization": ("Sorted POSIX path, NUL, lowercase SHA-256 of raw file bytes, LF."), + "sha256": "aa97624b86b2f5c9ad80ea52a345d5fd6f8fa89cad31070c5e4748c55510ba02", + } + source_files = {entry["path"]: entry["sha256"] for entry in record["source_files"]} + assert source_files["cyberbattle/__init__.py"] == ( + "146a6297d4361b7cd550e293e050c98b6795c4fc5416a1e72f62a8dc142668c4" + ) + artifacts = {artifact["name"]: artifact for artifact in record["runtime_artifacts"]} + assert set(artifacts) == {"cyberbattlesim", "gymnasium", "numpy"} + assert artifacts["cyberbattlesim"]["roots"] == [ + { + "path": "cyberbattle", + "file_count": 48, + "sha256": ("aa97624b86b2f5c9ad80ea52a345d5fd6f8fa89cad31070c5e4748c55510ba02"), + } + ] + assert artifacts["gymnasium"]["artifact"]["sha256"] == ( + "61c3384b5575985bb7f85e43213bcb40f36fcdff388cae6bc229304c71f2843e" + ) + assert artifacts["cyberbattlesim"]["artifact"]["require_direct_archive_sha256"] is False + assert artifacts["gymnasium"]["artifact"]["require_direct_archive_sha256"] is True + assert artifacts["numpy"]["artifact"]["require_direct_archive_sha256"] is True + assert {artifact["artifact"]["runtime_identity"] for artifact in artifacts.values()} == { + "complete-root-tree" + } + assert {root["path"] for root in artifacts["numpy"]["roots"]} == { + "numpy", + "numpy.libs", + } + + def test_protocol_fixes_every_identity_and_discloses_random_streams() -> None: record = cyberbattlesim.load_qualification() selection = record["protocol"]["selection"] @@ -114,16 +153,24 @@ def test_legal_maintenance_and_patch_dispositions_are_explicit() -> None: ) -def test_negative_qualification_does_not_publish_a_broken_extra() -> None: +def test_qualification_admits_selected_backend_and_bounds_claim_strength() -> None: record = cyberbattlesim.load_qualification() project = tomllib.loads((REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8")) extras = project["project"]["optional-dependencies"] - assert record["admissibility"]["decision"] == "not-admissible" - assert set(record["admissibility"]["blockers"]) == { + assert record["admission"]["decision"] == "admitted" + assert record["admission"]["authority"] == "maintainer-selection" + assert set(record["admission"]["limitations"]) == { "no-index-or-release-artifact", "incomplete-random-stream-binding", "unresolved-upstream-benchmark-defects", } - assert "cyberbattlesim" not in extras + assert record["admission"]["claim_strength"] == { + "source_identity": "attested", + "protocol_configuration": "attested", + "execution_controls": "partial", + "run_evidence": "attestable", + "outcome_reproduction": "stochastic-bounded", + } + assert extras["cyberbattlesim"] == [] assert extras["cyborg"] == [] diff --git a/tests/test_cyborg_provisioner.py b/tests/test_cyborg_provisioner.py index 73b102a..80f4eff 100644 --- a/tests/test_cyborg_provisioner.py +++ b/tests/test_cyborg_provisioner.py @@ -739,7 +739,7 @@ def test_failed_replacement_cleanup_compensates_candidate_without_advancing_snap def test_selected_cyborg_source_is_admitted_and_the_installation_limit_is_disclosed() -> None: record = load_qualification() - assert record["admissibility"]["decision"] == "admitted" - assert record["admissibility"]["scope"] == "maintainer-selected-cyborg-backend" + assert record["admission"]["decision"] == "admitted" + assert record["admission"]["scope"] == "maintainer-selected-cyborg-backend" assert record["packaging"]["patched_wheel"]["published"] is False assert CAGE2_SOURCE_26CE1C1.qualification_profile_id == record["profile_id"] diff --git a/tests/test_cyborg_qualification.py b/tests/test_cyborg_qualification.py index 3c89f4b..5f4b081 100644 --- a/tests/test_cyborg_qualification.py +++ b/tests/test_cyborg_qualification.py @@ -188,7 +188,7 @@ def test_selection_dependencies_and_stochastic_sources_are_explicit() -> None: "56e2c189d6fd4b195584f34d93f66500b30eba8eedff6fca869ed61a8ab4079a" ) assert record["dependency_resolution"]["uv_lock_sha256"] == ( - "7f815a2ed3d699432eb2bbfe5b3a2d6d09eae41be36ee794ee53cdb36d28b31b" + "2ee83cdc249ba289b818c89f2c1c5ddd3a03f7587652ad17bfb8c7065847082e" ) assert record["dependencies"] assert all( @@ -202,7 +202,7 @@ def test_selection_dependencies_and_stochastic_sources_are_explicit() -> None: } -def test_legal_defect_and_fail_closed_packaging_decisions_are_explicit() -> None: +def test_legal_defect_and_claim_bounded_packaging_decisions_are_explicit() -> None: record = cyborg.load_qualification() project = tomllib.loads((REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8")) extras = project["project"]["optional-dependencies"] @@ -226,12 +226,20 @@ def test_legal_defect_and_fail_closed_packaging_decisions_are_explicit() -> None "scenario-user3-port-mismatch", "evaluation-seed-not-bound", } - assert record["admissibility"]["decision"] == "admitted" - assert record["admissibility"]["scope"] == "maintainer-selected-cyborg-backend" - assert set(record["admissibility"]["installation_limitations"]) == { + assert record["admission"]["decision"] == "admitted" + assert record["admission"]["authority"] == "maintainer-selection" + assert set(record["admission"]["limitations"]) == { "no-governed-public-patched-artifact", "declared-python-range-not-qualified", "declared-platform-range-not-qualified", } + assert record["admission"]["scope"] == "maintainer-selected-cyborg-backend" + assert record["admission"]["claim_strength"] == { + "source_identity": "attested", + "protocol_configuration": "attested", + "execution_controls": "partial", + "run_evidence": "attestable", + "outcome_reproduction": "stochastic-bounded", + } assert extras["cyborg"] == [] assert "CybORG" not in project["project"]["dependencies"] diff --git a/tests/test_cyborg_source_ledger.py b/tests/test_cyborg_source_ledger.py index 7fc361b..e0ff434 100644 --- a/tests/test_cyborg_source_ledger.py +++ b/tests/test_cyborg_source_ledger.py @@ -228,7 +228,7 @@ def test_qualification_json_pointers_are_machine_resolvable() -> None: qualification = sl.load_qualification() assert sl.resolve_qualification_ref("qualification.json#/legal", qualification) assert sl.resolve_qualification_ref( - "qualification.json#/admissibility/decision", + "qualification.json#/admission/decision", qualification, ) assert not sl.resolve_qualification_ref("qualification.json#/not-there", qualification) diff --git a/uv.lock b/uv.lock index ff88a3f..faf444c 100644 --- a/uv.lock +++ b/uv.lock @@ -1422,7 +1422,7 @@ docs = [ [package.metadata] requires-dist = [{ name = "raes", specifier = "==2.0.0" }] -provides-extras = ["cyborg"] +provides-extras = ["cyberbattlesim", "cyborg"] [package.metadata.requires-dev] dev = [