diff --git a/README.md b/README.md index cd7aab4..984708c 100644 --- a/README.md +++ b/README.md @@ -27,17 +27,24 @@ and never define the shared semantic boundary (ADR-002). ## Install ```bash -pip install raes-adapters # shared base plumbing -pip install raes-adapters[cyborg] # + the CybORG backend +pip install raes-adapters # shared base plumbing + qualification evidence ``` -There is intentionally no `cyberbattlesim` extra yet. The +The `cyborg` extra key is reserved but intentionally empty. Issue +[#12](https://github.com/RAESystem/adapters/issues/12) qualified the official +CAGE Challenge 2 source and a packaging-only fix, but the upstream wheel omits +the version and Scenario2 runtime data. The fixed wheel passed a clean Python +3.12 smoke locally, but it is not a governed public artifact and the +distribution's declared Python/platform range is not yet qualified. Advertising +an editable checkout, install-time clone, or unpublished wheel as the extra +would hide those blockers. + +There is also intentionally no `cyberbattlesim` extra. The [qualification record](src/raes_adapters/cyberbattlesim/qualification.json) found Microsoft's source legally usable and runnable, but not currently admissible: no official index/release artifact exists, the public evaluator does not bind every random stream, and material upstream benchmark findings -remain open. Publishing an empty, direct-URL, or knowingly uninstallable extra -would hide those blockers. +remain open. ## One distribution, optional simulator extras @@ -63,7 +70,7 @@ raes-adapters/ scenario/ # authored RAES SDL scenario (portable topology/objective truth) experiment/ # published experiment contracts (reward/evaluator/stochastic intent) mapping/ # pinned CyberBattleSim → RAES source ledger + loss disclosures - cyborg/ # CybORG backend module (optional `cyborg` extra; ADR-069 §3) + cyborg/ # CybORG qualification, patch evidence, and future backend mapping/ # pinned CAGE-2 → RAES source ledger (REP-003) profiles/ # conformance profile overrides tests/ # pytest suite for the distribution @@ -116,6 +123,24 @@ the `not-admissible` decision or claim installability, replay, or equivalence; the [scenario/ledger guardrails](docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md) fix its boundaries. +## CybORG/CAGE-2 runtime qualification + +Issue [#12](https://github.com/RAESystem/adapters/issues/12) selects the +official CAGE Challenge 2 repository at commit +`26ce1c1253fa9e2e73f25e6a7f2da32860c11257`, including its bundled CybORG 2.1, +Scenario2, evaluator, wrappers, and baseline agents as one source closure. The +[qualification record](src/raes_adapters/cyborg/qualification.json) binds the +source and file digests, dependency resolution, legal decisions, known defects, +and sanitized red/blue/green smoke result. The accompanying +[packaging patch](src/raes_adapters/cyborg/cage2-wheel-package-data.patch) is +qualification evidence only; it is not silently applied or published. + +The separate +[architecture guardrails](docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md) +define the boundary: the result is not an adapter manifest, conformance claim, +replication claim, or permission to place native CybORG state in portable RAES +artifacts. + ## Development Requires [`uv`](https://docs.astral.sh/uv/). Repo-wide gates run through nox: diff --git a/docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md b/docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md new file mode 100644 index 0000000..6d9d244 --- /dev/null +++ b/docs/decisions/cyborg-cage2-runtime-qualification-guardrails.md @@ -0,0 +1,203 @@ +# CybORG/CAGE-2 runtime qualification guardrails + +Issue #12 is the authority for the qualification outcome. This note fixes the +repository and contract boundaries that outcome must respect; it does not select +an upstream revision, define a runtime-profile schema, or describe an +implementation plan. + +## Keep the source closure coherent + +The CAGE Challenge 2 repository contains a bundled `CybORG/` source tree, +Scenario2 content, wrappers, evaluation code, and baseline agents. The +standalone CybORG repository is a distinct, evolving source line. A version +label such as "2.1" does not prove that files from the two repositories are the +same. + +The selected profile must therefore name one coherent source closure: + +- the official repository and full commit id used as the root; +- the root tree id and the selected CybORG subtree identity; +- the simulator's own version declaration as descriptive metadata, not as the + immutable identity; +- each Scenario2 definition, evaluation entry point, wrapper, baseline red, + blue, and green agent, packaging file, and dependency declaration by + commit-qualified path or symbol and content digest where a file exists; and +- every separately sourced file, fork commit, patch, or built artifact with its + origin, digest, license, and relationship to the root closure. + +Do not combine a Challenge 2 scenario from one revision, wrappers or agents from +another, and a standalone CybORG package from a third while calling the result +"CAGE-2". If a mixed closure is unavoidable, it is a composed compatibility +profile: every delta and behavioral limit is explicit, and no equivalence to +the original public profile is implied. + +Git commit, Git tree, selected-file digests, installable artifact digest, and +resolved environment are different identities. GitHub-generated archive bytes +alone are not the source authority. The checked-in qualification record should +reference the repository's canonical `uv.lock` by digest and record the +resolved packages needed to interpret the result; it must not become a second +lockfile or dependency solver input. + +The qualification record is backend-local evidence under +`raes_adapters.cyborg`. It is distinct from: + +- the CAGE-2 mapping ledger, whose rows translate source facts into RAES + targets and should cite the selected source identities rather than choose a + second runtime; +- a RAES experiment specification or evidence record, which uses the published + RAES experiment contracts if one is later emitted; +- a backend manifest or conformance profile, which describes implemented RAES + capabilities rather than upstream installability; and +- a replication or outcome-equivalence claim, which cannot be established by + this smoke qualification. + +No shared qualification DTO, schema registry, simulator catalog, or profile +loader belongs in `raes_adapters.base`. + +## Legal, packaging, and patch boundary + +License conclusions are scoped to the exact selected bytes. Record the pinned +license and notice digests, copyright and attribution duties, redistribution +permission for source and built artifacts, retained-output terms, and the +provenance of bundled examples, scenario data, and baseline code. A repository +root license is not by itself evidence that every nested or separately sourced +asset can be redistributed. Dependency licenses and notice obligations remain +part of the resolved profile. + +The `cyborg` extra must be installable from the published `raes-adapters` +artifact on the declared Python/platform boundary. A successful editable +checkout, local path dependency, floating branch or tag, install-time clone, or +warm user cache is not that proof. If official source cannot produce a +publishable immutable dependency, any governed repackaging or maintained fork +must preserve notices and bind: + +- the unmodified upstream commit and tree; +- the fork or patch commit, patch digest, and resulting tree/wheel digest; +- the packaging-only and behavioral deltas separately; +- the reason, license, maintenance owner, and update policy; and +- a source-native compatibility test against the original public behavior. + +A dependency override, copied source file, monkey patch, or wrapper edit visible +only in setup code is a prohibited silent patch. A patch that changes scenario, +action, observation, scheduling, reward, or termination semantics is not merely +a Python/Gym compatibility patch; it changes the supported behavior boundary +and must be disclosed as such. + +`pyproject.toml` and the single `uv.lock` remain the packaging authorities. The +profile may summarize their resolution and pin their digests, but must not add +a second requirements lock, uv workspace, nested project, or dependency group +standing in for the published extra. Add a uv extras conflict only for a +demonstrated incompatibility with another simulator stack. + +The repository currently declares `requires-python = ">=3.12"` while CI selects +Python 3.12. Qualification must not use an environment marker to make the extra +silently empty on another version the distribution claims to support. Either +the selected stack is qualified across the declared boundary or that boundary +is narrowed explicitly and consistently in package metadata, lock resolution, +classifiers, CI, and documentation. + +Likewise, `pyproject.toml` currently classifies the distribution as operating +system independent while the canonical CI host is Ubuntu. Record OS, CPU, libc, +and interpreter identity for the proof. A Linux-only smoke does not establish +that the simulator extra is OS independent; either qualify the supported host +matrix or narrow the extra's documented and metadata boundary without changing +the portability claim of the RAES contracts themselves. + +## Reuse the canonical boundaries + +| Concern | Canonical incumbent | CybORG qualification boundary | +| --- | --- | --- | +| Packaging and resolution | `pyproject.toml`, one `uv.lock`, and ADR-003 | Populate only the existing `cyborg` extra; use frozen resolution and artifact hashes. | +| Base/extra isolation | `_extras()` and `_verification_envs()` in `noxfile.py`, plus `tests/test_cyborg_smoke.py` | Verify base and `cyborg` separately, never with `--all-extras`; base must not install or import the native simulator accidentally. | +| Built-artifact proof | `_distributions()` and `tools/probe_installed_identity.py` | Extend the existing wheel/sdist, throwaway-environment proof for both base and the extra instead of adding another build workflow or probe family. | +| Source-native precedent | `raes_adapters.cyberbattlesim` and `tests/test_cyberbattlesim_qualification.py` | Reuse backend-local evidence ownership, immutable identities, explicit legal/patch/defect decisions, and sanitized smoke summaries; do not copy its JSON shape as a shared schema. | +| CybORG source mapping | `raes_adapters.cyborg.mapping` | Cite one qualification-owned source selection; do not let the mapping ledger and runtime profile select competing revisions. | +| Portable contracts | `ContractModel` (`extra="forbid"`), `parse_experiment_spec`, and RAES cross-artifact validators | Use only if portable experiment/evidence artifacts are emitted. Qualification metadata itself remains backend-local and does not justify a parallel RAES model. | +| Portable source identity | `ExperimentArtifactRefModel` and `ExperimentChecksumModel` | Use their meanings and checksum representation when the selected source is referenced from a RAES artifact; do not overload free-form parameters. | +| Stochastic controls | `ExperimentStochasticControlModel`, `PublicSeedModel`, and `RandomStreamControlBindingModel` | Keep simulator, scenario generator, scheduler, red/green/baseline policy, wrapper/action-space, Python, and NumPy streams distinct. One seed is not complete binding. | +| Failures | RAES `Diagnostic`, `DiagnosticModel`, and `ApplyResult` | Later portable failures use bounded, input-free messages. Native exceptions, rejected actions, observations, object representations, and tracebacks remain backend-local. | +| Durable state | RAES `ControlPlaneStore` | Qualification uses checked-in immutable evidence and explicit ephemeral temporary directories; it introduces no database, cache authority, or evidence repository. | +| Workflow gates | the `policy`, `typecheck`, `tests`, `distributions`, `docs`, and `verify` nox sessions | Keep one canonical graph. Any new CI job would also have to join the `PR Gate` `needs` contract, so qualification belongs in the existing sessions unless isolation cannot otherwise be achieved. | + +The release workflow already installs the public +`raes-adapters[cyborg]` artifact on Python 3.12. It remains the public-index +check; source qualification and clean built-wheel smoke belong in the canonical +nox graph, not a second release path. + +## Native smoke and observability boundary + +The smoke operates on the selected source-native API before any RAES adapter +normalization. It must cover construction of Scenario2, reset, representative +participation by red, blue, and green, normal progress, the selected termination +path, and cleanup. Record scheduling and API semantics precisely: + +- distinguish one participant action from a complete environment turn; +- retain the selected blue/green/red execution order rather than relying on an + incidental call order; +- distinguish the core CybORG result from legacy Gym four-tuples and any + Gymnasium or PettingZoo normalization; +- distinguish reward vector, scalar aggregation, evaluation score, termination, + truncation/time limit, evaluator cutoff, and explicit close/cleanup; and +- bind every stochastic entry point observed during reset and stepping. + +The checked-in smoke result is a bounded structural summary: profile id, +source/artifact/lock digests, interpreter and platform identity, selected API +and wrapper identities, role participation, type/shape/arity facts, step count, +termination reason/class, and pass/fail status. It must not contain native +state, complete observations, hidden truth, action ids or object +representations, reward vectors, raw logs, environment or argument dumps, +credentials, or full tracebacks. A digest proves the identity of a named source +or artifact; hashing a native-state dump does not make that state portable. + +The extension seam is the immutable selected-profile identity passed to a +module-local source-native smoke driver. Scenario paths, wrapper choice, role +agents, seeds, representative actions, and termination bounds come from that +one selection rather than being repeated across tests, scripts, and prose. A +future patched profile or another CAGE scenario should be addable as another +immutable selection without changing the runner or inventing a central +simulator registry. + +## Security boundary + +Qualification is local simulator execution and introduces no HTTP, +authentication, authorization, or secret-binding surface. It must use public +sources and must not require credentials, private downloads, or user-home cache +state. Installation may use the package index; the source-native runtime must +be able to execute without network access unless an unavoidable access is +recorded as part of the profile. + +Commands use argument vectors rather than shell interpolation. Temporary and +cache paths are explicit, the working directory is isolated, `PYTHONPATH` is +cleared, and safe-path behavior is enabled. Do not place tokens in process +arguments, log environment mappings, or archive subprocess output wholesale. +Unexpected filesystem writes, runtime downloads, subprocesses, or socket use +are qualification findings. + +If later adapter work adds an HTTP surface, it must reuse +`ControlPlaneSecurityConfig.strict_defaults`, verified identities, target/role +authorization, request-size guards, denial audit, and the redacted exception +handler from `raes_runtime`. Qualification creates no alternate controller or +weaker endpoint. Standard module logging is for bounded local operational facts; +portable observability remains RAES diagnostics and evidence contracts. + +## Non-goals and anti-patterns + +- No adapter implementation, RAES SDL scenario, backend manifest, conformance + profile, evaluator implementation, or outcome-equivalence claim is part of + qualification. +- No native CybORG package becomes a base dependency, and + `raes_adapters.cyborg` must remain importable without the extra by keeping + native imports lazy and narrow. Do not catch broad `ImportError` in a way that + makes a broken installed extra look absent. +- Do not treat import success, one unseeded episode, a notebook run, or a green + canonical graph as proof of determinism, legal usability, public + installability, or CAGE-2 equivalence. +- Do not make the legacy Gym wrapper the RAES backend protocol, normalize tuple + differences without recording them, or reuse native action ids as portable + action identities. +- Do not duplicate dependency pins in prose/tests, duplicate source selection + in the mapping ledger, introduce a qualification exception hierarchy, or add + permissive parsing for a backend-local record. +- Do not vendor the whole upstream repository, datasets, submissions, images, + or examples merely to obtain a few qualified runtime files. Any vendoring + must be necessary, minimal, provenance-bound, and legally permitted. diff --git a/docs/index.md b/docs/index.md index efbe7c2..85bc582 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,10 +2,11 @@ This repository contains a single distribution, `raes-adapters`, that connects concrete simulator backends to published RAES contracts. It ships shared base -plumbing plus optional backend modules such as `raes-adapters[cyborg]`, so -incompatible simulator dependencies stay behind separate extras in one lock. -Backend-local qualification evidence may ship before an extra when the -qualification result is fail-closed. +plumbing plus backend modules whose dependencies can live behind separate +extras in one lock once qualified. The reserved `cyborg` extra remains empty +because the selected upstream source needs an unpublished packaging fix. +Backend-local qualification evidence ships fail-closed without making the +native simulator a base dependency. The shared `raes_adapters.base` module provides plumbing only. RAES remains the semantic and protocol authority. @@ -15,6 +16,7 @@ semantic and protocol authority. - [Repository overview](https://github.com/RAESystem/adapters#readme) - [Contribution guide](https://github.com/RAESystem/adapters/blob/dev/CONTRIBUTING.md) - [Architecture decisions](decisions/adrs/README.md) +- [CybORG/CAGE-2 runtime qualification guardrails](decisions/cyborg-cage2-runtime-qualification-guardrails.md) - [CyberBattleSim qualification guardrails](decisions/cyberbattlesim-qualification-guardrails.md) - [CyberBattleSim scenario and source-ledger guardrails](decisions/cyberbattlesim-scenario-ledger-guardrails.md) - [Project services](maintainers/project-services.md) diff --git a/mkdocs.yml b/mkdocs.yml index c3e48cd..ff8fd62 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -36,6 +36,7 @@ nav: - Decisions: - Overview: decisions/adrs/README.md - Template: decisions/adrs/TEMPLATE.md + - CybORG/CAGE-2 runtime qualification guardrails: decisions/cyborg-cage2-runtime-qualification-guardrails.md - CyberBattleSim qualification guardrails: decisions/cyberbattlesim-qualification-guardrails.md - CyberBattleSim scenario and source-ledger guardrails: decisions/cyberbattlesim-scenario-ledger-guardrails.md - ADR 003 — Single distribution and Trusted Publishing: decisions/adrs/adr-003-single-distribution-and-trusted-publishing.md diff --git a/noxfile.py b/noxfile.py index b76f9f4..6173605 100644 --- a/noxfile.py +++ b/noxfile.py @@ -142,6 +142,8 @@ def _tests(session: nox.Session) -> None: for sync_args, label in _verification_envs(): session.log(f"tests: {label}") _run(session, "uv", "sync", "--frozen", *sync_args) + if label == "cyborg": + _uv_run_root(session, "python", "tools/verify_cyborg_qualification.py") _run(session, "uv", "run", "--frozen", "coverage", "run", "--parallel-mode", "-m", "pytest") _run(session, "uv", "run", "--frozen", "coverage", "combine") _run(session, "uv", "run", "--frozen", "coverage", "xml") diff --git a/pyproject.toml b/pyproject.toml index f84bcf5..7eb54f5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -36,8 +36,11 @@ classifiers = [ dependencies = ["raes==2.0.0"] [project.optional-dependencies] -# CybORG backend. Declared now so the extra and its Trusted-Publishing pipeline -# exist before the concrete simulator pin lands under REP-004. +# Reserved CybORG backend key. Issue #12 found the official CAGE-2 source +# runnable only after a packaging-data patch that has no governed public +# artifact. Keep the extra empty rather than publish an editable checkout, +# install-time clone, or ungoverned wheel; the backend-local qualification +# record carries the exact source, patch, dependency, and smoke evidence. cyborg = [] [project.urls] diff --git a/src/raes_adapters/cyborg/__init__.py b/src/raes_adapters/cyborg/__init__.py index 792417d..d2d7bd4 100644 --- a/src/raes_adapters/cyborg/__init__.py +++ b/src/raes_adapters/cyborg/__init__.py @@ -1,15 +1,31 @@ -"""CybORG simulator backend adapter for RAES (optional ``cyborg`` extra). +"""CybORG backend-local qualification evidence. -Per RAES ADR-069 §3 the CybORG adapter is a conformant simulator backend behind -the RAES backend protocol surface (Provisioner, Orchestrator, Evaluator, -ParticipantRuntime). It keeps native CybORG state, gym/PettingZoo tuples, reward -vectors, action ids, and simulator object reprs adapter-private. +Issue #12 binds one immutable CAGE-2 source closure and its qualification +outcome here. The record and packaging-only patch are evidence; they do not +implement an adapter, backend manifest, conformance profile, RAES semantic +model, or native-state serializer. -Install the simulator dependencies with ``pip install raes-adapters[cyborg]``. -The backend protocol implementations land under REP-004; the CAGE-2 mapping -ledger (:mod:`raes_adapters.cyborg.mapping`) under REP-003. +The native simulator is deliberately not imported by this module. The current +qualification is fail-closed because no governed public artifact contains the +required packaging fix, so base-only installations remain independent. """ from __future__ import annotations -__all__: list[str] = [] +import json +from importlib.resources import files +from typing import Any, cast + +__all__ = ["load_qualification", "read_compatibility_patch"] + + +def load_qualification() -> dict[str, Any]: + """Load a fresh copy of the selected CAGE-2 qualification record.""" + resource = files(__package__).joinpath("qualification.json") + return cast(dict[str, Any], json.loads(resource.read_text(encoding="utf-8"))) + + +def read_compatibility_patch() -> str: + """Read the qualification-only packaging patch verbatim.""" + resource = files(__package__).joinpath("cage2-wheel-package-data.patch") + return resource.read_text(encoding="utf-8") diff --git a/src/raes_adapters/cyborg/cage2-wheel-package-data.patch b/src/raes_adapters/cyborg/cage2-wheel-package-data.patch new file mode 100644 index 0000000..6c2de8b --- /dev/null +++ b/src/raes_adapters/cyborg/cage2-wheel-package-data.patch @@ -0,0 +1,17 @@ +diff --git a/CybORG/setup.py b/CybORG/setup.py +index b41e2ba..4b8b58e 100644 +--- a/CybORG/setup.py ++++ b/CybORG/setup.py +@@ -17,4 +17,12 @@ setup( + version=CYBORG_VERSION, + install_requires=requirements, + description="A Cyber Security Research Environment", ++ package_data={ ++ "CybORG": [ ++ "version.txt", ++ "Shared/Config/*.ini", ++ "Shared/Scenarios/*.yaml", ++ "Shared/Scenarios/images/*.yaml", ++ ] ++ }, + ) diff --git a/src/raes_adapters/cyborg/qualification.json b/src/raes_adapters/cyborg/qualification.json new file mode 100644 index 0000000..9d804b4 --- /dev/null +++ b/src/raes_adapters/cyborg/qualification.json @@ -0,0 +1,431 @@ +{ + "record_version": "cyborg-cage2-qualification/1", + "profile_id": "cage2-cyborg-2.1-source-26ce1c1", + "qualified_at": "2026-07-30", + "scope": "RAESystem/adapters#12", + "source": { + "repository": "https://github.com/cage-challenge/cage-challenge-2", + "commit": "26ce1c1253fa9e2e73f25e6a7f2da32860c11257", + "tree": "f2ed32db4ac93500659d6e320e898c83e544e1c6", + "cyborg_tree": "a125e32fb9b7392ae2ac1b94485f53570298974a", + "package_tree": "b4180d2be22baf79e16a84bd375e03c2d42565c9", + "archive_sha256": "b1127d8f77ddf8f69ec6ebc21691ae8484e7040e71704493ad20d81a2b280d1a", + "package": "CybORG", + "version": "2.1" + }, + "selected_files": [ + { + "path": "LICENSE", + "sha256": "feaf0fa4b07710228ac471388210bdfb442932af277938d73f29f12104bb29fb" + }, + { + "path": "README.md", + "sha256": "f48bc9ec73a59b401da77806a68c18c06b86428d116bc226ece8d5d8831fb754" + }, + { + "path": "CybORG/LICENSE.txt", + "sha256": "1a6b129c24eb4fd73c1b20cb12eae7d96217f03bb1acc57ac9321fdcd9cb562c" + }, + { + "path": "CybORG/setup.py", + "sha256": "62a0d4ff3888eaeb4cd3bf96bbbd8592b6b442f4d291675a5ef5a4d8eeb76797" + }, + { + "path": "CybORG/Requirements.txt", + "sha256": "a930508faee1766fb685ffa2fdd924121fdd0c056c92144850d20383951ba301" + }, + { + "path": "CybORG/CybORG/version.txt", + "sha256": "bf8bd8826b985abf5e6347211af927ee7130a5df00cfed5f1517cbcc1134b0c3" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/Scenario2.yaml", + "sha256": "c2f15ec1bd9ab94d2a44acadf6cb9a71acdcb4cbb8fc946129c832139b155b1b" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/Gateway_image.yaml", + "sha256": "3606cf43fbb8c259a6ab74c717c6e5c7a4e069d816f312d17c0d512862996915" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/Internal_image.yaml", + "sha256": "301c42ee3e9c410fc7df778d450d315aff1eedad873e01b8d79dfcac3a299ae8" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/Kali_Box_image.yaml", + "sha256": "119b10680bca8e20981b63aca3b5f726aa905f14523199b20985e4796d10aecb" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/OP_Server_image.yaml", + "sha256": "43fcbe629ce635edb6647ed60abb3365f276c226edaba9888be2f6705de2c04d" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/Velociraptor_Server_image.yaml", + "sha256": "6a890510d62e68417f59637c82caed873801c742e71199d6f18738a96f3adf2e" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/images.yaml", + "sha256": "8467fca6b6f6baff4514993b8516cb9d3efd629f14f7afacea95f6f141722993" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/linux_decoy_host_image.yaml", + "sha256": "846dd8848a720341b0207d6dc58c3e05f199cf33c6164ef9a55184b4c349358d" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/linux_user_host_image1.yaml", + "sha256": "a474343f4cd6c922eeb9ec9f24c57bd2b93c01ab7a7bc18fa11f2b97bf1bb6f1" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/linux_user_host_image2.yaml", + "sha256": "c77e490c7eb2e972be7e5c71daa94d7a38d3b6995653a3b888453d63962e3940" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/windows_user_host_image1.yaml", + "sha256": "895a902fb2294da3dd6f1da373222d6cc8dd60ed861190b49f6da104944673f9" + }, + { + "path": "CybORG/CybORG/Shared/Scenarios/images/windows_user_host_image2.yaml", + "sha256": "b8a0f29b5424e7c91ca2586b863c45f91cef402ddb0c91b84c17568adcec8e5a" + }, + { + "path": "CybORG/CybORG/Evaluation/evaluation.py", + "sha256": "e4980dea8f1243ffda44bbae92521fc79ce89e74183623cce4d21715b940d51c" + }, + { + "path": "CybORG/CybORG/Agents/Wrappers/ChallengeWrapper.py", + "sha256": "61ff99c786c63613c8db2a908c43a5e2835bff9c0fd760a6d1064cc5eb37102c" + }, + { + "path": "CybORG/CybORG/Agents/SimpleAgents/B_line.py", + "sha256": "d08d86225c62da0081c3c99bf63cee9bb32670e4d7d56382d3cd1d61e384ae2f" + }, + { + "path": "CybORG/CybORG/Agents/SimpleAgents/Meander.py", + "sha256": "518b6e65c9a810dea434c83f1aa30d9662d5afd253d8c27c4d0375de42e0d173" + }, + { + "path": "CybORG/CybORG/Agents/SimpleAgents/GreenAgent.py", + "sha256": "eaaa58a79db113752a273030ff0cb346c2b1dbb59b7fbb35622081b4a59c271c" + }, + { + "path": "CybORG/CybORG/Agents/SimpleAgents/SleepAgent.py", + "sha256": "59407ee56bf1cf5cfa7a5d6929874e7bb444e06e052daa4456a7bf1cef051248" + }, + { + "path": "CybORG/CybORG/Agents/SimpleAgents/BlueLoadAgent.py", + "sha256": "86ee7e45b053bd66d4cbf42fed6bd915c8bde40fd9da0f8bc0230da8337ba453" + } + ], + "selection": { + "scenario": { + "path": "CybORG/CybORG/Shared/Scenarios/Scenario2.yaml", + "environment": "sim" + }, + "evaluation": { + "path": "CybORG/CybORG/Evaluation/evaluation.py", + "episodes": 100, + "trial_lengths": [ + 30, + 50, + 100 + ] + }, + "wrapper": { + "symbol": "CybORG.Agents.Wrappers.ChallengeWrapper", + "agent_name": "Blue", + "smoke_max_steps": 2 + }, + "agents": { + "blue": "external Sleep action", + "red": "CybORG.Agents.B_lineAgent", + "green": "CybORG.Agents.GreenAgent", + "evaluation_red_set": [ + "B_lineAgent", + "RedMeanderAgent", + "SleepAgent" + ], + "evaluation_blue": "BlueLoadAgent" + }, + "supported_boundary": "Scenario2 simulation through the source-native CybORG 2.1 core and legacy ChallengeWrapper only; no adapter, conformance, replay, determinism, or outcome-equivalence claim" + }, + "packaging": { + "documented_install": "editable checkout from the bundled CybORG directory", + "unmodified_wheel": { + "sha256": "80708d7e8c964a598530718895c374242b03992d6189c3de9a00666d6c67275d", + "normalized_sha256": "ee61283f78df460441285263c861fc9c7ce65737b39f131125f889aadfd21bcd", + "build": "passed", + "import": "failed", + "missing_runtime_files": [ + "CybORG/version.txt", + "CybORG/Shared/Scenarios/Scenario2.yaml", + "CybORG/Shared/Scenarios/images/*.yaml" + ] + }, + "original_behavior_probe": { + "installation": "editable-source", + "result": "passed", + "python": "3.12.3", + "reset_type": "Results", + "step_type": "Results", + "reward_type": "float", + "done": false, + "role_actions": { + "blue": "Sleep", + "red": "DiscoverRemoteSystems", + "green": "GreenPortScan" + } + }, + "patched_wheel": { + "filename": "cyborg-2.1-py3-none-any.whl", + "sha256": "c855bb40c88adbd02a6959ecfc325ca9cb1119680dd339a1220c6ae758b04347", + "normalized_sha256": "9b6ecd41749466f3f6f5cc118be9536f153e57fb66b400aaa949f88d1c736039", + "clean_install": "passed", + "source_native_smoke": "passed", + "published": false + } + }, + "patches": [ + { + "resource": "cage2-wheel-package-data.patch", + "sha256": "8d5ce9ceed1631227a8afd1d77aab68c920a5c5d62bda93911cdd20dfbaa8832", + "kind": "packaging-only", + "status": "qualification-candidate-only", + "original_setup_blob": "b41e2ba35fb8ac87bbda68bdd3f1a65a0467f4f5", + "patched_setup_blob": "4b8b58ef4131d86fa3fe6863a6601eddf485779b", + "patched_setup_sha256": "65472d96adadad8a03bd64ca983e890169ab6c5ccb06bd4e9b80332d16ad9886", + "behavioral_delta": "none", + "purpose": "include the version, configuration, Scenario2, and scenario image YAML files required by the installed runtime", + "license": "MIT", + "maintenance_owner": "none; not approved for publication", + "update_policy": "requalify against a new upstream commit or governed patch artifact" + } + ], + "runtime": { + "python": "3.12.3", + "platform": { + "system": "Linux", + "kernel": "6.8.0-117-generic", + "machine": "x86_64", + "libc": "glibc-2.39" + }, + "working_directory": "isolated-temporary-directory", + "pythonpath": "cleared", + "python_safe_path": true, + "network_after_install": "not-required", + "smoke": { + "seed": 3, + "scenario_exists": true, + "reset_type": "ndarray", + "reset_shape": [ + 52 + ], + "step_arity": 4, + "reward_types": [ + "float", + "float" + ], + "step_count": 2, + "done_sequence": [ + false, + true + ], + "termination": "ChallengeWrapper.max_steps", + "role_actions": { + "blue": [ + "Sleep", + "Sleep" + ], + "red": [ + "DiscoverRemoteSystems", + "DiscoverNetworkServices" + ], + "green": [ + "GreenPortScan", + "GreenPortScan" + ] + } + } + }, + "dependency_resolution": { + "resolver": "uv", + "python": "3.12.3", + "requirements_sha256": "a930508faee1766fb685ffa2fdd924121fdd0c056c92144850d20383951ba301", + "normalized_environment_sha256": "56e2c189d6fd4b195584f34d93f66500b30eba8eedff6fca869ed61a8ab4079a", + "uv_lock_sha256": "7f815a2ed3d699432eb2bbfe5b3a2d6d09eae41be36ee794ee53cdb36d28b31b", + "candidate_resolution": "passed", + "canonical_lock_disposition": "The current single uv.lock intentionally excludes the unpublished candidate CybORG artifact; the resolved environment is qualification evidence, not a second lockfile." + }, + "dependencies": [ + { + "name": "bcrypt", + "version": "5.0.0", + "license": "Apache-2.0" + }, + { + "name": "cffi", + "version": "2.1.0", + "license": "MIT-0" + }, + { + "name": "cloudpickle", + "version": "3.1.2", + "license": "BSD-3-Clause" + }, + { + "name": "cryptography", + "version": "49.0.0", + "license": "Apache-2.0 OR BSD-3-Clause" + }, + { + "name": "docutils", + "version": "0.23", + "license": "metadata-unknown" + }, + { + "name": "gym", + "version": "0.26.2", + "license": "MIT" + }, + { + "name": "gym-notices", + "version": "0.1.0", + "license": "metadata-unknown" + }, + { + "name": "iniconfig", + "version": "2.3.0", + "license": "MIT" + }, + { + "name": "invoke", + "version": "3.0.3", + "license": "BSD-2-Clause" + }, + { + "name": "numpy", + "version": "2.5.1", + "license": "BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0" + }, + { + "name": "packaging", + "version": "26.2", + "license": "Apache-2.0 OR BSD-2-Clause" + }, + { + "name": "paramiko", + "version": "5.0.0", + "license": "LGPL-2.1" + }, + { + "name": "pluggy", + "version": "1.6.0", + "license": "MIT" + }, + { + "name": "prettytable", + "version": "3.18.0", + "license": "BSD-3-Clause" + }, + { + "name": "pycparser", + "version": "3.0", + "license": "BSD-3-Clause" + }, + { + "name": "Pygments", + "version": "2.20.0", + "license": "BSD-2-Clause" + }, + { + "name": "PyNaCl", + "version": "1.6.2", + "license": "Apache-2.0" + }, + { + "name": "pytest", + "version": "9.1.1", + "license": "MIT" + }, + { + "name": "PyYAML", + "version": "6.0.3", + "license": "MIT" + }, + { + "name": "wcwidth", + "version": "0.8.2", + "license": "MIT" + } + ], + "stochastic_sources": [ + { + "owner": "cyborg-python-random", + "binding": "CybORG.set_seed(3)", + "coverage": "simulator, red agent, green agent, and Python-random action paths" + }, + { + "owner": "gym-action-space", + "binding": "not sampled by the smoke; explicit action 0 is used", + "coverage": "legacy wrapper action-space RNG remains a separate stream" + }, + { + "owner": "numpy-global", + "binding": "numpy.random.seed(3)", + "coverage": "legacy wrapper and array-library global RNG" + }, + { + "owner": "external-blue-policy", + "binding": "deterministic Sleep action in the smoke", + "coverage": "the official BlueLoadAgent evaluator policy does not bind its framework RNG" + } + ], + "legal": { + "root_license": "MIT", + "bundled_cyborg_license": "MIT", + "notice_required": true, + "source_redistribution": "permitted-with-notice", + "patched_wheel_redistribution": "permitted-with-notice", + "scenario_redistribution": "permitted-with-notice", + "bundled_examples_redistribution": "permitted-with-notice", + "retained_output": "permitted", + "external_downloads": [], + "model_weights": [] + }, + "known_defects": [ + { + "id": "wheel-omits-runtime-data", + "evidence": "The unmodified wheel omits CybORG/version.txt and Scenario2 YAML resources, so importing CybORG fails after a normal wheel install.", + "disposition": "blocking for the published extra" + }, + { + "id": "legacy-gym-unmaintained", + "evidence": "Gym 0.26.2 emits its upstream unmaintained/NumPy 2 warning during the clean patched-wheel smoke.", + "disposition": "supported only inside the recorded legacy wrapper boundary" + }, + { + "id": "remove-success-misreport", + "evidence": "The selected Remove action initializes a successful observation even when no suspicious process is removed; the maintained CybORG++ source documents and patches this behavior.", + "disposition": "unpatched behavioral limitation" + }, + { + "id": "scenario-user3-port-mismatch", + "evidence": "The selected linux_user_host_image1.yaml records local port 3389 where the maintained CybORG++ source documents 3390 for the SQL-injection path.", + "disposition": "unpatched scenario limitation" + }, + { + "id": "evaluation-seed-not-bound", + "evidence": "The pinned evaluation.py does not call random.seed or CybORG.set_seed even though the leaderboard README describes validated results with random.seed(153).", + "disposition": "no outcome-reproducibility claim" + } + ], + "admissibility": { + "decision": "not-admissible", + "scope": "published raes-adapters[cyborg] runtime", + "blockers": [ + "no-governed-public-patched-artifact", + "declared-python-range-not-qualified", + "declared-platform-range-not-qualified" + ], + "extra_disposition": "leave the cyborg extra empty rather than publish a local path, editable checkout, install-time clone, or ungoverned wheel", + "requalification_trigger": "an immutable governed artifact that contains the packaging-only fix and is qualified across the declared Python and platform support boundary" + } +} diff --git a/tests/test_cyborg_qualification.py b/tests/test_cyborg_qualification.py new file mode 100644 index 0000000..bba9a07 --- /dev/null +++ b/tests/test_cyborg_qualification.py @@ -0,0 +1,209 @@ +"""Qualification evidence for the selected CybORG/CAGE-2 runtime profile.""" + +from __future__ import annotations + +import hashlib +import json +import tomllib +from pathlib import Path + +import raes_adapters.cyborg as cyborg + +REPO_ROOT = Path(__file__).resolve().parents[1] + + +def test_resources_bind_one_immutable_cage2_source_closure() -> None: + record = cyborg.load_qualification() + + assert cyborg.__all__ == ["load_qualification", "read_compatibility_patch"] + assert record["profile_id"] == "cage2-cyborg-2.1-source-26ce1c1" + assert record["source"] == { + "repository": "https://github.com/cage-challenge/cage-challenge-2", + "commit": "26ce1c1253fa9e2e73f25e6a7f2da32860c11257", + "tree": "f2ed32db4ac93500659d6e320e898c83e544e1c6", + "cyborg_tree": "a125e32fb9b7392ae2ac1b94485f53570298974a", + "package_tree": "b4180d2be22baf79e16a84bd375e03c2d42565c9", + "archive_sha256": "b1127d8f77ddf8f69ec6ebc21691ae8484e7040e71704493ad20d81a2b280d1a", + "package": "CybORG", + "version": "2.1", + } + selected = {item["path"]: item["sha256"] for item in record["selected_files"]} + assert selected["CybORG/CybORG/Shared/Scenarios/Scenario2.yaml"] == ( + "c2f15ec1bd9ab94d2a44acadf6cb9a71acdcb4cbb8fc946129c832139b155b1b" + ) + assert selected["CybORG/CybORG/Evaluation/evaluation.py"] == ( + "e4980dea8f1243ffda44bbae92521fc79ce89e74183623cce4d21715b940d51c" + ) + assert selected["CybORG/CybORG/Agents/Wrappers/ChallengeWrapper.py"] == ( + "61ff99c786c63613c8db2a908c43a5e2835bff9c0fd760a6d1064cc5eb37102c" + ) + assert selected["README.md"] == ( + "f48bc9ec73a59b401da77806a68c18c06b86428d116bc226ece8d5d8831fb754" + ) + scenario_images = { + path for path in selected if path.startswith("CybORG/CybORG/Shared/Scenarios/images/") + } + assert scenario_images == { + "CybORG/CybORG/Shared/Scenarios/images/Gateway_image.yaml", + "CybORG/CybORG/Shared/Scenarios/images/Internal_image.yaml", + "CybORG/CybORG/Shared/Scenarios/images/Kali_Box_image.yaml", + "CybORG/CybORG/Shared/Scenarios/images/OP_Server_image.yaml", + "CybORG/CybORG/Shared/Scenarios/images/Velociraptor_Server_image.yaml", + "CybORG/CybORG/Shared/Scenarios/images/images.yaml", + "CybORG/CybORG/Shared/Scenarios/images/linux_decoy_host_image.yaml", + "CybORG/CybORG/Shared/Scenarios/images/linux_user_host_image1.yaml", + "CybORG/CybORG/Shared/Scenarios/images/linux_user_host_image2.yaml", + "CybORG/CybORG/Shared/Scenarios/images/windows_user_host_image1.yaml", + "CybORG/CybORG/Shared/Scenarios/images/windows_user_host_image2.yaml", + } + + +def test_packaging_patch_is_explicit_and_preserves_original_behavior() -> None: + record = cyborg.load_qualification() + patch = cyborg.read_compatibility_patch() + packaging = record["packaging"] + patch_record = record["patches"][0] + + assert hashlib.sha256(patch.encode("utf-8")).hexdigest() == patch_record["sha256"] + assert patch_record["kind"] == "packaging-only" + assert patch_record["behavioral_delta"] == "none" + assert patch_record["status"] == "qualification-candidate-only" + assert "CybORG/setup.py" in patch + assert "package_data" in patch + assert "Shared/Scenarios/*.yaml" in patch + assert "Shared/Scenarios/images/*.yaml" in patch + + assert packaging["unmodified_wheel"] == { + "sha256": "80708d7e8c964a598530718895c374242b03992d6189c3de9a00666d6c67275d", + "normalized_sha256": "ee61283f78df460441285263c861fc9c7ce65737b39f131125f889aadfd21bcd", + "build": "passed", + "import": "failed", + "missing_runtime_files": [ + "CybORG/version.txt", + "CybORG/Shared/Scenarios/Scenario2.yaml", + "CybORG/Shared/Scenarios/images/*.yaml", + ], + } + assert packaging["original_behavior_probe"]["installation"] == "editable-source" + assert packaging["original_behavior_probe"]["result"] == "passed" + assert packaging["patched_wheel"]["clean_install"] == "passed" + assert packaging["patched_wheel"]["source_native_smoke"] == "passed" + assert packaging["patched_wheel"]["normalized_sha256"] == ( + "9b6ecd41749466f3f6f5cc118be9536f153e57fb66b400aaa949f88d1c736039" + ) + + +def test_clean_patched_wheel_smoke_is_bounded_and_covers_all_roles() -> None: + record = cyborg.load_qualification() + runtime = record["runtime"] + smoke = runtime["smoke"] + + assert runtime["python"] == "3.12.3" + assert runtime["platform"] == { + "system": "Linux", + "kernel": "6.8.0-117-generic", + "machine": "x86_64", + "libc": "glibc-2.39", + } + assert runtime["working_directory"] == "isolated-temporary-directory" + assert runtime["pythonpath"] == "cleared" + assert runtime["python_safe_path"] is True + assert runtime["network_after_install"] == "not-required" + assert smoke == { + "seed": 3, + "scenario_exists": True, + "reset_type": "ndarray", + "reset_shape": [52], + "step_arity": 4, + "reward_types": ["float", "float"], + "step_count": 2, + "done_sequence": [False, True], + "termination": "ChallengeWrapper.max_steps", + "role_actions": { + "blue": ["Sleep", "Sleep"], + "red": ["DiscoverRemoteSystems", "DiscoverNetworkServices"], + "green": ["GreenPortScan", "GreenPortScan"], + }, + } + serialized = json.dumps(smoke) + assert "observation_values" not in serialized + assert "reward_vector" not in serialized + assert "action_id" not in serialized + assert "native_state" not in serialized + + +def test_selection_dependencies_and_stochastic_sources_are_explicit() -> None: + record = cyborg.load_qualification() + selection = record["selection"] + + assert selection["scenario"] == { + "path": "CybORG/CybORG/Shared/Scenarios/Scenario2.yaml", + "environment": "sim", + } + assert selection["evaluation"] == { + "path": "CybORG/CybORG/Evaluation/evaluation.py", + "episodes": 100, + "trial_lengths": [30, 50, 100], + } + assert selection["wrapper"] == { + "symbol": "CybORG.Agents.Wrappers.ChallengeWrapper", + "agent_name": "Blue", + "smoke_max_steps": 2, + } + assert selection["agents"] == { + "blue": "external Sleep action", + "red": "CybORG.Agents.B_lineAgent", + "green": "CybORG.Agents.GreenAgent", + "evaluation_red_set": ["B_lineAgent", "RedMeanderAgent", "SleepAgent"], + "evaluation_blue": "BlueLoadAgent", + } + assert record["dependency_resolution"]["normalized_environment_sha256"] == ( + "56e2c189d6fd4b195584f34d93f66500b30eba8eedff6fca869ed61a8ab4079a" + ) + assert record["dependency_resolution"]["uv_lock_sha256"] == ( + "7f815a2ed3d699432eb2bbfe5b3a2d6d09eae41be36ee794ee53cdb36d28b31b" + ) + assert record["dependencies"] + assert all( + {"name", "version", "license"} <= dependency.keys() for dependency in record["dependencies"] + ) + assert {stream["owner"] for stream in record["stochastic_sources"]} == { + "cyborg-python-random", + "gym-action-space", + "numpy-global", + "external-blue-policy", + } + + +def test_legal_defect_and_fail_closed_packaging_decisions_are_explicit() -> None: + record = cyborg.load_qualification() + project = tomllib.loads((REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8")) + extras = project["project"]["optional-dependencies"] + + assert record["legal"] == { + "root_license": "MIT", + "bundled_cyborg_license": "MIT", + "notice_required": True, + "source_redistribution": "permitted-with-notice", + "patched_wheel_redistribution": "permitted-with-notice", + "scenario_redistribution": "permitted-with-notice", + "bundled_examples_redistribution": "permitted-with-notice", + "retained_output": "permitted", + "external_downloads": [], + "model_weights": [], + } + assert {defect["id"] for defect in record["known_defects"]} == { + "wheel-omits-runtime-data", + "legacy-gym-unmaintained", + "remove-success-misreport", + "scenario-user3-port-mismatch", + "evaluation-seed-not-bound", + } + assert record["admissibility"]["decision"] == "not-admissible" + assert set(record["admissibility"]["blockers"]) == { + "no-governed-public-patched-artifact", + "declared-python-range-not-qualified", + "declared-platform-range-not-qualified", + } + assert extras["cyborg"] == [] + assert "CybORG" not in project["project"]["dependencies"] diff --git a/tests/test_cyborg_qualification_driver.py b/tests/test_cyborg_qualification_driver.py new file mode 100644 index 0000000..dd9862c --- /dev/null +++ b/tests/test_cyborg_qualification_driver.py @@ -0,0 +1,127 @@ +from __future__ import annotations + +import os +import tempfile +import unittest +import zipfile +from pathlib import Path + +from tools.verify_cyborg_qualification import ( + normalized_wheel_sha256, + sanitized_subprocess_env, + validate_reproducer_runtime, + validate_smoke, +) + + +class NormalizedWheelDigestTests(unittest.TestCase): + def test_zip_timestamps_do_not_change_normalized_identity(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + first = root / "first.whl" + second = root / "second.whl" + for path, timestamp in ( + (first, (2020, 1, 1, 0, 0, 0)), + (second, (2026, 7, 30, 0, 0, 0)), + ): + with zipfile.ZipFile(path, "w") as archive: + info = zipfile.ZipInfo("CybORG/version.txt", timestamp) + archive.writestr(info, "2.1\n") + + self.assertEqual( + normalized_wheel_sha256(first), + normalized_wheel_sha256(second), + ) + + +class SanitizedEnvironmentTests(unittest.TestCase): + def test_only_non_secret_process_inputs_are_forwarded(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + previous = os.environ.get("QUALIFICATION_TEST_TOKEN") + os.environ["QUALIFICATION_TEST_TOKEN"] = "must-not-propagate" + try: + environment = sanitized_subprocess_env(root) + finally: + if previous is None: + os.environ.pop("QUALIFICATION_TEST_TOKEN", None) + else: + os.environ["QUALIFICATION_TEST_TOKEN"] = previous + + self.assertNotIn("QUALIFICATION_TEST_TOKEN", environment) + self.assertEqual(environment["HOME"], str(root / "home")) + self.assertEqual(environment["UV_CACHE_DIR"], str(root / "uv-cache")) + self.assertEqual(environment["PYTHONPATH"], "") + self.assertEqual(environment["PYTHONSAFEPATH"], "1") + self.assertEqual(environment["GIT_TERMINAL_PROMPT"], "0") + + +class SmokeValidationTests(unittest.TestCase): + def test_exact_bounded_structural_result_passes(self) -> None: + expected = { + "seed": 3, + "step_count": 2, + "done_sequence": [False, True], + "role_actions": {"blue": ["Sleep", "Sleep"]}, + } + validate_smoke(dict(expected), expected) + + def test_native_state_or_shape_drift_fails(self) -> None: + expected = {"seed": 3, "step_count": 2} + + with self.assertRaisesRegex(RuntimeError, "smoke result"): + validate_smoke({"seed": 3, "step_count": 2, "native_state": {}}, expected) + with self.assertRaisesRegex(RuntimeError, "smoke result"): + validate_smoke({"seed": 3, "step_count": 3}, expected) + + +class RuntimeValidationTests(unittest.TestCase): + evidence = { + "python": "3.12.3", + "platform": { + "system": "Linux", + "kernel": "6.8.0-117-generic", + "machine": "x86_64", + "libc": "glibc-2.39", + }, + } + + def test_same_runtime_family_accepts_ci_patch_and_kernel_variation(self) -> None: + validate_reproducer_runtime( + actual_python="3.12.11", + actual_platform={ + "system": "Linux", + "kernel": "6.11.0-1018-azure", + "machine": "x86_64", + "libc": "glibc-2.39", + }, + evidence_runtime=self.evidence, + ) + + def test_different_python_or_platform_family_fails(self) -> None: + with self.assertRaisesRegex(RuntimeError, "runtime is incompatible"): + validate_reproducer_runtime( + actual_python="3.13.1", + actual_platform={ + "system": "Linux", + "kernel": "6.8.0", + "machine": "x86_64", + "libc": "glibc-2.39", + }, + evidence_runtime=self.evidence, + ) + with self.assertRaisesRegex(RuntimeError, "runtime is incompatible"): + validate_reproducer_runtime( + actual_python="3.12.3", + actual_platform={ + "system": "Darwin", + "kernel": "25.0.0", + "machine": "arm64", + "libc": "libSystem", + }, + evidence_runtime=self.evidence, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_cyborg_smoke.py b/tests/test_cyborg_smoke.py index c03c7a1..eb5f687 100644 --- a/tests/test_cyborg_smoke.py +++ b/tests/test_cyborg_smoke.py @@ -2,10 +2,14 @@ from __future__ import annotations +import sys + import raes_adapters.cyborg def test_cyborg_module_imports_without_the_extra() -> None: - # The skeleton imports cleanly with only the base install; the heavy CybORG - # dependency (guarded behind the ``cyborg`` extra) lands under REP-004. - assert raes_adapters.cyborg.__all__ == [] + assert raes_adapters.cyborg.__all__ == [ + "load_qualification", + "read_compatibility_patch", + ] + assert "CybORG" not in sys.modules diff --git a/tools/probe_installed_identity.py b/tools/probe_installed_identity.py index 93a3daa..8363d66 100644 --- a/tools/probe_installed_identity.py +++ b/tools/probe_installed_identity.py @@ -68,6 +68,20 @@ def main(argv: list[str]) -> int: except ImportError as exc: _fail(f"published RAES surface {surface!r} unreachable from the install: {exc}") + # Backend-local qualification evidence is part of the published adapter + # artifact even when the native simulator extra fails closed. Reading it + # here proves wheel builds did not drop the JSON or patch resources. + try: + cyborg = importlib.import_module(f"{module_name}.cyborg") + qualification = cyborg.load_qualification() + compatibility_patch = cyborg.read_compatibility_patch() + except (AttributeError, ImportError, OSError, ValueError): + _fail("installed CybORG qualification resources are unreadable") + if qualification.get("profile_id") != "cage2-cyborg-2.1-source-26ce1c1": + _fail("installed CybORG qualification profile identity is wrong") + if "CybORG/setup.py" not in compatibility_patch: + _fail("installed CybORG qualification patch identity is wrong") + print(f"installed-identity probe: OK ({module_name})") return 0 diff --git a/tools/verify_cyborg_qualification.py b/tools/verify_cyborg_qualification.py new file mode 100644 index 0000000..2e59b3f --- /dev/null +++ b/tools/verify_cyborg_qualification.py @@ -0,0 +1,504 @@ +#!/usr/bin/env python3 +"""Reproduce the bounded CybORG/CAGE-2 qualification evidence.""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys +import tempfile +import zipfile +from pathlib import Path +from typing import Any +from urllib.request import ProxyHandler, Request, build_opener + +import raes_adapters.cyborg as cyborg + +REPO_ROOT = Path(__file__).resolve().parent.parent +_RUNTIME_FILES = { + "CybORG/version.txt", + "CybORG/Shared/Config/defaultconfig.ini", + "CybORG/Shared/Scenarios/Scenario2.yaml", +} +_SCENARIO_IMAGE_PREFIX = "CybORG/Shared/Scenarios/images/" +_SMOKE_CODE = r""" +import json +from importlib.metadata import distributions +from pathlib import Path +import platform +import random +import sys + +import numpy as np +import CybORG as package +from CybORG import CYBORG_VERSION, CybORG +from CybORG.Agents import B_lineAgent, GreenAgent +from CybORG.Agents.Wrappers import ChallengeWrapper + +seed = int(sys.argv[1]) +max_steps = int(sys.argv[2]) +random.seed(seed) +np.random.seed(seed) +scenario = Path(package.__file__).parent / "Shared" / "Scenarios" / "Scenario2.yaml" +native = CybORG( + str(scenario), + "sim", + agents={"Red": B_lineAgent, "Green": GreenAgent}, +) +native.set_seed(seed) +environment = ChallengeWrapper(agent_name="Blue", env=native, max_steps=max_steps) +reset = environment.reset() +steps = [] +for _ in range(max_steps): + result = environment.step(0) + steps.append( + { + "reward_type": type(result[1]).__name__, + "done": result[2], + "actions": { + role.lower(): type(native.get_last_action(role)).__name__ + for role in ("Blue", "Red", "Green") + }, + } + ) +dependencies = { + (dist.metadata["Name"] or "").lower().replace("_", "-"): dist.version + for dist in distributions() + if (dist.metadata["Name"] or "").lower() != "cyborg" +} +print( + json.dumps( + { + "version": CYBORG_VERSION, + "python": platform.python_version(), + "platform": { + "system": platform.system(), + "kernel": platform.release(), + "machine": platform.machine(), + "libc": "-".join(platform.libc_ver()), + }, + "dependencies": dependencies, + "smoke": { + "seed": seed, + "scenario_exists": scenario.is_file(), + "reset_type": type(reset).__name__, + "reset_shape": list(reset.shape), + "step_arity": len(result), + "reward_types": [step["reward_type"] for step in steps], + "step_count": len(steps), + "done_sequence": [step["done"] for step in steps], + "termination": "ChallengeWrapper.max_steps", + "role_actions": { + role: [step["actions"][role] for step in steps] + for role in ("blue", "red", "green") + }, + }, + }, + sort_keys=True, + ) +) +""" + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def normalized_wheel_sha256(path: Path) -> str: + """Hash wheel member paths and bytes without timestamp-bearing ZIP metadata.""" + digest = hashlib.sha256() + with zipfile.ZipFile(path) as archive: + for member in sorted(archive.infolist(), key=lambda item: item.filename): + digest.update(member.filename.encode("utf-8")) + digest.update(b"\0") + digest.update(archive.read(member.filename)) + return digest.hexdigest() + + +def sanitized_subprocess_env(root: Path) -> dict[str, str]: + """Create the small public-build environment forwarded to child processes.""" + home = root / "home" + cache = root / "cache" + uv_cache = root / "uv-cache" + for path in (home, cache, uv_cache): + path.mkdir(parents=True, exist_ok=True) + return { + "PATH": os.environ.get("PATH", ""), + "LANG": "C.UTF-8", + "LC_ALL": "C.UTF-8", + "HOME": str(home), + "XDG_CACHE_HOME": str(cache), + "UV_CACHE_DIR": str(uv_cache), + "PIP_CACHE_DIR": str(root / "pip-cache"), + "PYTHONPATH": "", + "PYTHONSAFEPATH": "1", + "GIT_CONFIG_NOSYSTEM": "1", + "GIT_TERMINAL_PROMPT": "0", + "PIP_DISABLE_PIP_VERSION_CHECK": "1", + } + + +def validate_smoke(actual: dict[str, Any], expected: dict[str, Any]) -> None: + """Require the exact bounded structural smoke result and forbid native state.""" + forbidden = {"native_state", "observation_values", "reward_vector", "action_id"} + if actual != expected or forbidden & set(actual): + raise RuntimeError("CybORG qualification smoke result mismatch") + + +def validate_reproducer_runtime( + *, + actual_python: str, + actual_platform: dict[str, str], + evidence_runtime: dict[str, Any], +) -> None: + """Keep the reproducer on the evidence family without claiming range support.""" + expected_python_family = evidence_runtime["python"].split(".")[:2] + actual_python_family = actual_python.split(".")[:2] + expected_platform = evidence_runtime["platform"] + same_platform_family = ( + actual_platform.get("system") == expected_platform["system"] + and actual_platform.get("machine") == expected_platform["machine"] + and actual_platform.get("libc", "").partition("-")[0] + == expected_platform["libc"].partition("-")[0] + ) + if actual_python_family != expected_python_family or not same_platform_family: + raise RuntimeError("CybORG qualification reproducer runtime is incompatible") + + +def _run( + stage: str, + args: list[str], + *, + cwd: Path, + env: dict[str, str], + check: bool = True, +) -> subprocess.CompletedProcess[str]: + result = subprocess.run( + args, + cwd=cwd, + env=env, + text=True, + capture_output=True, + check=False, + timeout=300, + ) + if check and result.returncode != 0: + raise RuntimeError(f"CybORG qualification stage failed: {stage}") + return result + + +def _git_value(source: Path, env: dict[str, str], revision: str) -> str: + return _run( + "source identity", + ["git", "rev-parse", revision], + cwd=source, + env=env, + ).stdout.strip() + + +def _download_sha256(url: str) -> str: + digest = hashlib.sha256() + opener = build_opener(ProxyHandler({})) + request = Request(url, headers={"User-Agent": "raes-adapters-qualification/1"}) + with opener.open(request, timeout=60) as response: + for chunk in iter(lambda: response.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _single_wheel(directory: Path) -> Path: + wheels = sorted(directory.glob("*.whl")) + if len(wheels) != 1: + raise RuntimeError("CybORG qualification wheel count mismatch") + return wheels[0] + + +def _wheel_members(path: Path) -> set[str]: + with zipfile.ZipFile(path) as archive: + return set(archive.namelist()) + + +def _venv_python(venv: Path) -> Path: + return venv / ("Scripts/python.exe" if os.name == "nt" else "bin/python") + + +def _normalized_freeze( + python: Path, + *, + cwd: Path, + env: dict[str, str], +) -> tuple[list[str], str]: + output = _run( + "dependency freeze", + ["uv", "pip", "freeze", "--python", str(python)], + cwd=cwd, + env=env, + ).stdout + lines = sorted( + line.strip() + for line in output.splitlines() + if line.strip() and not line.lower().startswith("cyborg") + ) + normalized = "".join(f"{line}\n" for line in lines) + return lines, hashlib.sha256(normalized.encode("utf-8")).hexdigest() + + +def _verify_source_files(source: Path, selected_files: list[dict[str, str]]) -> None: + root = source.resolve() + for item in selected_files: + relative = Path(item["path"]) + candidate = (root / relative).resolve() + if relative.is_absolute() or not candidate.is_relative_to(root) or not candidate.is_file(): + raise RuntimeError("CybORG qualification selected path is invalid") + if _sha256(candidate) != item["sha256"]: + raise RuntimeError("CybORG qualification selected source digest mismatch") + + +def verify_qualification(repo_root: Path = REPO_ROOT) -> None: + """Rebuild and execute the selected profile from public immutable inputs.""" + record = cyborg.load_qualification() + patch_text = cyborg.read_compatibility_patch() + source_record = record["source"] + patch_record = record["patches"][0] + runtime_record = record["runtime"] + dependency_record = record["dependency_resolution"] + expected_versions = { + item["name"].lower().replace("_", "-"): item["version"] for item in record["dependencies"] + } + dependency_pins = sorted(f"{name}=={version}" for name, version in expected_versions.items()) + + if _sha256(repo_root / "uv.lock") != dependency_record["uv_lock_sha256"]: + raise RuntimeError("CybORG qualification canonical lock digest mismatch") + if hashlib.sha256(patch_text.encode("utf-8")).hexdigest() != patch_record["sha256"]: + raise RuntimeError("CybORG qualification patch digest mismatch") + + with tempfile.TemporaryDirectory(prefix="raes-cyborg-qualification-") as directory: + root = Path(directory) + source = root / "source" + env = sanitized_subprocess_env(root) + _run( + "source clone", + [ + "git", + "clone", + "--no-checkout", + "--filter=blob:none", + source_record["repository"], + str(source), + ], + cwd=root, + env=env, + ) + _run( + "source checkout", + ["git", "checkout", "--detach", source_record["commit"]], + cwd=source, + env=env, + ) + identities = { + "commit": _git_value(source, env, "HEAD"), + "tree": _git_value(source, env, "HEAD^{tree}"), + "cyborg_tree": _git_value(source, env, "HEAD:CybORG"), + "package_tree": _git_value(source, env, "HEAD:CybORG/CybORG"), + } + if any(identities[key] != source_record[key] for key in identities): + raise RuntimeError("CybORG qualification source identity mismatch") + _verify_source_files(source, record["selected_files"]) + + archive_url = ( + "https://codeload.github.com/cage-challenge/cage-challenge-2/tar.gz/" + f"{source_record['commit']}" + ) + if _download_sha256(archive_url) != source_record["archive_sha256"]: + raise RuntimeError("CybORG qualification source archive digest mismatch") + + original_dist = root / "original-dist" + _run( + "original wheel build", + [ + "uv", + "build", + "--wheel", + "--out-dir", + str(original_dist), + str(source / "CybORG"), + ], + cwd=root, + env=env, + ) + original_wheel = _single_wheel(original_dist) + original_record = record["packaging"]["unmodified_wheel"] + if normalized_wheel_sha256(original_wheel) != original_record["normalized_sha256"]: + raise RuntimeError("CybORG qualification original wheel identity mismatch") + original_members = _wheel_members(original_wheel) + if original_members & _RUNTIME_FILES: + raise RuntimeError( + "CybORG qualification original wheel unexpectedly includes runtime files" + ) + if any( + member.startswith(_SCENARIO_IMAGE_PREFIX) and member.endswith(".yaml") + for member in original_members + ): + raise RuntimeError( + "CybORG qualification original wheel unexpectedly includes scenarios" + ) + + original_venv = root / "original-venv" + _run( + "original venv", + ["uv", "venv", "--python", "3.12", str(original_venv)], + cwd=root, + env=env, + ) + original_python = _venv_python(original_venv) + _run( + "original wheel install", + [ + "uv", + "pip", + "install", + "--python", + str(original_python), + str(original_wheel), + *dependency_pins, + ], + cwd=root, + env=env, + ) + import_result = _run( + "original wheel import", + [str(original_python), "-c", "import CybORG"], + cwd=root, + env=env, + check=False, + ) + if import_result.returncode == 0 or "CybORG/version.txt" not in import_result.stderr: + raise RuntimeError("CybORG qualification original wheel failure mismatch") + + patch_path = root / "cage2-wheel-package-data.patch" + patch_path.write_text(patch_text, encoding="utf-8") + _run( + "patch applicability", + ["git", "apply", "--check", str(patch_path)], + cwd=source, + env=env, + ) + _run( + "patch application", + ["git", "apply", str(patch_path)], + cwd=source, + env=env, + ) + setup_path = source / "CybORG/setup.py" + if _sha256(setup_path) != patch_record["patched_setup_sha256"]: + raise RuntimeError("CybORG qualification patched setup digest mismatch") + patched_blob = _run( + "patched setup identity", + ["git", "hash-object", str(setup_path)], + cwd=source, + env=env, + ).stdout.strip() + if patched_blob != patch_record["patched_setup_blob"]: + raise RuntimeError("CybORG qualification patched setup identity mismatch") + + patched_dist = root / "patched-dist" + _run( + "patched wheel build", + [ + "uv", + "build", + "--wheel", + "--out-dir", + str(patched_dist), + str(source / "CybORG"), + ], + cwd=root, + env=env, + ) + patched_wheel = _single_wheel(patched_dist) + patched_record = record["packaging"]["patched_wheel"] + if normalized_wheel_sha256(patched_wheel) != patched_record["normalized_sha256"]: + raise RuntimeError("CybORG qualification patched wheel identity mismatch") + patched_members = _wheel_members(patched_wheel) + if not patched_members >= _RUNTIME_FILES: + raise RuntimeError("CybORG qualification patched wheel runtime data mismatch") + if not any( + member.startswith(_SCENARIO_IMAGE_PREFIX) and member.endswith(".yaml") + for member in patched_members + ): + raise RuntimeError("CybORG qualification patched wheel scenario data mismatch") + + patched_venv = root / "patched-venv" + _run( + "patched venv", + ["uv", "venv", "--python", "3.12", str(patched_venv)], + cwd=root, + env=env, + ) + patched_python = _venv_python(patched_venv) + _run( + "patched wheel install", + [ + "uv", + "pip", + "install", + "--python", + str(patched_python), + str(patched_wheel), + *dependency_pins, + ], + cwd=root, + env=env, + ) + frozen, frozen_sha256 = _normalized_freeze(patched_python, cwd=root, env=env) + if frozen_sha256 != dependency_record["normalized_environment_sha256"]: + raise RuntimeError("CybORG qualification dependency resolution mismatch") + if frozen != sorted(f"{name}=={version}" for name, version in expected_versions.items()): + raise RuntimeError("CybORG qualification dependency set mismatch") + + runtime_dir = root / "runtime" + runtime_dir.mkdir() + smoke_result = _run( + "source-native smoke", + [ + str(patched_python), + "-c", + _SMOKE_CODE, + str(runtime_record["smoke"]["seed"]), + str(record["selection"]["wrapper"]["smoke_max_steps"]), + ], + cwd=runtime_dir, + env=env, + ) + try: + probe = json.loads(smoke_result.stdout) + except json.JSONDecodeError as exc: + raise RuntimeError("CybORG qualification smoke output is invalid") from exc + if probe.get("version") != source_record["version"]: + raise RuntimeError("CybORG qualification simulator version mismatch") + validate_reproducer_runtime( + actual_python=probe.get("python", ""), + actual_platform=probe.get("platform", {}), + evidence_runtime=runtime_record, + ) + if probe.get("dependencies") != expected_versions: + raise RuntimeError("CybORG qualification dependency metadata mismatch") + validate_smoke(probe.get("smoke", {}), runtime_record["smoke"]) + + +def main(argv: list[str]) -> int: + if argv: + raise RuntimeError("CybORG qualification driver accepts no arguments") + verify_qualification() + print("CybORG qualification: OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:]))