diff --git a/README.md b/README.md index 8752a61..cd7aab4 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,9 @@ raes-adapters/ src/raes_adapters/ base/ # shared adapter plumbing (ADR-069 §4) cyberbattlesim/ # immutable qualification + selected public protocol + scenario/ # authored RAES SDL scenario (portable topology/objective truth) + experiment/ # published experiment contracts (reward/evaluator/stochastic intent) + mapping/ # pinned CyberBattleSim → RAES source ledger + loss disclosures cyborg/ # CybORG backend module (optional `cyborg` extra; ADR-069 §3) mapping/ # pinned CAGE-2 → RAES source ledger (REP-003) profiles/ # conformance profile overrides @@ -95,6 +98,24 @@ semantics. The separate [architecture guardrails](docs/decisions/cyberbattlesim-qualification-guardrails.md) explain why this evidence is not an adapter manifest or RAES conformance claim. +Issue [#26](https://github.com/RAESystem/adapters/issues/26) authors the +portable evidence set for that case: an authored RAES SDL scenario +(`scenario/cyberbattle-chain.sdl.yaml`) that validates and compiles against +`raes==2.0.0`, companion published experiment contracts +(`experiment/`) for the reward, evaluator, metric, episode/termination, and +descriptive stochastic intent that RAES excludes from SDL, and a pinned +[source → RAES mapping ledger](src/raes_adapters/cyberbattlesim/mapping/) whose +rows are each `mapped`, `excluded`, or `loss-disclosed`, with every disclosed +loss bound to the ADR-069 equivalence tier it weakens. Deterministic tests fail +CI on source drift, a missing category, a duplicate row, an unresolvable target, +a broken cross-artifact reference, an undisclosed loss, or leakage of a known +native identifier or object representation (raw native arrays, reward vectors, +and action ids are excluded structurally by the closed RAES models). The +evidence set does not change +the `not-admissible` decision or claim installability, replay, or equivalence; +the [scenario/ledger guardrails](docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md) +fix its boundaries. + ## Development Requires [`uv`](https://docs.astral.sh/uv/). Repo-wide gates run through nox: diff --git a/docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md b/docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md new file mode 100644 index 0000000..955132b --- /dev/null +++ b/docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md @@ -0,0 +1,182 @@ +# CyberBattleSim scenario and source-ledger guardrails + +Issue #26 is the authority for the deliverable. This note fixes the repository +and contract boundaries the implementation must respect; it is not an +implementation plan and does not author the scenario or ledger. + +## Keep the artifacts and claims separate + +The deliverable is a package-local evidence set under +`raes_adapters.cyberbattlesim`, not one overloaded document: + +- `qualification.json` remains the source of truth for the selected repository, + commit, tree, archive and file digests, legal disposition, known defects, and + admissibility outcome. +- `public-protocol.md` remains source-native evidence for the selected + CyberBattleSim case. +- The authored RAES SDL carries portable scenario meaning only. +- Published RAES experiment contracts carry pre-run episode, evaluator, + metric, capture, and stochastic intent that RAES deliberately excludes from + SDL. +- The source ledger is backend-local traceability evidence. It explains how + pinned source facts reach those portable surfaces, or why they do not. +- Loss disclosures bound the strength of the applicable ADR-069 equivalence + tier; they are not substitute portable state. + +The issue's request for SDL covering rewards, termination, and stochastic +controls must therefore be read as a request for one portable scenario +artifact set. RAES 2.0.0 rejects SDL scoring/reward sections. Objective truth +may be authored in SDL, while numeric reward, evaluator metrics, episode +cutoffs, and random-stream controls belong in companion published experiment +contracts. + +The current qualification is `not-admissible`. Authoring and validating this +evidence set does not change that decision and must not be described as +installability, runnable adapter support, deterministic replay, backend +conformance, or scientific equivalence. + +## Reuse the published semantic owners + +| Source concern | Canonical portable owner | +| --- | --- | +| Fixed topology, nodes, services, reachability, vulnerabilities, identities, accounts, roles, and credential intent | Existing SDL `nodes`, `infrastructure`, `relationships`, `vulnerabilities`, `entities`, `accounts`, and related closed models | +| Generated topology choices | SDL `variables` and `variation_points`, followed by RAES instantiation; not a CyberBattleSim generator block or opaque `metadata` | +| Initial attacker position or ownership, defender detection/reimage behavior, and availability | SDL agents, propositions, assertions, objectives, action contracts, observation boundaries, behavior specifications, outcome interpretation, workflows, and evidence requirements as their source semantics require | +| Participant-visible actions and observations | SDL `action_contracts`, `observation_boundaries`, `behavior_specifications`, and `agents`; native action ids, masks, tuples, caches, and hidden state remain source-private | +| Objective satisfaction | SDL propositions, assertions, and objectives | +| Reward components, cumulative reward, evaluator policy, and study metrics | `ExperimentTaskModel.evaluation_protocol`, `ExperimentEvaluationProtocolModel`, and `ExperimentMetricDefinitionModel`; later run output uses evidence and derived-measure contracts | +| Turn order, source-native termination, evaluator cutoff, episode count, and selected participant variant | `ExperimentSpecModel.run_plan`, `ExperimentEpisodeControlModel`, and the published reference models it composes | +| Seeds and stochastic entry points | One `ExperimentStochasticControlModel` per source; add `RandomStreamControlBindingModel` only when an admitted executable profile truly governs that source | +| Captured observations and calculated results | `ExperimentCaptureSpecModel`, `ExperimentEvidenceRecordModel`, and `ExperimentDerivedMeasureModel`; do not fabricate run evidence during authoring | +| Repository, file, protocol, license, and notice provenance | Existing qualification resources and backend-local mapping evidence; never SDL fields or arbitrary experiment parameters | + +Do not use `SDLLineageLedgerModel` for this mapping. That published contract +records the ancestry of the RAES language and its normative artifacts, not the +provenance of one simulator-derived scenario. Likewise, do not mint a backend +manifest or participant implementation manifest before an implementation and +evidence-backed capabilities exist. + +## Source-ledger boundary + +Follow the existing CybORG directory's module-local ownership pattern, but do +not copy its CAGE-specific field names into a shared schema or generalize either +ledger into `raes_adapters.base`. + +The CyberBattleSim ledger is a closed, flat evidence record whose rows carry +only the minimum issue-required facts: a stable row identity, an immutable +source coordinate and selector, source category, one of `mapped`, `excluded`, +or `loss-disclosed`, resolvable RAES target references where applicable, the +mapping or exclusion rationale, verification evidence, and loss references +where applicable. This is package data, not a public DTO, profile, vocabulary, +manifest extension, or semantic contract. + +Validation must fail closed on all of the following: + +- malformed or unknown row fields, duplicate JSON keys, non-finite numbers, + blank identities, and duplicate row ids; +- a repository, commit, source path, or digest that does not join to the + canonical qualification record, and a local protocol digest that does not + match its recorded digest; +- absence of any issue-required category: topology/state, identities, + participants, actions, observations, controls, rewards/objectives, + termination, evaluator facts, stochastic policy, and provenance/licensing; +- a row without exactly one disposition, an excluded row without a rationale, + or a lossy row without a loss disclosure; +- a mapped target that does not resolve either to an actual validated artifact + path or to a contract id and property path in + `raes_contracts.contracts.schema_bundle()`; +- a loss not referenced by a source row, a lossy row without a referenced loss, + or a loss that does not name the ADR-069 equivalence tier it weakens; and +- any native action id, Gym tuple, raw object/state, credential value, reward + vector, hidden truth, simulator log, argument/environment dump, token, or full + traceback in portable content. + +Target validity must be derived from parsed artifacts and the published schema +bundle. Do not maintain a second list of RAES sections, contract ids, enums, or +JSON Schemas. The three row dispositions and the required coverage categories +are issue-local validation labels, not new RAES vocabulary. + +Source drift is an offline integrity join. CI validates rows against the +already-qualified immutable commit and file digests plus the checked-in protocol +digest; it must not fetch a floating branch, execute upstream code, or turn +network availability into a validation prerequisite. A newly consumed source +file must first become part of the reviewed qualification evidence. + +## Compose the existing validation graph + +The implementation must build on these incumbents: + +- `raes==2.0.0` in `pyproject.toml` and `uv.lock`; +- `raes.parse_sdl_file()` with default semantic validation and parser limits; +- RAES instantiation and canonical digest functions for each representative + fixed/generated assignment; +- `raes_processor.compiler.compile_scenario_runtime_model()` for compilation + without inventing a CyberBattleSim backend manifest; +- `ContractModel` (`extra="forbid"`), `parse_experiment_spec()`, and direct + published-model validation for companion experiment artifacts; +- `schema_bundle()` for published contract identity and target resolution; +- `load_qualification()`, `read_public_protocol()`, `importlib.resources`, and + the SHA-256 join pattern already exercised by + `tests/test_cyberbattlesim_qualification.py`; +- the existing `tests` session and canonical `nox -s verify` graph; and +- the existing wheel/sdist and clean-install proof in `_distributions()`. + +Ledger-only relational checks may be small module-local helpers or tests. They +must not become a Pydantic model, generated JSON Schema, schema registry, +standalone policy gate, second CI workflow, reusable exception hierarchy, or +parallel conformance harness. + +The environment-pack repository owns its own +`environment-pack-provenance/v3` licensing, safety, distribution, and +publication schema. This repository should expose immutable source and legal +facts as package resources; an environment pack maps those facts into its own +provenance ledger and validates them with `raes-pack-validate`. Neither +repository should import or duplicate the other's schema, and no runtime +dependency between the distributions is justified. + +## Cross-cutting security and operational boundary + +| Layer | Required treatment | +| --- | --- | +| Authentication/authorization | None is introduced: these are static local artifacts with no HTTP, control-plane caller, or runtime mutation surface. A later service must reuse RAES runtime strict defaults rather than this path. | +| Secret handling | SDL may express synthetic account and credential intent, never a real password, learned credential, credential cache, token, or secret reference value. The repository secret store is not an input. | +| Input shape and parsing | SDL passes through the bounded RAES YAML parser and semantic validator; experiment artifacts pass through closed RAES models; ledger parsing is strict UTF-8/JSONL with duplicate-key and unknown-field rejection. | +| Configuration and environment | The qualification selection is canonical. No environment variable may override source identity or semantics. Verification uses the frozen project lock and `--skip-requirement` for this requirement-free issue. | +| OS/process exposure | Validation runs in process over checked-in package resources. It performs no checkout-relative upstream import, shell interpolation, network fetch, home-cache write, or native simulator execution, and places no sensitive value in argv or environment output. | +| Errors and observability | Reuse RAES exceptions/diagnostics at their existing boundaries. Ledger failures identify only the row id and field/reason; they do not echo source payloads or native values. No new logger, portable log envelope, or full traceback artifact is needed. | +| Persistence | Checked-in immutable package data is the only durable state. Do not add a database, cache, repository service, `ControlPlaneStore`, or evidence store for static authoring evidence. | +| Distribution | Keep artifacts under the package tree so the existing Hatch wheel/sdist and clean-install checks prove their presence. Do not add another distribution, lockfile, optional extra, or dependency for this issue. | + +## Extension seam + +The validator's external parameter is the explicit module-local artifact root +or selection identity. Scenario size and the fixed/generated choice belong in +SDL variables/variation points and explicit instantiation inputs, not duplicated +Python branches. A second CyberBattleSim scenario should be addable as another +immutable selection using the same validator and published RAES contracts, +without editing a central simulator registry or the current selection's +evidence. + +## Gotchas, non-goals, and anti-patterns + +- Do not treat the source-native `size=10` topology, a generated topology + family, and an instantiated canonical snapshot as the same artifact. +- Do not publish learned credentials merely because SDL has an `accounts` + section; SDL records credential intent, while native discovered material + remains hidden. +- Do not turn the smoke reward `14.0`, winning reward `5000.0`, or a cumulative + reward into an SDL score, objective truth, study metric, or equivalence claim + without the explicit evaluator mapping. +- Do not collapse source termination, attacker ownership, defender SLA failure, + defender eviction, evaluator cutoff, Gym `terminated`, and Gym `truncated`. + Preserve source precedence and the known off-by-one limitation. +- Do not attach an executable random-stream binding to the four recorded + CyberBattleSim randomness sources while the public evaluator drops or omits + those bindings. Descriptive controls must retain that loss. +- Do not combine the historical benchmark snapshot with the newer selected + source commit as though they were one run. +- Do not add an adapter, simulator dependency/extra, backend or participant + manifest, runtime smoke, conformance profile, evidence run, environment pack, + persistence layer, or equivalence claim in issue #26. +- Do not change `raes_adapters.base`, `raes_adapters.cyborg`, the RAES contract + corpus, or environment-pack schemas for mapping convenience. diff --git a/docs/index.md b/docs/index.md index 967a447..efbe7c2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -16,6 +16,7 @@ semantic and protocol authority. - [Contribution guide](https://github.com/RAESystem/adapters/blob/dev/CONTRIBUTING.md) - [Architecture decisions](decisions/adrs/README.md) - [CyberBattleSim qualification guardrails](decisions/cyberbattlesim-qualification-guardrails.md) +- [CyberBattleSim scenario and source-ledger guardrails](decisions/cyberbattlesim-scenario-ledger-guardrails.md) - [Project services](maintainers/project-services.md) ## Verify a checkout diff --git a/mkdocs.yml b/mkdocs.yml index b14341a..c3e48cd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -37,6 +37,7 @@ nav: - Overview: decisions/adrs/README.md - Template: decisions/adrs/TEMPLATE.md - CyberBattleSim qualification guardrails: decisions/cyberbattlesim-qualification-guardrails.md + - CyberBattleSim scenario and source-ledger guardrails: decisions/cyberbattlesim-scenario-ledger-guardrails.md - ADR 003 — Single distribution and Trusted Publishing: decisions/adrs/adr-003-single-distribution-and-trusted-publishing.md - ADR 002 — RAES authority and adapter boundaries: decisions/adrs/adr-002-raes-authority-and-adapter-boundaries.md - ADR 000 — Use ADRs (superseded): decisions/adrs/adr-000-use-adrs.md diff --git a/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.spec.exp.yaml b/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.spec.exp.yaml new file mode 100644 index 0000000..d762977 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.spec.exp.yaml @@ -0,0 +1,62 @@ +schema_version: experiment-authoring-input/v1 +spec_id: cyberbattlesim-chain-spec +spec_version: 1.0.0 +title: CyberBattleChain-v0 public run plan +description: >- + Authoring input for the selected public CyberBattleSim protocol. Carries the + episode/turn/termination control and the four descriptive stochastic controls + that RAES SDL excludes. The stochastic controls are descriptive only: the + public evaluator drops the seed bindings, so no executable random-stream + binding is claimed and the determinism loss is retained. +task_ref: + ref_kind: task + ref_id: cyberbattlesim-chain-task +intended_scenario_ref: + ref_kind: scenario-snapshot + ref_id: cyberbattlesim-chain + ref_digest: sha256:9d696ea7fa23a1e7cf4c1cbc145a7989370dc4e9afff2cd5a17d6d1af887b528 + ref_path: scenario/cyberbattle-chain.sdl.yaml +run_plan: + episode_control: + turn_order: sequential + termination_rule: >- + After each attacker and defender step, evaluate in precedence order: + (1) attacker goal reached or defender SLA broken -> terminate with the + winning reward; (2) else defender eviction goal reached -> terminate with + the losing reward; (3) else continue. Gym always returns truncated=false; + the evaluator iteration cutoff is not Gym truncation. + max_steps: 600 + termination_condition_refs: + - attacker-ownership + - defender-sla + - defender-eviction + - evaluator-cutoff + description: >- + Source-native termination causes stay distinct from the evaluator cutoff + and from Gym terminated/truncated. + stochastic_controls: + - control_id: gym-environment + role: seed + value: "20260729" + description: >- + CyberBattleEnv.reset(seed=20260729). Lost: the public evaluator calls + reset without forwarding a seed. Descriptive only; no executable binding. + - control_id: gym-action-space + role: sampling + value: "20260729" + description: >- + env.action_space.seed(20260729). Not bound by the public notebook. + Descriptive only; no executable binding. + - control_id: python-random + role: randomization + value: "20260729" + description: >- + random.seed(20260729) for the defender's Python random calls. Not bound + by the public notebook. Descriptive only; no executable binding. + - control_id: numpy-global + role: randomization + value: "20260729" + description: >- + numpy.random.seed(20260729) for defender and baseline NumPy calls. Not + bound by the public notebook. Descriptive only; no executable binding. + target_run_count: 10 diff --git a/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.task.exp.yaml b/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.task.exp.yaml new file mode 100644 index 0000000..835b8e1 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/experiment/cyberbattle-chain.task.exp.yaml @@ -0,0 +1,125 @@ +schema_version: experiment-task/v1 +task_id: cyberbattlesim-chain-task +task_version: 1.0.0 +title: CyberBattleChain-v0 credential-cache evaluation task +description: >- + Published evaluation-protocol contract for the selected public CyberBattleSim + case (CyberBattleChain-v0, size=10, CredentialCacheExploiter attacker, + ScanAndReimage defender). Carries the reward, evaluator, and metric intent + that RAES SDL deliberately excludes. Native reward vectors, action ids, and + observations remain source-private; only bounded measure definitions are + portable. +scenario_ref: + ref_kind: scenario-snapshot + ref_id: cyberbattlesim-chain + ref_digest: sha256:9d696ea7fa23a1e7cf4c1cbc145a7989370dc4e9afff2cd5a17d6d1af887b528 + ref_path: scenario/cyberbattle-chain.sdl.yaml +evaluation_protocol: + protocol_id: cyberbattlesim-chain-protocol + protocol_version: 1.0.0 + intent: >- + Measure attacker progress, reward, availability, and terminal cause for the + selected public protocol without asserting reproducibility or admissibility. + unit_of_analysis: episode + metric_definitions: + steps_to_termination: + metric_id: steps_to_termination + metric_version: 1.0.0 + name: Steps to termination + measured_construct: >- + Attacker steps to source-native termination, or 600 when the evaluator + iteration cutoff fires first. + unit_of_analysis: episode + value_kind: count + direction: descriptive + evidence_requirements: + - ref_kind: evidence + ref_id: attacker-action-log + cumulative_attacker_reward: + metric_id: cumulative_attacker_reward + metric_version: 1.0.0 + name: Cumulative attacker reward + measured_construct: >- + Per-episode cumulative attacker reward. Source-native winning reward is + 5000.0 and losing reward is 0.0; these constants are the evaluator's, + not an SDL objective score or equivalence claim. + unit_of_analysis: episode + value_kind: number + direction: higher-is-better + evidence_requirements: + - ref_kind: evidence + ref_id: attacker-action-log + network_availability: + metric_id: network_availability + metric_version: 1.0.0 + name: Network availability + measured_construct: >- + Per-step network availability series; the defender maintains a 0.80 SLA + floor as a source-native termination constraint. + unit_of_analysis: episode + value_kind: number + direction: target + evidence_requirements: + - ref_kind: evidence + ref_id: availability-series + terminal_cause: + metric_id: terminal_cause + metric_version: 1.0.0 + name: Terminal cause + measured_construct: >- + Reconstructed terminal cause, one of attacker-ownership, defender-sla, + defender-eviction, or evaluator-cutoff. These causes stay distinct from + Gym terminated/truncated and from the off-by-one reward-goal predicate. + unit_of_analysis: episode + value_kind: category + direction: descriptive + evidence_requirements: + - ref_kind: evidence + ref_id: attacker-action-log + observation_requirements: + - ref_kind: evidence + ref_id: attacker-action-log + - ref_kind: evidence + ref_id: availability-series +intended_use: >- + Define the portable evaluation protocol for the selected public CyberBattleSim + case so a future admitted run can be scored consistently. +population_or_construct: >- + A single credential-cache baseline attacker against a basic scan-and-reimage + defender on the fixed CyberBattleChain-v0 topology. +split_and_leakage_controls: + partitioning_strategy: >- + Single held-out public protocol run; the historical benchmark snapshot is + excluded and never combined with the selected source commit. +apparatus_constraints: + notes: + - >- + No CyberBattleSim adapter, backend, or participant manifest is implemented. + The case is not-admissible to RAESystem/research#12 (no index/release + artifact, incomplete random-stream binding, unresolved benchmark defects). +validity_notes: + - category: construct + note: >- + Cumulative reward is an evaluator signal, not the study metric; the source + reward-goal predicate reads prior-step rewards (off-by-one limitation). + - category: internal + note: >- + The public evaluator resets without a seed and does not bind the + action-space, Python random, or global NumPy streams, so no run under this + protocol is a deterministic replay. + - category: external + note: >- + Upstream benchmark/evaluation defects (microsoft/CyberBattleSim #87, #115, + #117, #156) remain open, weakening outcome comparability. +artifact_refs: + - artifact_id: cyberbattlesim-public-protocol + role: protocol + media_type: text/markdown + uri: file://public-protocol.md + checksum: + algorithm: sha256 + value: e20b197f48100f0c33b4b7ae55d02bbace8fc2da5afe6552b14566648371e4c6 + size_bytes: 6804 + created_at: "2026-07-29T00:00:00Z" + source: RAESystem/adapters#25 qualification + sensitivity: public diff --git a/src/raes_adapters/cyberbattlesim/mapping/README.md b/src/raes_adapters/cyberbattlesim/mapping/README.md new file mode 100644 index 0000000..24cf6a5 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/mapping/README.md @@ -0,0 +1,61 @@ +# CyberBattleSim → RAES mapping ledger + +**Authored under RAESystem/adapters#26.** This directory is backend-local +traceability evidence for the selected public CyberBattleSim case. It records +how each pinned upstream source fact reaches a portable RAES surface — the +authored SDL scenario or a published experiment contract — or why it does not. +It follows the CybORG mapping directory's module-local ownership *pattern*; it +does not copy CAGE field names, and neither ledger is generalized into +`raes_adapters.base`. + +Per RAES ADR-069 and `docs/decisions/cyberbattlesim-scenario-ledger-guardrails.md`, +the mapping is a **pinned ledger** over the immutable source identifiers already +recorded in [`../qualification.json`](../qualification.json). It is package data, +not a public DTO, schema, profile, vocabulary, or semantic contract. + +## Files + +- `source-ledger.jsonl` — one JSON object per source fact. Fields: + - `source_id` — stable, unique row identity. + - `source_repo`, `source_commit`, `source_path`, `source_selector` — immutable + upstream coordinate and the symbol/path the fact comes from. + - `source_digest` (optional) — the file SHA-256; when present it must join the + matching `qualification.json` `source_files` entry (offline source-drift + check — CI never fetches a floating branch or executes CyberBattleSim). + - `category` — one of the required coverage categories (see below). + - `disposition` — exactly one of `mapped`, `excluded`, `loss-disclosed`. + - `raes_target` (mapped rows) — a portable surface reference that must resolve + to an authored artifact element (`sdl:
[.]`, + `resource:`) or a published contract property path + (`contract:.` in + `raes_contracts.contracts.schema_bundle()`). No second target catalog is + maintained. + - `loss_ref` + `equivalence_tier` (loss-disclosed rows) — the disclosure id in + `loss-disclosures.md` and the ADR-069 tier it weakens. + - `mapping_rule` — the mapping or exclusion rationale. + - `verification` — how the fact was checked. +- `loss-disclosures.md` — narrative disclosures for source facts RAES cannot + carry exactly, each bound to the equivalence tier it weakens. + +## Required coverage categories + +Every category must appear at least once, or CI fails: `topology-state`, +`identities`, `participants`, `actions`, `observations`, `controls`, +`rewards-objectives`, `termination`, `evaluator`, `stochastic`, and +`provenance-licensing`. + +## Governance + +Every upstream source fact is `mapped`, explicitly `excluded`, or +`loss-disclosed`. Native simulator state — action ids, Gym tuples, raw +observations, credential caches, reward vectors, hidden truth, and logs — stays +`excluded` and source-private and never enters a portable artifact. That +boundary is enforced two ways: the closed RAES SDL and experiment models have no +field to carry raw native arrays, reward vectors, or action ids (a structural +exclusion), and a deterministic scan rejects known native identifiers (grounded +in the qualification record's recorded observation keys) and native object/array +representations in portable content. No claim rests on CI success, a single +cumulative score, or a top-level seed. The current +qualification is `not-admissible`; authoring and validating this evidence set +does not change that and asserts no installability, replay, conformance, or +equivalence. diff --git a/src/raes_adapters/cyberbattlesim/mapping/loss-disclosures.md b/src/raes_adapters/cyberbattlesim/mapping/loss-disclosures.md new file mode 100644 index 0000000..99c674d --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/mapping/loss-disclosures.md @@ -0,0 +1,72 @@ +# CyberBattleSim → RAES loss disclosures + +**Authored under RAESystem/adapters#26.** Each entry records a pinned +CyberBattleSim source fact that the portable RAES evidence set cannot carry +exactly, the reason, and the ADR-069 equivalence tier it weakens (RAES ADR-069 +§2, §7). A disclosed gap weakens or fails that tier explicitly; it is never +filled by raw simulator logs, native observations, fixture-local assertions, or +prose-only evidence. + +The equivalence tiers named below are issue-local validation labels for the +ADR-069 tiered-equivalence ladder — the same status as the ledger's disposition +and category labels — not a new RAES vocabulary. Every `loss-disclosed` row in +`source-ledger.jsonl` references one of these losses and repeats its tier; the +deterministic validator fails closed when a loss is unreferenced, a lossy row +names no loss, or a loss declares no tier. + +## loss-no-source-artifact + +**Equivalence tier:** reproducibility + +The official Microsoft source declares `cyberbattlesim==0.1.0` but publishes +neither a public index distribution nor a release artifact, and PyPI does not +accept direct-URL dependencies. The exact selected commit therefore cannot be +reinstalled through a published, hash-locked route, so no `cyberbattlesim` extra +is advertised. This defeats the base reproducibility precondition on which every +higher equivalence tier depends: an apparatus that cannot be reconstructed from a +published artifact cannot ground a portable replication claim. The gap is a +standing admissibility blocker for RAESystem/research#12, not a packaging +convenience to be papered over with a floating Git reference or an empty extra. + +## loss-unbound-random-streams + +**Equivalence tier:** deterministic-replay + +The selected public case has four randomness sources — the Gym environment +generator, the Gym action space, Python `random`, and global NumPy. The public +`epsilon_greedy_search` evaluator calls `reset()` without forwarding a seed and +the public notebook binds none of the other three. A single top-level seed is +therefore not evidence that the run is controlled, and the published stochastic +controls remain **descriptive only** (no `RandomStreamControlBindingModel`). No +run under this protocol is a deterministic replay until the upstream seam is +fixed or an explicit, identified compatibility patch is qualified. + +## loss-abstracted-topology + +**Equivalence tier:** outcome-equivalence + +The selected source case is `CyberBattleChain-v0` at generation size 10, produced +by `new_environment(10)`. RAES SDL is declarative and has no construct that +structurally materializes a generator's node graph, so the authored scenario +encodes the chain *pattern and host roles* as a bounded representative topology +(an entry host, alternating Linux/Windows relay hosts, and a terminal +high-value host) rather than a node-for-node expansion of the size-10 case. The +selected size is recorded as the `chain_size` SDL variable and in the companion +run plan, but the exact per-host structure of the generated graph is not +reproduced. This weakens outcome comparability between a run on this +representative topology and a run on the full generated chain, so no outcome +equivalence is claimed from topology alone. + +## loss-benchmark-defects + +**Equivalence tier:** outcome-equivalence + +Material upstream benchmark and evaluation defects remain open +(microsoft/CyberBattleSim #87, #115, #117, #156), the attacker reward-goal +predicate reads prior-step rewards (an off-by-one limitation), and the evaluator +iteration cutoff is not Gym truncation (Gym always returns `truncated=false`). +These weaken outcome comparability: cumulative reward, terminal cause, and the +historical benchmark snapshot cannot be treated as equivalent study outcomes of +the selected commit. The portable evidence keeps reward, terminal result, and +derived measures distinct and never converts the ownership goal into a silent +reward threshold. diff --git a/src/raes_adapters/cyberbattlesim/mapping/source-ledger.jsonl b/src/raes_adapters/cyberbattlesim/mapping/source-ledger.jsonl new file mode 100644 index 0000000..f262b6f --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/mapping/source-ledger.jsonl @@ -0,0 +1,29 @@ +{"source_id": "topology-chain-pattern", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/samples/chainpattern/chainpattern.py", "source_selector": "ChainPattern.new_environment", "source_digest": "b13973ff36f2246a51c405c9d57c05f8ce448257f179ca13c68dd5d4926b4d12", "category": "topology-state", "disposition": "mapped", "raes_target": "sdl:nodes", "mapping_rule": "The linear chain of alternating Linux/Windows hosts and the terminal data host become authored SDL nodes with services and asset value.", "verification": "digest joins qualification.source_files; sdl:nodes resolves after parse_sdl."} +{"source_id": "topology-environment", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_chain.py", "source_selector": "CyberBattleChain environment", "source_digest": "6c15457a6659dc883a4eb35cb454f72c9c339751d38843ada3cec5902ad11e43", "category": "topology-state", "disposition": "mapped", "raes_target": "sdl:infrastructure", "mapping_rule": "The CyberBattleChain environment's flat subnet becomes the SDL enterprise-lan switch infrastructure entry.", "verification": "digest joins qualification.source_files; sdl:infrastructure resolves after parse_sdl."} +{"source_id": "topology-reachability", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/samples/chainpattern/chainpattern.py", "source_selector": "chain adjacency edges", "source_digest": "b13973ff36f2246a51c405c9d57c05f8ce448257f179ca13c68dd5d4926b4d12", "category": "topology-state", "disposition": "mapped", "raes_target": "sdl:relationships", "mapping_rule": "Chain adjacency (each host reaches the next) becomes SDL connects_to relationships.", "verification": "digest joins qualification.source_files; sdl:relationships resolves after parse_sdl."} +{"source_id": "vulnerability-model", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/samples/chainpattern/chainpattern.py", "source_selector": "chain vulnerability library", "source_digest": "b13973ff36f2246a51c405c9d57c05f8ce448257f179ca13c68dd5d4926b4d12", "category": "topology-state", "disposition": "mapped", "raes_target": "sdl:vulnerabilities", "mapping_rule": "The chain's credential-leak and access-control vulnerabilities become CWE-classed SDL vulnerabilities.", "verification": "digest joins qualification.source_files; sdl:vulnerabilities resolves after parse_sdl."} +{"source_id": "topology-abstraction", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/samples/chainpattern/chainpattern.py", "source_selector": "new_environment(size) node generation", "source_digest": "b13973ff36f2246a51c405c9d57c05f8ce448257f179ca13c68dd5d4926b4d12", "category": "topology-state", "disposition": "loss-disclosed", "loss_ref": "loss-abstracted-topology", "equivalence_tier": "outcome-equivalence", "mapping_rule": "RAES SDL encodes the chain pattern and host roles as a bounded representative topology; the generated size-10 node graph is not structurally materialized.", "verification": "digest joins qualification.source_files; loss-abstracted-topology declared with tier outcome-equivalence."} +{"source_id": "credential-intent", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/samples/chainpattern/chainpattern.py", "source_selector": "leaked credential entries", "source_digest": "b13973ff36f2246a51c405c9d57c05f8ce448257f179ca13c68dd5d4926b4d12", "category": "identities", "disposition": "mapped", "raes_target": "sdl:accounts", "mapping_rule": "The per-host service credentials leaked along the chain become synthetic SDL account intent; no learned credential value is carried.", "verification": "digest joins qualification.source_files; sdl:accounts resolves after parse_sdl."} +{"source_id": "credential-cache-native", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "CyberBattleEnv credential cache", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "identities", "disposition": "excluded", "mapping_rule": "The native credential cache and its observation matrix are simulator-private discovered material and never enter a portable artifact.", "verification": "no raes_target; native leakage scan asserts the field names are absent from portable artifacts."} +{"source_id": "participant-attacker", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/agents/baseline/agent_randomcredlookup.py", "source_selector": "CredentialCacheExploiter", "source_digest": "ea69bf6f79bbac328ea8b531167e7785c6222fb359617b7e361f522ebb80a741", "category": "participants", "disposition": "mapped", "raes_target": "sdl:agents.attacker", "mapping_rule": "The CredentialCacheExploiter baseline becomes the SDL attacker agent bound to the red-team entity.", "verification": "digest joins qualification.source_files; sdl:agents.attacker resolves after parse_sdl."} +{"source_id": "participant-defender", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/defender.py", "source_selector": "ScanAndReimageCompromisedMachines", "source_digest": "d83abc58b30c20d25f06298d1230b554f72c8f7a97c7dbb6634e6a2a78033074", "category": "participants", "disposition": "mapped", "raes_target": "sdl:agents.defender", "mapping_rule": "ScanAndReimageCompromisedMachines becomes the SDL defender agent with a scan-and-reimage action contract.", "verification": "digest joins qualification.source_files; sdl:agents.defender resolves after parse_sdl."} +{"source_id": "actions-portable", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "local_vulnerability/remote_vulnerability/connect", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "actions", "disposition": "mapped", "raes_target": "sdl:action_contracts", "mapping_rule": "The local-vulnerability, remote-vulnerability, and connect action kinds become SDL action contracts with preconditions, effects, and failure classes.", "verification": "digest joins qualification.source_files; sdl:action_contracts resolves after parse_sdl."} +{"source_id": "actions-native-ids", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "action space and action mask", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "actions", "disposition": "excluded", "mapping_rule": "Native action identifiers, the action mask, and Gym action tuples are simulator-private and never enter a portable artifact.", "verification": "no raes_target; native leakage scan asserts these constructs are absent from portable artifacts."} +{"source_id": "observations-boundary", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "observation space projection", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "observations", "disposition": "mapped", "raes_target": "sdl:observation_boundaries", "mapping_rule": "The attacker and defender partial-observability views become SDL observation boundaries; only bounded projection basis and redaction policy are portable.", "verification": "digest joins qualification.source_files; sdl:observation_boundaries resolves after parse_sdl."} +{"source_id": "observations-native-dict", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "observation dictionary", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "observations", "disposition": "excluded", "mapping_rule": "The native observation dictionary, explored-network graph, and privilege matrices are simulator-private and never enter a portable artifact.", "verification": "no raes_target; native leakage scan asserts the native observation key names are absent from portable artifacts."} +{"source_id": "control-availability", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/defender.py", "source_selector": "defender availability behavior", "source_digest": "d83abc58b30c20d25f06298d1230b554f72c8f7a97c7dbb6634e6a2a78033074", "category": "controls", "disposition": "mapped", "raes_target": "sdl:objectives.maintain-availability", "mapping_rule": "The defender's availability-preservation behavior becomes the SDL maintain-availability objective and sla-maintained proposition.", "verification": "digest joins qualification.source_files; sdl:objectives.maintain-availability resolves after parse_sdl."} +{"source_id": "control-sla-threshold", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "defender_constraint.maintain_sla", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "controls", "disposition": "mapped", "raes_target": "contract:ExperimentSpecModel.run_plan.episode_control", "mapping_rule": "The numeric 0.80 maintain-SLA termination constraint is an evaluator control recorded in the published run-plan episode control, not an SDL score.", "verification": "digest joins qualification.source_files; target resolves in schema_bundle()."} +{"source_id": "objective-ownership", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "attacker_goal.own_atleast_percent", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "rewards-objectives", "disposition": "mapped", "raes_target": "sdl:objectives.own-network", "mapping_rule": "The attacker own-at-least-100-percent ownership goal becomes the SDL own-network objective and network-owned proposition.", "verification": "digest joins qualification.source_files; sdl:objectives.own-network resolves after parse_sdl."} +{"source_id": "reward-constants", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "winning_reward/losing_reward", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "rewards-objectives", "disposition": "mapped", "raes_target": "contract:ExperimentTaskModel.evaluation_protocol.metric_definitions", "mapping_rule": "The winning (5000.0) and losing (0.0) reward constants and cumulative reward become the published cumulative-attacker-reward metric definition, not an SDL objective score.", "verification": "digest joins qualification.source_files; target resolves in schema_bundle()."} +{"source_id": "reward-goal-off-by-one", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "attacker goal reward predicate", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "rewards-objectives", "disposition": "loss-disclosed", "loss_ref": "loss-benchmark-defects", "equivalence_tier": "outcome-equivalence", "mapping_rule": "The reward-goal predicate reads prior-step rewards (off-by-one); the portable evidence does not silently convert the goal to a reward threshold.", "verification": "digest joins qualification.source_files; loss-benchmark-defects declared with tier outcome-equivalence."} +{"source_id": "termination-precedence", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "step termination checks", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "termination", "disposition": "mapped", "raes_target": "contract:ExperimentSpecModel.run_plan.episode_control", "mapping_rule": "The attacker-goal / defender-SLA / defender-eviction precedence and winning/losing reward replacement become the published episode-control termination rule.", "verification": "digest joins qualification.source_files; target resolves in schema_bundle()."} +{"source_id": "termination-cutoff-not-truncation", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "truncated flag", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "termination", "disposition": "loss-disclosed", "loss_ref": "loss-benchmark-defects", "equivalence_tier": "outcome-equivalence", "mapping_rule": "The evaluator iteration cutoff is not Gym truncation (Gym always returns truncated=false); conflating them would misreport termination cause.", "verification": "digest joins qualification.source_files; loss-benchmark-defects declared with tier outcome-equivalence."} +{"source_id": "evaluator-policy", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/agents/baseline/learner.py", "source_selector": "epsilon_greedy_search", "source_digest": "1cb04dc62f19924453a037bcfc2b4ce375857396010b6b84036c8fff9f830712", "category": "evaluator", "disposition": "mapped", "raes_target": "contract:ExperimentTaskModel.evaluation_protocol", "mapping_rule": "The epsilon_greedy_search evaluator and its metric intent become the published evaluation protocol.", "verification": "digest joins qualification.source_files; target resolves in schema_bundle()."} +{"source_id": "evaluator-run-plan", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/agents/baseline/learner.py", "source_selector": "epsilon_greedy_search episode/iteration bounds", "source_digest": "1cb04dc62f19924453a037bcfc2b4ce375857396010b6b84036c8fff9f830712", "category": "evaluator", "disposition": "mapped", "raes_target": "contract:ExperimentSpecModel.run_plan", "mapping_rule": "The episode_count (10) and iteration_count (600) evaluator loop bounds become the published run plan's target run count and episode-control cutoff.", "verification": "digest joins qualification.source_files; target resolves in schema_bundle()."} +{"source_id": "benchmark-snapshot", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "docs/benchmark.md", "source_selector": "benchmark results table", "source_digest": "0462976356adbaad02bfc0ef125ec286d56065986d887e389b8b9419d40be6d1", "category": "evaluator", "disposition": "excluded", "mapping_rule": "The historical benchmark snapshot predates the selected source commit and is not combined with it as though one run; it is not portable output of this selection.", "verification": "digest joins qualification.source_files; qualification.benchmark_snapshot disposition historical-output-only."} +{"source_id": "stochastic-controls", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/_env/cyberbattle_env.py", "source_selector": "reset/action_space/random/numpy seeds", "source_digest": "8369a1784e6ef77796766bd336c2ed7b3f3136e61c89d54903f9742372666cc9", "category": "stochastic", "disposition": "mapped", "raes_target": "contract:ExperimentSpecModel.run_plan.stochastic_controls", "mapping_rule": "The gym-environment, gym-action-space, python-random, and numpy-global sources with the public seed 20260729 become descriptive published stochastic controls.", "verification": "digest joins qualification.source_files; target resolves in schema_bundle()."} +{"source_id": "stochastic-binding-loss", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/agents/baseline/learner.py", "source_selector": "epsilon_greedy_search reset without seed", "source_digest": "1cb04dc62f19924453a037bcfc2b4ce375857396010b6b84036c8fff9f830712", "category": "stochastic", "disposition": "loss-disclosed", "loss_ref": "loss-unbound-random-streams", "equivalence_tier": "deterministic-replay", "mapping_rule": "The public evaluator calls reset without forwarding a seed and binds none of the four streams, so the stochastic controls stay descriptive with no executable binding.", "verification": "digest joins qualification.source_files; loss-unbound-random-streams declared with tier deterministic-replay."} +{"source_id": "provenance-source-identity", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "setup.py", "source_selector": "setup name and version", "source_digest": "69af36bcb04450d6dd8f71f23b9340288cbb461021128915eacd28408647c949", "category": "provenance-licensing", "disposition": "mapped", "raes_target": "resource:qualification.json", "mapping_rule": "The distribution name and version (cyberbattlesim 0.1.0) and immutable commit/tree/archive identity are owned by the qualification record.", "verification": "digest joins qualification.source_files; resource:qualification.json present in package."} +{"source_id": "provenance-license", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "LICENSE", "source_selector": "MIT license text", "source_digest": "3d4ada4e04d153d74f5bc4e5e6aebd12ef20077716529c36089dc07f14fd0dcf", "category": "provenance-licensing", "disposition": "mapped", "raes_target": "resource:qualification.json", "mapping_rule": "The MIT license and redistribution/attribution duties are owned by the qualification record's legal disposition.", "verification": "digest joins qualification.source_files; resource:qualification.json present in package."} +{"source_id": "provenance-notice", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "cyberbattle/NOTICE", "source_selector": "NOTICE third-party attributions", "source_digest": "5fc64ab06837e47cb22c6fbbad556734947ec0c4af6eb0d2700668447b63d01b", "category": "provenance-licensing", "disposition": "mapped", "raes_target": "resource:qualification.json", "mapping_rule": "The bundled NOTICE and its third-party attribution components are owned by the qualification record's legal disposition.", "verification": "digest joins qualification.source_files; resource:qualification.json present in package."} +{"source_id": "admissibility-no-artifact", "source_repo": "https://github.com/microsoft/CyberBattleSim", "source_commit": "854d6966607fb68645651f55b0f97221bd293e0d", "source_path": "requirements.txt", "source_selector": "install requirements", "source_digest": "76f9e396a2c21af090844e756b2aa965f5eda3f15537daa70bc135e365d9c494", "category": "provenance-licensing", "disposition": "loss-disclosed", "loss_ref": "loss-no-source-artifact", "equivalence_tier": "reproducibility", "mapping_rule": "The official source declares cyberbattlesim 0.1.0 but publishes no index distribution or release artifact, so the exact source cannot be portably reinstalled and no cyberbattlesim extra is published.", "verification": "digest joins qualification.source_files; loss-no-source-artifact declared with tier reproducibility."} diff --git a/src/raes_adapters/cyberbattlesim/scenario/cyberbattle-chain.sdl.yaml b/src/raes_adapters/cyberbattlesim/scenario/cyberbattle-chain.sdl.yaml new file mode 100644 index 0000000..c6ff973 --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/scenario/cyberbattle-chain.sdl.yaml @@ -0,0 +1,364 @@ +name: cyberbattlesim-chain +version: "1.0.0" +description: >- + Portable RAES model of the selected CyberBattleSim CyberBattleChain-v0 case + (Microsoft CyberBattleSim @ 854d696). It encodes the chain PATTERN and host + roles - an attacker entry host, alternating Linux/Windows relay hosts that + leak credentials toward a terminal high-value host, and a scan-and-reimage + defender - as a bounded representative topology. The selected source size is + ${chain_size}, recorded here as the generation parameter and in the companion + run plan; RAES SDL declares that parameter but has no construct to structurally + materialize the generated node graph, so this scenario is not a node-for-node + expansion of the size-10 case (disclosed as loss-abstracted-topology in the + source ledger). Objective and topology truth only; reward, evaluator, + termination, and stochastic controls live in companion published experiment + contracts, and native simulator state stays source-private. + +variables: + chain_size: + type: integer + default: 10 + description: >- + Selected CyberBattleChain generation parameter (new_environment size). + Recorded as the selected value; RAES SDL does not structurally expand it + into materialized nodes (see the loss-abstracted-topology disclosure). + +vulnerabilities: + leaked-credentials: + name: "Credential leak enabling lateral movement" + description: >- + A chain host exposes credentials for the next host, modeling the + CyberBattleChain credential-cache leak that the attacker exploits. + class: "CWE-522" + technical: true + improper-access-control: + name: "Improper access control on relay service" + description: >- + A relay host accepts a leaked credential without adequate authorization, + allowing the attacker to own the next node in the chain. + class: "CWE-284" + technical: true + +infrastructure: + enterprise-lan: + count: 1 + +nodes: + entry-client: + type: vm + description: "Attacker's initial foothold at the head of the chain." + os: linux + os_version: "22.04" + resources: + ram: "2 GB" + cpu: 2 + services: + - port: 22 + protocol: tcp + name: ssh + description: "Administrative shell service." + asset_value: + confidentiality: low + integrity: low + availability: medium + linux-relay: + type: vm + description: "Linux relay host midway along the chain." + os: linux + os_version: "22.04" + resources: + ram: "2 GB" + cpu: 2 + services: + - port: 22 + protocol: tcp + name: ssh + description: "Administrative shell service." + vulnerabilities: + - leaked-credentials + asset_value: + confidentiality: medium + integrity: medium + availability: medium + windows-relay: + type: vm + description: "Windows relay host along the chain." + os: windows + os_version: "2022" + resources: + ram: "4 GB" + cpu: 2 + services: + - port: 3389 + protocol: tcp + name: rdp + description: "Remote desktop service." + vulnerabilities: + - improper-access-control + asset_value: + confidentiality: medium + integrity: medium + availability: medium + customer-data: + type: vm + description: "Terminal high-value host holding the objective data." + os: linux + os_version: "22.04" + resources: + ram: "4 GB" + cpu: 4 + services: + - port: 443 + protocol: tcp + name: https + description: "Customer data web service." + vulnerabilities: + - leaked-credentials + asset_value: + confidentiality: critical + integrity: high + availability: high + enterprise-lan: + type: switch + description: "Flat enterprise subnet carrying the CyberBattleChain hosts." + +relationships: + entry-to-linux: + type: connects_to + source: entry-client + target: linux-relay + description: "Chain reachability from the entry client to the first relay." + linux-to-windows: + type: connects_to + source: linux-relay + target: windows-relay + description: "Chain reachability from the Linux relay to the Windows relay." + windows-to-customer: + type: connects_to + source: windows-relay + target: customer-data + description: "Chain reachability from the Windows relay to the terminal host." + +entities: + red-team: + description: "Autonomous credential-cache attacker (CredentialCacheExploiter)." + role: red + blue-team: + description: "Scan-and-reimage defender maintaining an availability SLA." + role: blue + +accounts: + relay-service: + username: "svc_relay" + node: linux-relay + password_strength: weak + groups: + - "operators" + windows-service: + username: "svc_win" + node: windows-relay + password_strength: weak + groups: + - "operators" + +action_contracts: + local-vulnerability: + semantic_version: "1.0.0" + behavioral_granularity: atomic + procedure_basis: "CyberBattleSim local_vulnerability action semantics." + realization_profile: "portable-scenario/v1" + fidelity_claim: "abstract portable projection of the source action." + preconditions: + - precondition_id: pre-owns-host + precondition_class: authority + description: "Attacker already owns the acting host." + effects: + - effect_id: eff-leak-local + effect_class: intended_effect + description: "Local exploitation reveals credentials cached on the host." + target_refs: + - linux-relay + failure_classes: + - target_unavailable + - authority_denied + remote-vulnerability: + semantic_version: "1.0.0" + behavioral_granularity: atomic + procedure_basis: "CyberBattleSim remote_vulnerability action semantics." + realization_profile: "portable-scenario/v1" + fidelity_claim: "abstract portable projection of the source action." + preconditions: + - precondition_id: pre-reachable + precondition_class: target + description: "Target host is reachable from an owned host." + effects: + - effect_id: eff-leak-remote + effect_class: intended_effect + description: "Remote exploitation reveals credentials for an adjacent host." + target_refs: + - windows-relay + failure_classes: + - target_unavailable + - authority_denied + connect: + semantic_version: "1.0.0" + behavioral_granularity: atomic + procedure_basis: "CyberBattleSim connect-and-infect action semantics." + realization_profile: "portable-scenario/v1" + fidelity_claim: "abstract portable projection of the source action." + preconditions: + - precondition_id: pre-has-credential + precondition_class: knowledge + description: "Attacker holds a valid credential for the target service." + effects: + - effect_id: eff-own-next + effect_class: intended_effect + description: "Connecting with a leaked credential takes ownership of the target." + target_refs: + - customer-data + failure_classes: + - authority_denied + - target_unavailable + scan-and-reimage: + semantic_version: "1.0.0" + behavioral_granularity: procedure + procedure_basis: "CyberBattleSim ScanAndReimageCompromisedMachines defender." + realization_profile: "portable-scenario/v1" + fidelity_claim: "abstract portable projection of the defender behavior." + preconditions: + - precondition_id: pre-detected + precondition_class: interaction + description: "Defender scan detects a compromised host." + effects: + - effect_id: eff-reimage + effect_class: intended_effect + description: "Compromised host is reimaged, evicting the attacker." + target_refs: + - linux-relay + failure_classes: + - target_unavailable + - unsupported_action + +observation_boundaries: + attacker-view: + projection_basis: "Attacker's partially observed network view." + redaction_policy: "Native observation dict, action mask, and caches stay source-private." + latency_profile: "per-step" + observable_refs: + - entry-client + defender-view: + projection_basis: "Defender scan visibility over the subnet." + redaction_policy: "Native detection state stays source-private." + latency_profile: "per-scan" + observable_refs: + - enterprise-lan + +propositions: + network-owned: + description: "The attacker owns the terminal customer-data host." + subjects: + - customer-data + basis: declared_state + predicate: + kind: presence + property: ownership.attacker + semantic_ref: "urn:raes:cyberbattlesim:ownership" + operator: exists + sla-maintained: + description: "The defender preserves network availability above the SLA floor." + subjects: + - enterprise-lan + basis: declared_state + predicate: + kind: boolean + property: availability.sla_maintained + semantic_ref: "urn:raes:cyberbattlesim:sla" + operator: equals + expected: true + +assertions: + ownership-postcondition: + description: "Attacker ownership of the terminal host is achieved." + proposition: network-owned + role: postcondition + polarity: positive + sla-invariant: + description: "Availability SLA holds throughout the episode." + proposition: sla-maintained + role: invariant + polarity: positive + +objectives: + own-network: + description: "Red team owns the terminal host at the end of the chain." + agent: attacker + actions: + - local-vulnerability + - remote-vulnerability + - connect + targets: + - customer-data + success: + mode: all_of + assertions: + - ownership-postcondition + maintain-availability: + description: "Blue team keeps the network available above the SLA floor." + entity: blue-team + targets: + - enterprise-lan + success: + mode: all_of + assertions: + - sla-invariant + +agents: + attacker: + entity: red-team + description: "CredentialCacheExploiter baseline attacker." + actions: + - local-vulnerability + - remote-vulnerability + - connect + starting_accounts: + - relay-service + initial_knowledge: + hosts: + - entry-client + observation_boundaries: + - attacker-view + allowed_subnets: + - enterprise-lan + defender: + entity: blue-team + description: "ScanAndReimageCompromisedMachines basic defender." + actions: + - scan-and-reimage + observation_boundaries: + - defender-view + allowed_subnets: + - enterprise-lan + +evidence_requirements: + attacker-action-log: + description: "Portable record of attacker action outcomes for adjudication." + source_class: participant_action + scope: "attacker actions across the chain" + window: "full episode" + channel: log + sensitivity: redacted + redaction: redact_secrets + integrity: checksum + retention: run_lifetime + loss_disclosure: best_effort + availability-series: + description: "Portable record of per-step network availability." + source_class: scenario_state + scope: "network availability over the episode" + window: "per-step" + channel: metric + sensitivity: plain + redaction: none + integrity: checksum + retention: run_lifetime + loss_disclosure: best_effort diff --git a/src/raes_adapters/cyberbattlesim/scenario_ledger.py b/src/raes_adapters/cyberbattlesim/scenario_ledger.py new file mode 100644 index 0000000..b25a41e --- /dev/null +++ b/src/raes_adapters/cyberbattlesim/scenario_ledger.py @@ -0,0 +1,834 @@ +"""Loaders and deterministic validators for the CyberBattleSim scenario evidence. + +Issue #26 authors a portable evidence set for the qualified CyberBattleSim case: + +- an authored RAES SDL scenario carrying topology/objective truth + (:func:`load_scenario`), validated and compiled through the pinned + ``raes==2.0.0`` release; +- a companion published experiment contract pair (:func:`load_experiment_task`, + :func:`load_experiment_spec`) carrying the reward, evaluator, metric, + episode/termination, and stochastic intent that RAES excludes from SDL; and +- a pinned source-mapping ledger (:func:`load_source_ledger`) recording how each + immutable CyberBattleSim source fact reaches a portable RAES surface, or why it + does not, with every disclosed loss bound to the ADR-069 equivalence tier it + weakens. + +The validators are module-local helpers over checked-in package resources. They +compose the published RAES parser, closed contract models, the published +``schema_bundle`` for target resolution, and the already-qualified +``qualification.json`` pins. They define no new schema, DTO, vocabulary, policy +gate, or exception hierarchy; the disposition, category, and equivalence-tier +labels are issue-local validation labels, not RAES vocabulary. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from importlib.resources import files +from typing import NamedTuple, cast + +import raes # type: ignore[import-untyped] +from raes_contracts.contracts import ( # type: ignore[import-untyped] + ExperimentSpecModel, + ExperimentTaskModel, + schema_bundle, +) +from raes_contracts.experiment_spec import parse_experiment_spec # type: ignore[import-untyped] +from raes_processor.compiler import compile_scenario_runtime_model # type: ignore[import-untyped] + +from . import load_qualification, read_public_protocol + +# A decoded JSON value and object. Used instead of ``Any`` so the ledger and +# schema-bundle traversal carry a concrete (if dynamic) type. +type JsonValue = None | bool | int | float | str | list[JsonValue] | dict[str, JsonValue] +type JsonObject = dict[str, JsonValue] + +_PACKAGE = __package__ or "raes_adapters.cyberbattlesim" +_DEFS = "$defs" +_REF = "$ref" +_DEFS_PREFIX = "#/$defs/" + + +class EvidenceSelection(NamedTuple): + """One immutable CyberBattleSim evidence selection. + + Per the scenario/ledger guardrails the extension seam is an explicit + selection identity: a second CyberBattleSim case is another + ``EvidenceSelection`` passed to the same loaders and validator, not an edit + to the validation code. Every cross-artifact identity the joins check + (scenario id + pinned digest, task id, spec id) lives here, so the members + are validated together rather than as independent singletons. + """ + + scenario_id: str + scenario_resource: tuple[str, ...] + pinned_scenario_digest: str + task_id: str + task_resource: tuple[str, ...] + spec_id: str + spec_resource: tuple[str, ...] + ledger_resource: tuple[str, ...] + losses_resource: tuple[str, ...] + protocol_resource: str + + +CYBERBATTLE_CHAIN = EvidenceSelection( + scenario_id="cyberbattlesim-chain", + scenario_resource=("scenario", "cyberbattle-chain.sdl.yaml"), + pinned_scenario_digest="sha256:9d696ea7fa23a1e7cf4c1cbc145a7989370dc4e9afff2cd5a17d6d1af887b528", + task_id="cyberbattlesim-chain-task", + task_resource=("experiment", "cyberbattle-chain.task.exp.yaml"), + spec_id="cyberbattlesim-chain-spec", + spec_resource=("experiment", "cyberbattle-chain.spec.exp.yaml"), + ledger_resource=("mapping", "source-ledger.jsonl"), + losses_resource=("mapping", "loss-disclosures.md"), + protocol_resource="public-protocol.md", +) + +# Pinned instantiated-snapshot digest of the default selection's scenario. Any +# change to the scenario source changes this value and fails CI, so the portable +# scenario cannot drift silently from its reviewed form. +PINNED_SCENARIO_DIGEST = CYBERBATTLE_CHAIN.pinned_scenario_digest + +DISPOSITIONS = frozenset({"mapped", "excluded", "loss-disclosed"}) + +REQUIRED_CATEGORIES = frozenset( + { + "topology-state", + "identities", + "participants", + "actions", + "observations", + "controls", + "rewards-objectives", + "termination", + "evaluator", + "stochastic", + "provenance-licensing", + } +) + +# Issue-local labels for the ADR-069 tiered-equivalence ladder (see +# loss-disclosures.md); not a new RAES vocabulary. +RECOGNIZED_EQUIVALENCE_TIERS = frozenset( + {"reproducibility", "deterministic-replay", "outcome-equivalence"} +) + +_REQUIRED_ROW_FIELDS = frozenset( + { + "source_id", + "source_repo", + "source_commit", + "source_path", + "source_selector", + "source_digest", + "category", + "disposition", + "mapping_rule", + "verification", + } +) +_OPTIONAL_ROW_FIELDS = frozenset({"raes_target", "loss_ref", "equivalence_tier"}) +_ALLOWED_ROW_FIELDS = _REQUIRED_ROW_FIELDS | _OPTIONAL_ROW_FIELDS + +_MISSING_FIELD_REASON = "missing or blank required field" + +# Native simulator object/array representation markers that must never appear in +# a portable RAES artifact. The native observation-field identifiers are not +# hand-listed here; they are read from the qualification record's recorded +# ``smoke.observation_keys`` (provenance) at scan time. These representation +# markers add object/array reprs and tracebacks. Raw native arrays, reward +# vectors, and action ids are separately excluded *structurally* by the closed +# RAES SDL and experiment models, which have no field to carry them; this scan +# is defense in depth over the native identifier and representation classes. +_NATIVE_REPR_MARKERS: tuple[str, ...] = ( + "traceback (most recent call last)", + " object at 0x", + "ndarray(", + "array([", + "\S+)\s*$") +_LOSS_TIER = re.compile(r"^\*\*Equivalence tier:\*\*\s*(?P\S+)\s*$") + + +class LedgerProblem(NamedTuple): + """A single fail-closed validation problem (row id + field + reason only).""" + + row_id: str + field: str + reason: str + + def __str__(self) -> str: + """Render the problem as ``[row_id] field: reason``.""" + return f"[{self.row_id}] {self.field}: {self.reason}" + + +# --------------------------------------------------------------------------- # +# JSON helpers +# --------------------------------------------------------------------------- # +def _as_object(value: JsonValue) -> JsonObject: + """Return the value when it is a JSON object, else an empty object.""" + return value if isinstance(value, dict) else {} + + +def _as_list(value: JsonValue) -> list[JsonValue]: + """Return the value when it is a JSON array, else an empty list.""" + return value if isinstance(value, list) else [] + + +def _as_str(value: JsonValue) -> str: + """Return the value when it is a string, else the empty string.""" + return value if isinstance(value, str) else "" + + +# --------------------------------------------------------------------------- # +# resource loaders +# --------------------------------------------------------------------------- # +def _read_text(*parts: str) -> str: + """Read a package resource as UTF-8 text.""" + return files(_PACKAGE).joinpath(*parts).read_text(encoding="utf-8") + + +def _resource_is_file(name: str) -> bool: + """Return whether the named package resource exists as a file.""" + return bool(files(_PACKAGE).joinpath(name).is_file()) + + +def scenario_source(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> str: + """Return the authored RAES SDL scenario source verbatim.""" + return _read_text(*selection.scenario_resource) + + +def experiment_task_source(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> str: + """Return the published experiment-task contract source verbatim.""" + return _read_text(*selection.task_resource) + + +def experiment_spec_source(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> str: + """Return the published experiment-spec contract source verbatim.""" + return _read_text(*selection.spec_resource) + + +def _yaml_safe_load(text: str) -> JsonValue: + """Parse YAML text into a JSON-like value (lazy ``yaml`` import).""" + import yaml # type: ignore[import-untyped] + + return cast("JsonValue", yaml.safe_load(text)) + + +def load_scenario(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> raes.Scenario: + """Parse and semantically validate the authored scenario via ``raes``.""" + return raes.parse_sdl(scenario_source(selection), path=None) + + +def scenario_canonical_digest(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> str: + """Instantiate the scenario and return its canonical snapshot digest.""" + instantiated = raes.instantiate_scenario(load_scenario(selection)) + return cast(str, raes.canonical_instantiated_sdl_digest(instantiated).value) + + +def load_experiment_task(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> ExperimentTaskModel: + """Validate the companion experiment-task contract (closed RAES model).""" + return ExperimentTaskModel.model_validate(_yaml_safe_load(experiment_task_source(selection))) + + +def load_experiment_spec(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> ExperimentSpecModel: + """Validate the companion experiment-spec contract via ``raes_contracts``.""" + return parse_experiment_spec(experiment_spec_source(selection)) + + +def parse_ledger_rows(text: str) -> list[JsonObject]: + """Strictly parse the JSONL ledger, rejecting malformed evidence. + + Fails closed on invalid JSON, a non-object row, duplicate JSON keys, and + non-finite numbers, so the source ledger cannot carry silently corrupt rows. + """ + + def _no_duplicate_keys(pairs: list[tuple[str, JsonValue]]) -> JsonObject: + """Build a JSON object, rejecting duplicate keys.""" + seen: JsonObject = {} + for key, value in pairs: + if key in seen: + raise ValueError(f"duplicate JSON key: {key!r}") + seen[key] = value + return seen + + def _reject_non_finite(value: str) -> float: + """Reject NaN/Infinity JSON constants.""" + raise ValueError(f"non-finite number is not permitted: {value}") + + rows: list[JsonObject] = [] + for lineno, line in enumerate(text.splitlines(), start=1): + if not line.strip(): + continue + row = json.loads( + line, object_pairs_hook=_no_duplicate_keys, parse_constant=_reject_non_finite + ) + if not isinstance(row, dict): + raise ValueError(f"ledger line {lineno} is not a JSON object") + rows.append(cast("JsonObject", row)) + return rows + + +def load_source_ledger(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> list[JsonObject]: + """Load and strictly parse the pinned source-mapping ledger.""" + return parse_ledger_rows(_read_text(*selection.ledger_resource)) + + +def parse_loss_disclosures(text: str) -> dict[str, str]: + """Parse the disclosures doc into ``{loss_id: equivalence_tier}``. + + Every ``## `` heading is preserved. A heading with no equivalence + tier line is recorded with an empty tier so it fails closed downstream + (unrecognized tier / disagreement) rather than silently disappearing. + """ + disclosures: dict[str, str] = {} + current: str | None = None + for line in text.splitlines(): + heading = _LOSS_HEADING.match(line) + if heading: + if current is not None: + disclosures.setdefault(current, "") + current = heading.group("loss_id") + continue + tier = _LOSS_TIER.match(line) + if tier and current is not None: + disclosures[current] = tier.group("tier") + current = None + if current is not None: + disclosures.setdefault(current, "") + return disclosures + + +def load_loss_disclosures(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> dict[str, str]: + """Return ``{loss_id: equivalence_tier}`` parsed from the disclosures doc.""" + return parse_loss_disclosures(_read_text(*selection.losses_resource)) + + +# --------------------------------------------------------------------------- # +# target resolution +# --------------------------------------------------------------------------- # +def _contract_indexes( + bundle: dict[str, JsonObject], +) -> tuple[dict[str, JsonObject], dict[str, JsonObject]]: + """Index the schema bundle by contract title and by ``$defs`` class name.""" + title_index: dict[str, JsonObject] = {} + defs_root: dict[str, JsonObject] = {} + for schema in bundle.values(): + title = schema.get("title") + if isinstance(title, str): + title_index[title] = schema + for name in _as_object(schema.get(_DEFS)): + defs_root.setdefault(name, schema) + return title_index, defs_root + + +def _follow_ref(root: JsonObject, node: JsonObject) -> JsonObject: + """Follow a ``$ref`` (through list items or any/oneOf) into the bundle defs.""" + if _REF not in node and _REF in _as_object(node.get("items")): + node = _as_object(node.get("items")) + if _REF not in node: + for combinator in ("anyOf", "oneOf"): + for sub in _as_list(node.get(combinator)): + sub_object = _as_object(sub) + if _REF in sub_object: + node = sub_object + break + ref = node.get(_REF) + if isinstance(ref, str) and ref.startswith(_DEFS_PREFIX): + return _as_object(_as_object(root.get(_DEFS)).get(ref.split("/")[-1])) + return node + + +def resolve_contract_target(path: str, bundle: dict[str, JsonObject]) -> bool: + """Resolve a ``Contract.property.path`` against the published schema bundle.""" + head, *rest = path.split(".") + title_index, defs_root = _contract_indexes(bundle) + if head in title_index: + root = node = title_index[head] + elif head in defs_root: + root = defs_root[head] + node = _as_object(_as_object(root.get(_DEFS)).get(head)) + else: + return False + for index, prop in enumerate(rest): + properties = _as_object(node.get("properties")) + if prop not in properties: + return False + node = _as_object(properties.get(prop)) + if index < len(rest) - 1: + node = _follow_ref(root, node) + return True + + +def resolve_sdl_target(body: str, scenario: raes.Scenario) -> bool: + """Resolve a ``section`` or ``section.key`` against the parsed SDL scenario.""" + missing = object() + if "." in body: + section, key = body.split(".", 1) + collection = getattr(scenario, section, missing) + return isinstance(collection, dict) and key in collection + return getattr(scenario, body, missing) is not missing + + +def resolve_target(target: str, *, scenario: raes.Scenario, bundle: dict[str, JsonObject]) -> bool: + """Resolve a ``raes_target`` against parsed artifacts or the schema bundle.""" + prefix, _, body = target.partition(":") + if prefix == "sdl": + return resolve_sdl_target(body, scenario) + if prefix == "contract": + return resolve_contract_target(body, bundle) + return prefix == "resource" and _resource_is_file(body) + + +# --------------------------------------------------------------------------- # +# ledger validation +# --------------------------------------------------------------------------- # +def _string_field(row: JsonObject, field: str) -> str | None: + """Return a non-blank string field, or None when missing/blank/non-string.""" + value = row.get(field) + return value if isinstance(value, str) and value.strip() else None + + +def _row_shape_problems(row: JsonObject, row_id: str) -> list[LedgerProblem]: + """Report unknown and missing/blank required fields for one row.""" + problems = [ + LedgerProblem(row_id, field, "unknown ledger field") + for field in sorted(set(row) - _ALLOWED_ROW_FIELDS) + ] + problems += [ + LedgerProblem(row_id, field, _MISSING_FIELD_REASON) + for field in sorted(_REQUIRED_ROW_FIELDS) + if _string_field(row, field) is None + ] + return problems + + +def _row_source_problems( + row: JsonObject, row_id: str, source: JsonObject, digest_by_path: dict[str, str] +) -> list[LedgerProblem]: + """Report source-identity and offline source-drift join problems for one row.""" + problems: list[LedgerProblem] = [] + if row["source_repo"] != source.get("repository"): + problems.append( + LedgerProblem(row_id, "source_repo", "does not match qualification source repository") + ) + if row["source_commit"] != source.get("commit"): + problems.append( + LedgerProblem(row_id, "source_commit", "does not match qualification source commit") + ) + expected = digest_by_path.get(_as_str(row["source_path"])) + digest = _string_field(row, "source_digest") + if expected is None: + problems.append(LedgerProblem(row_id, "source_path", "not a qualified source file")) + elif digest is not None and expected != digest: + problems.append( + LedgerProblem( + row_id, "source_digest", "does not match the qualified file digest (source drift)" + ) + ) + return problems + + +def _mapped_row_problems( + row_id: str, + target: str | None, + loss_ref: str | None, + tier: str | None, + *, + scenario: raes.Scenario, + bundle: dict[str, JsonObject], +) -> list[LedgerProblem]: + """Problems for a ``mapped`` row: a resolvable target and no loss fields.""" + problems: list[LedgerProblem] = [] + if target is None: + problems.append( + LedgerProblem(row_id, "raes_target", "mapped row must name a resolvable target") + ) + elif not resolve_target(target, scenario=scenario, bundle=bundle): + problems.append(LedgerProblem(row_id, "raes_target", f"target does not resolve: {target}")) + if loss_ref is not None or tier is not None: + problems.append(LedgerProblem(row_id, "loss_ref", "mapped row must not carry loss fields")) + return problems + + +def _excluded_row_problems( + row_id: str, target: str | None, loss_ref: str | None, tier: str | None +) -> list[LedgerProblem]: + """Problems for an ``excluded`` row: no target and no loss fields.""" + problems: list[LedgerProblem] = [] + if target is not None: + problems.append(LedgerProblem(row_id, "raes_target", "excluded row must not name a target")) + if loss_ref is not None or tier is not None: + problems.append( + LedgerProblem(row_id, "loss_ref", "excluded row must not carry loss fields") + ) + return problems + + +def _loss_row_problems( + row_id: str, + target: str | None, + loss_ref: str | None, + tier: str | None, + *, + losses: dict[str, str], + referenced: set[str], +) -> list[LedgerProblem]: + """Problems for a ``loss-disclosed`` row: a known loss ref and an agreeing tier.""" + problems: list[LedgerProblem] = [] + if target is not None: + problems.append( + LedgerProblem(row_id, "raes_target", "loss-disclosed row must not name a target") + ) + if loss_ref is None: + problems.append( + LedgerProblem(row_id, "loss_ref", "loss-disclosed row must reference a loss") + ) + return problems + if loss_ref not in losses: + problems.append(LedgerProblem(row_id, "loss_ref", f"unknown loss {loss_ref!r}")) + return problems + referenced.add(loss_ref) + if tier is None: + problems.append( + LedgerProblem(row_id, "equivalence_tier", "loss-disclosed row must name a tier") + ) + elif tier not in RECOGNIZED_EQUIVALENCE_TIERS: + problems.append(LedgerProblem(row_id, "equivalence_tier", f"unrecognized tier {tier!r}")) + elif tier != losses[loss_ref]: + problems.append( + LedgerProblem( + row_id, "equivalence_tier", f"tier {tier!r} disagrees with disclosure {loss_ref!r}" + ) + ) + return problems + + +def _row_disposition_problems( + row: JsonObject, + row_id: str, + *, + scenario: raes.Scenario, + bundle: dict[str, JsonObject], + losses: dict[str, str], + referenced: set[str], +) -> list[LedgerProblem]: + """Dispatch to the disposition-specific target/loss checks for one row.""" + disposition = row["disposition"] + target = _string_field(row, "raes_target") + loss_ref = _string_field(row, "loss_ref") + tier = _string_field(row, "equivalence_tier") + if disposition == "mapped": + problems = _mapped_row_problems( + row_id, target, loss_ref, tier, scenario=scenario, bundle=bundle + ) + elif disposition == "excluded": + problems = _excluded_row_problems(row_id, target, loss_ref, tier) + elif disposition == "loss-disclosed": + problems = _loss_row_problems( + row_id, target, loss_ref, tier, losses=losses, referenced=referenced + ) + else: + problems = [LedgerProblem(row_id, "disposition", f"unknown disposition {disposition!r}")] + return problems + + +def _digest_by_path(qualification: JsonObject) -> dict[str, str]: + """Map each qualified source path to its recorded SHA-256 digest.""" + result: dict[str, str] = {} + for entry in _as_list(qualification.get("source_files")): + obj = _as_object(entry) + result[_as_str(obj.get("path"))] = _as_str(obj.get("sha256")) + return result + + +class _LedgerContext(NamedTuple): + """Shared inputs and running accumulators for one ledger validation pass. + + Bundling these keeps :func:`_validate_row` to a small parameter list; the + ``seen_ids``/``categories``/``referenced`` sets accumulate across rows. + """ + + source: JsonObject + digest_by_path: dict[str, str] + scenario: raes.Scenario + bundle: dict[str, JsonObject] + losses: dict[str, str] + seen_ids: set[str] + categories: set[str] + referenced: set[str] + + +def _validate_row(row: JsonObject, position: int, context: _LedgerContext) -> list[LedgerProblem]: + """Validate one ledger row (shape, identity, source join, disposition).""" + raw_id = row.get("source_id") + row_id = raw_id if isinstance(raw_id, str) else f"row[{position}]" + shape = _row_shape_problems(row, row_id) + if any(problem.reason == _MISSING_FIELD_REASON for problem in shape): + return shape + + problems = list(shape) + source_id = _as_str(row["source_id"]) + if source_id in context.seen_ids: + problems.append(LedgerProblem(row_id, "source_id", "duplicate row id")) + context.seen_ids.add(source_id) + + category = _as_str(row["category"]) + if category not in REQUIRED_CATEGORIES: + problems.append(LedgerProblem(row_id, "category", f"unknown category {category!r}")) + context.categories.add(category) + + problems += _row_source_problems(row, row_id, context.source, context.digest_by_path) + problems += _row_disposition_problems( + row, + row_id, + scenario=context.scenario, + bundle=context.bundle, + losses=context.losses, + referenced=context.referenced, + ) + return problems + + +def _coverage_problems(categories: set[str]) -> list[LedgerProblem]: + """Report any required coverage category absent from the ledger.""" + return [ + LedgerProblem("", "category", f"required category absent: {category}") + for category in sorted(REQUIRED_CATEGORIES - categories) + ] + + +def _disclosure_problems(losses: dict[str, str], referenced: set[str]) -> list[LedgerProblem]: + """Report disclosures with a bad/absent tier or with no referencing row.""" + problems: list[LedgerProblem] = [] + for loss_id, tier in sorted(losses.items()): + if tier not in RECOGNIZED_EQUIVALENCE_TIERS: + reason = ( + "disclosure has no equivalence tier" + if not tier + else f"disclosure names unrecognized tier: {tier}" + ) + problems.append(LedgerProblem(loss_id, "equivalence_tier", reason)) + for unreferenced in sorted(set(losses) - referenced): + problems.append( + LedgerProblem( + unreferenced, "loss_ref", "disclosed loss is not referenced by any source row" + ) + ) + return problems + + +def validate_source_ledger( + rows: list[JsonObject], + *, + qualification: JsonObject, + scenario: raes.Scenario, + bundle: dict[str, JsonObject], + losses: dict[str, str], +) -> list[LedgerProblem]: + """Return every fail-closed problem in the ledger (empty list == valid).""" + context = _LedgerContext( + source=_as_object(qualification.get("source")), + digest_by_path=_digest_by_path(qualification), + scenario=scenario, + bundle=bundle, + losses=losses, + seen_ids=set(), + categories=set(), + referenced=set(), + ) + + problems: list[LedgerProblem] = [] + for position, row in enumerate(rows): + problems += _validate_row(row, position, context) + problems += _coverage_problems(context.categories) + problems += _disclosure_problems(losses, context.referenced) + return problems + + +# --------------------------------------------------------------------------- # +# native-leakage scan +# --------------------------------------------------------------------------- # +def _native_identifier_markers() -> set[str]: + """Native markers to reject in portable content, grounded in provenance. + + The native observation-field identifiers come from the qualification + record's recorded ``smoke.observation_keys``; the representation markers are + the fixed object/array/traceback reprs. Matching is case-insensitive. + """ + observation_keys = load_qualification()["runtime"]["smoke"]["observation_keys"] + markers = {str(key).lower() for key in observation_keys} + markers.update(_NATIVE_REPR_MARKERS) + return markers + + +def native_leakage_problems(portable_texts: dict[str, str]) -> list[LedgerProblem]: + """Return a problem for every native simulator marker in portable content. + + This enforces the native identifier and object-representation classes. Raw + native arrays, reward vectors, and action ids are excluded structurally by + the closed RAES SDL and experiment models rather than by this scan. + """ + markers = _native_identifier_markers() + problems: list[LedgerProblem] = [] + for name, text in portable_texts.items(): + lowered = text.lower() + problems += [ + LedgerProblem(name, "native-leakage", f"native marker present: {marker}") + for marker in sorted(markers) + if marker in lowered + ] + return problems + + +# --------------------------------------------------------------------------- # +# cross-artifact selection joins +# --------------------------------------------------------------------------- # +def _scenario_join_problems( + scenario: raes.Scenario, digest: str, selection: EvidenceSelection +) -> list[LedgerProblem]: + """Join the loaded scenario identity and digest to the selection pin.""" + problems: list[LedgerProblem] = [] + if scenario.name != selection.scenario_id: + problems.append( + LedgerProblem("scenario", "name", "scenario name does not match the selection id") + ) + if digest != selection.pinned_scenario_digest: + problems.append( + LedgerProblem("scenario", "digest", "canonical digest does not match the selection pin") + ) + return problems + + +def _task_join_problems( + task: ExperimentTaskModel, selection: EvidenceSelection +) -> list[LedgerProblem]: + """Join the task identity and its scenario reference to the selection.""" + problems: list[LedgerProblem] = [] + if task.task_id != selection.task_id: + problems.append(LedgerProblem("task", "task_id", "task id does not match the selection")) + scenario_ref = task.scenario_ref + if getattr(scenario_ref, "ref_id", None) != selection.scenario_id: + problems.append( + LedgerProblem("task", "scenario_ref", "task scenario_ref id does not match scenario") + ) + if getattr(scenario_ref, "ref_digest", None) != selection.pinned_scenario_digest: + problems.append( + LedgerProblem("task", "scenario_ref", "task scenario_ref digest does not match the pin") + ) + return problems + + +def _spec_join_problems( + spec: ExperimentSpecModel, selection: EvidenceSelection +) -> list[LedgerProblem]: + """Join the spec identity and its task/scenario references to the selection.""" + problems: list[LedgerProblem] = [] + if spec.spec_id != selection.spec_id: + problems.append(LedgerProblem("spec", "spec_id", "spec id does not match the selection")) + if getattr(spec.task_ref, "ref_id", None) != selection.task_id: + problems.append(LedgerProblem("spec", "task_ref", "spec task_ref does not match the task")) + scenario_ref = getattr(spec, "intended_scenario_ref", None) + if scenario_ref is None: + return problems + if getattr(scenario_ref, "ref_id", None) != selection.scenario_id: + problems.append( + LedgerProblem("spec", "intended_scenario_ref", "spec scenario_ref id does not match") + ) + if getattr(scenario_ref, "ref_digest", None) != selection.pinned_scenario_digest: + problems.append( + LedgerProblem("spec", "intended_scenario_ref", "spec scenario_ref digest mismatch") + ) + return problems + + +def _protocol_artifact_problems( + task: ExperimentTaskModel, protocol_bytes: bytes +) -> list[LedgerProblem]: + """Join the task's protocol artifact reference to the actual protocol bytes.""" + protocol_sha = hashlib.sha256(protocol_bytes).hexdigest() + protocol_refs = [ref for ref in task.artifact_refs if getattr(ref, "role", None) == "protocol"] + problems: list[LedgerProblem] = [] + if not protocol_refs: + problems.append(LedgerProblem("task", "artifact_refs", "no protocol artifact reference")) + for ref in protocol_refs: + if getattr(getattr(ref, "checksum", None), "value", None) != protocol_sha: + problems.append( + LedgerProblem("task", "artifact_refs", "protocol checksum does not match resource") + ) + if getattr(ref, "size_bytes", None) != len(protocol_bytes): + problems.append( + LedgerProblem("task", "artifact_refs", "protocol size does not match resource") + ) + return problems + + +def validate_selection_joins( + selection: EvidenceSelection = CYBERBATTLE_CHAIN, +) -> list[LedgerProblem]: + """Validate the cross-artifact identity joins of one evidence selection. + + Independently valid artifacts are not enough: a stale reference or a drifted + protocol digest must fail closed. This joins the scenario, task, and spec + references to the selection's pinned identity and verifies the task's public + protocol artifact against the actual protocol resource. + """ + scenario = load_scenario(selection) + task = load_experiment_task(selection) + spec = load_experiment_spec(selection) + problems = _scenario_join_problems(scenario, scenario_canonical_digest(selection), selection) + problems += _task_join_problems(task, selection) + problems += _spec_join_problems(spec, selection) + problems += _protocol_artifact_problems(task, read_public_protocol().encode("utf-8")) + return problems + + +# --------------------------------------------------------------------------- # +# scenario pipeline + top-level validation +# --------------------------------------------------------------------------- # +def validate_scenario_pipeline( + selection: EvidenceSelection = CYBERBATTLE_CHAIN, +) -> list[LedgerProblem]: + """Validate that the scenario parses, instantiates, compiles, and admits.""" + problems: list[LedgerProblem] = [] + scenario = load_scenario(selection) + instantiated = raes.instantiate_scenario(scenario) + digest = cast(str, raes.canonical_instantiated_sdl_digest(instantiated).value) + if digest != selection.pinned_scenario_digest: + problems.append( + LedgerProblem("scenario", "digest", f"canonical digest drifted to {digest}") + ) + runtime = compile_scenario_runtime_model(instantiated) + for diagnostic in getattr(runtime, "diagnostics", []): + problems.append(LedgerProblem("scenario", "compile", str(diagnostic))) + raes.admit_instantiated_scenario(instantiated) + return problems + + +def validate_all(selection: EvidenceSelection = CYBERBATTLE_CHAIN) -> list[LedgerProblem]: + """Run every deterministic check over the checked-in evidence set.""" + problems: list[LedgerProblem] = [] + problems += validate_scenario_pipeline(selection) + # validate_selection_joins loads (and thus closed-model-validates) the task + # and spec, and joins every cross-artifact reference. + problems += validate_selection_joins(selection) + problems += validate_source_ledger( + load_source_ledger(selection), + qualification=cast("JsonObject", load_qualification()), + scenario=load_scenario(selection), + bundle=schema_bundle(), + losses=load_loss_disclosures(selection), + ) + problems += native_leakage_problems( + { + "scenario": scenario_source(selection), + "experiment-task": experiment_task_source(selection), + "experiment-spec": experiment_spec_source(selection), + } + ) + return problems diff --git a/tests/test_cyberbattlesim_scenario_ledger.py b/tests/test_cyberbattlesim_scenario_ledger.py new file mode 100644 index 0000000..ba3a7cd --- /dev/null +++ b/tests/test_cyberbattlesim_scenario_ledger.py @@ -0,0 +1,377 @@ +"""Scenario + pinned source-mapping ledger evidence for CyberBattleSim (#26). + +The authored RAES SDL scenario carries portable topology/objective truth only; +reward, evaluator, termination, and stochastic controls live in the companion +published experiment contracts. The source ledger records how each pinned +CyberBattleSim source fact reaches a portable RAES surface, or why it does not, +and every disclosed loss binds the ADR-069 equivalence tier it weakens. + +The negative tests below are the fail-closed guarantee: each one proves the +deterministic validator rejects a specific corruption (source drift, a missing +category, a duplicate row, an unresolvable target, an undisclosed loss, native +leakage), so CI fails when the checked-in evidence set drifts into that state. +""" + +from __future__ import annotations + +import copy +import hashlib +from types import SimpleNamespace +from typing import Any + +import pytest +from raes_contracts.contracts import schema_bundle + +import raes_adapters.cyberbattlesim.scenario_ledger as sl + + +def _context() -> dict[str, Any]: + return { + "qualification": sl.load_qualification(), + "scenario": sl.load_scenario(), + "bundle": schema_bundle(), + "losses": sl.load_loss_disclosures(), + } + + +def _validate(rows: list[dict[str, Any]]) -> list[sl.LedgerProblem]: + return sl.validate_source_ledger(rows, **_context()) + + +def _row(source_id: str) -> dict[str, Any]: + for row in sl.load_source_ledger(): + if row["source_id"] == source_id: + return copy.deepcopy(row) + raise AssertionError(f"no such row: {source_id}") + + +# --------------------------------------------------------------------------- # +# positive: the checked-in evidence set validates end to end +# --------------------------------------------------------------------------- # +def test_sdl_scenario_validates_instantiates_and_admits() -> None: + scenario = sl.load_scenario() + assert scenario.name == "cyberbattlesim-chain" + assert scenario.semantic_validated is True + digest = sl.scenario_canonical_digest() + assert digest == sl.PINNED_SCENARIO_DIGEST + assert digest.startswith("sha256:") + + +def test_scenario_pipeline_compiles_with_pinned_release() -> None: + # parse -> instantiate -> pinned canonical digest -> compile -> admit. + assert sl.validate_scenario_pipeline() == [] + + +def test_companion_experiment_contracts_validate() -> None: + task = sl.load_experiment_task() + spec = sl.load_experiment_spec() + assert task.scenario_ref.ref_digest == sl.PINNED_SCENARIO_DIGEST + assert sorted(task.evaluation_protocol.metric_definitions) == [ + "cumulative_attacker_reward", + "network_availability", + "steps_to_termination", + "terminal_cause", + ] + # Stochastic controls stay descriptive: no executable random-stream binding, + # so the determinism loss is retained rather than silently claimed. + assert spec.run_plan.target_run_count == 10 + assert spec.run_plan.episode_control.max_steps == 600 + assert len(spec.run_plan.stochastic_controls) == 4 + assert all(c.executable_binding is None for c in spec.run_plan.stochastic_controls) + + +def test_full_evidence_set_is_valid() -> None: + assert sl.validate_all() == [] + + +def test_selection_joins_pass_for_the_checked_in_selection() -> None: + assert sl.validate_selection_joins() == [] + # The extension seam is a descriptor: the parameterized entry point accepts + # the selection object, so a second scenario is another descriptor, not a + # validator edit. + assert isinstance(sl.CYBERBATTLE_CHAIN, sl.EvidenceSelection) + assert sl.validate_all(sl.CYBERBATTLE_CHAIN) == [] + + +def test_ledger_covers_every_required_category() -> None: + rows = sl.load_source_ledger() + assert {row["category"] for row in rows} == set(sl.REQUIRED_CATEGORIES) + assert {row["disposition"] for row in rows} <= sl.DISPOSITIONS + assert len({row["source_id"] for row in rows}) == len(rows) + + +def test_loss_disclosures_bind_recognized_tiers() -> None: + losses = sl.load_loss_disclosures() + assert losses == { + "loss-no-source-artifact": "reproducibility", + "loss-unbound-random-streams": "deterministic-replay", + "loss-abstracted-topology": "outcome-equivalence", + "loss-benchmark-defects": "outcome-equivalence", + } + assert set(losses.values()) <= sl.RECOGNIZED_EQUIVALENCE_TIERS + + +# --------------------------------------------------------------------------- # +# target resolution +# --------------------------------------------------------------------------- # +def test_target_resolution_positive_and_negative() -> None: + ctx = {"scenario": sl.load_scenario(), "bundle": schema_bundle()} + assert sl.resolve_target("sdl:nodes", **ctx) + assert sl.resolve_target("sdl:agents.attacker", **ctx) + assert sl.resolve_target("contract:ExperimentTaskModel.evaluation_protocol", **ctx) + assert sl.resolve_target("contract:ExperimentSpecModel.run_plan.episode_control", **ctx) + assert sl.resolve_target("resource:qualification.json", **ctx) + + assert not sl.resolve_target("sdl:not_a_section", **ctx) + assert not sl.resolve_target("sdl:agents.ghost", **ctx) + assert not sl.resolve_target("contract:NotARealModel.foo", **ctx) + assert not sl.resolve_target("contract:ExperimentTaskModel.nope", **ctx) + assert not sl.resolve_target("resource:missing-file.json", **ctx) + assert not sl.resolve_target("mystery:thing", **ctx) + + +# --------------------------------------------------------------------------- # +# fail-closed negatives +# --------------------------------------------------------------------------- # +def test_duplicate_row_id_is_rejected() -> None: + rows = sl.load_source_ledger() + rows.append(_row("topology-chain-pattern")) + problems = _validate(rows) + assert any(p.field == "source_id" and "duplicate" in p.reason for p in problems) + + +def test_unknown_category_is_rejected() -> None: + row = _row("topology-chain-pattern") + row["category"] = "not-a-category" + problems = _validate([row]) + assert any(p.field == "category" and "unknown category" in p.reason for p in problems) + + +def test_missing_required_category_is_rejected() -> None: + rows = [r for r in sl.load_source_ledger() if r["category"] != "stochastic"] + problems = _validate(rows) + assert any(p.field == "category" and "stochastic" in p.reason for p in problems) + + +def test_unknown_disposition_is_rejected() -> None: + row = _row("topology-chain-pattern") + row["disposition"] = "sort-of-mapped" + problems = _validate([row]) + assert any(p.field == "disposition" for p in problems) + + +def test_missing_required_field_is_rejected() -> None: + row = _row("topology-chain-pattern") + del row["source_selector"] + problems = _validate([row]) + assert any(p.field == "source_selector" for p in problems) + + +def test_unknown_field_is_rejected() -> None: + row = _row("topology-chain-pattern") + row["surprise"] = "value" + problems = _validate([row]) + assert any(p.field == "surprise" for p in problems) + + +def test_source_drift_digest_mismatch_is_rejected() -> None: + row = _row("topology-chain-pattern") + row["source_digest"] = "0" * 64 + problems = _validate([row]) + assert any(p.field == "source_digest" and "drift" in p.reason for p in problems) + + +def test_source_commit_mismatch_is_rejected() -> None: + row = _row("topology-chain-pattern") + row["source_commit"] = "deadbeef" * 5 + problems = _validate([row]) + assert any(p.field == "source_commit" for p in problems) + + +def test_unqualified_source_path_is_rejected() -> None: + row = _row("topology-chain-pattern") + row["source_path"] = "cyberbattle/not/a/real/file.py" + problems = _validate([row]) + assert any(p.field == "source_path" for p in problems) + + +def test_omitting_source_digest_does_not_bypass_the_drift_join() -> None: + # Regression: the qualification join must not be skippable by dropping the + # optional-looking digest. source_digest is required, so omission fails closed. + row = _row("topology-chain-pattern") + del row["source_digest"] + assert any(p.field == "source_digest" for p in _validate([row])) + + # And the source_path -> qualification join runs independently of the digest + # value: an unqualified path is caught even when a (wrong) digest is present. + row = _row("topology-chain-pattern") + row["source_path"] = "cyberbattle/not/a/real/file.py" + row["source_digest"] = "0" * 64 + assert any(p.field == "source_path" for p in _validate([row])) + + +def test_mapped_row_with_unresolvable_target_is_rejected() -> None: + row = _row("topology-chain-pattern") + row["raes_target"] = "sdl:not_a_section" + problems = _validate([row]) + assert any(p.field == "raes_target" and "does not resolve" in p.reason for p in problems) + + +def test_mapped_row_without_target_is_rejected() -> None: + row = _row("topology-chain-pattern") + del row["raes_target"] + problems = _validate([row]) + assert any(p.field == "raes_target" for p in problems) + + +def test_excluded_row_with_target_is_rejected() -> None: + row = _row("credential-cache-native") + row["raes_target"] = "sdl:nodes" + problems = _validate([row]) + assert any(p.field == "raes_target" for p in problems) + + +def test_loss_row_without_loss_ref_is_rejected() -> None: + row = _row("stochastic-binding-loss") + del row["loss_ref"] + problems = _validate([row]) + assert any(p.field == "loss_ref" for p in problems) + + +def test_loss_row_with_wrong_tier_is_rejected() -> None: + row = _row("stochastic-binding-loss") + row["equivalence_tier"] = "outcome-equivalence" # disclosure says deterministic-replay + problems = _validate([row]) + assert any(p.field == "equivalence_tier" and "disagrees" in p.reason for p in problems) + + +def test_loss_row_with_unrecognized_tier_is_rejected() -> None: + row = _row("stochastic-binding-loss") + row["equivalence_tier"] = "vibes-equivalence" + problems = _validate([row]) + assert any(p.field == "equivalence_tier" for p in problems) + + +def test_unreferenced_disclosure_is_rejected() -> None: + ctx = _context() + ctx["losses"] = {**ctx["losses"], "loss-orphan": "reproducibility"} + problems = sl.validate_source_ledger(sl.load_source_ledger(), **ctx) + assert any(p.row_id == "loss-orphan" and p.field == "loss_ref" for p in problems) + + +def test_native_leakage_marker_is_rejected() -> None: + problems = sl.native_leakage_problems({"scenario": "nodes with credential_cache_matrix leaked"}) + assert any(p.field == "native-leakage" for p in problems) + assert sl.native_leakage_problems({"scenario": "no native identifiers here"}) == [] + + +# --------------------------------------------------------------------------- # +# strict JSONL parsing +# --------------------------------------------------------------------------- # +def test_strict_jsonl_rejects_duplicate_keys() -> None: + with pytest.raises(ValueError, match="duplicate JSON key"): + sl.parse_ledger_rows('{"source_id": "a", "source_id": "b"}') + + +def test_strict_jsonl_rejects_non_object_row() -> None: + with pytest.raises(ValueError, match="not a JSON object"): + sl.parse_ledger_rows("[1, 2, 3]") + + +def test_strict_jsonl_rejects_non_finite_number() -> None: + with pytest.raises(ValueError, match="non-finite"): + sl.parse_ledger_rows('{"n": NaN}') + + +# --------------------------------------------------------------------------- # +# cross-artifact selection joins +# --------------------------------------------------------------------------- # +def test_join_rejects_wrong_pinned_scenario_digest() -> None: + bad = sl.CYBERBATTLE_CHAIN._replace(pinned_scenario_digest="sha256:" + "0" * 64) + problems = sl.validate_selection_joins(bad) + assert any(p.field == "digest" for p in problems) + assert any(p.row_id in {"task", "spec"} and p.field.endswith("scenario_ref") for p in problems) + + +def test_join_rejects_stale_task_reference() -> None: + bad = sl.CYBERBATTLE_CHAIN._replace(task_id="stale-task-id") + problems = sl.validate_selection_joins(bad) + assert any(p.row_id == "task" and p.field == "task_id" for p in problems) + assert any(p.row_id == "spec" and p.field == "task_ref" for p in problems) + + +def test_join_rejects_wrong_scenario_and_spec_identity() -> None: + bad = sl.CYBERBATTLE_CHAIN._replace(scenario_id="wrong-scn", spec_id="wrong-spec") + problems = sl.validate_selection_joins(bad) + assert any(p.row_id == "scenario" and p.field == "name" for p in problems) + assert any(p.row_id == "spec" and p.field == "spec_id" for p in problems) + + +def test_protocol_artifact_join_detects_checksum_and_size_drift() -> None: + body = b"protocol bytes" + good = SimpleNamespace( + artifact_refs=[ + SimpleNamespace( + role="protocol", + checksum=SimpleNamespace(value=hashlib.sha256(body).hexdigest()), + size_bytes=len(body), + ) + ] + ) + assert sl._protocol_artifact_problems(good, body) == [] + + drifted = SimpleNamespace( + artifact_refs=[ + SimpleNamespace(role="protocol", checksum=SimpleNamespace(value="dead"), size_bytes=1) + ] + ) + problems = sl._protocol_artifact_problems(drifted, body) + assert any("checksum" in p.reason for p in problems) + assert any("size" in p.reason for p in problems) + + missing = SimpleNamespace(artifact_refs=[]) + assert any("no protocol" in p.reason for p in sl._protocol_artifact_problems(missing, body)) + + +# --------------------------------------------------------------------------- # +# native-leakage classes (grounded in qualification provenance) +# --------------------------------------------------------------------------- # +def test_native_identifier_leakage_is_rejected_case_insensitively() -> None: + assert sl.native_leakage_problems({"a": "leaks credential_cache_matrix here"}) + assert sl.native_leakage_problems({"a": "LEAKS CREDENTIAL_CACHE_MATRIX HERE"}) + assert sl.native_leakage_problems({"a": "nodes_privilegelevel exposed"}) + + +def test_native_representation_leakage_is_rejected() -> None: + for text in ( + "Traceback (most recent call last):", + "x = ndarray(3)", + "y = array([1, 2, 3])", + "", + "repr is ", + ): + assert sl.native_leakage_problems({"a": text}), text + + +def test_leakage_markers_are_grounded_in_qualification_provenance() -> None: + markers = sl._native_identifier_markers() + keys = sl.load_qualification()["runtime"]["smoke"]["observation_keys"] + assert {key.lower() for key in keys} <= markers + assert "credential_cache_matrix" in markers + + +def test_loss_heading_without_tier_is_preserved_and_flagged() -> None: + # Regression: a disclosure heading with no equivalence-tier line must not be + # silently dropped; it is preserved with an empty tier and fails closed. + doc = "# losses\n\n## loss-no-tier\n\nNarrative with no tier line.\n" + parsed = sl.parse_loss_disclosures(doc) + assert parsed == {"loss-no-tier": ""} + + # A source row that references it (empty tier) is rejected, and the empty + # disclosure itself is reported. + row = _row("stochastic-binding-loss") + row["loss_ref"] = "loss-no-tier" + ctx = _context() + ctx["losses"] = {"loss-no-tier": ""} + problems = sl.validate_source_ledger([row], **ctx) + assert any(p.row_id == "loss-no-tier" and p.field == "equivalence_tier" for p in problems)