Repository navigation
fix: remove hardcoded capability and evidence-satisfaction gates from every adapter #84
Description
Activity
🛠️ Picked up by /implement - driver codex, branch
84-remove-hardcoded-gates, 2026-08-12T15:08:55.826Z.gc workflow phase recorded:
preflight(issue #84). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.gc workflow phase recorded:
plan(issue #84). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Implementation plan
Claim boundaries and fail-closed admission
- Remove
_BackendAdapter.evidence_satisfies_refsand every backend-name evidence-reference allowlist. Episode evidence artifacts will describe only the evaluator-summary bytes actually emitted and will carry no semanticsatisfies_refsclaim. - Add a pre-effect researcher admission check over the published
ExperimentTaskModeland liveBackendManifest. The current RAES 3.3 manifest/run contracts cannot bind a semantic evidence reference to required artifact fields or their unavailable/redacted/withheld/lossy status, so every current task with semantic evidence requirements is rejected before runtime planning, native source import, output reservation, or simulator effects. Do not weaken the authored tasks; issues 85–88 own real per-adapter remediation. - Keep
validate_experiment_run_against_task()and study validation as the authoritative post-run joins. Add no adapter schema, vocabulary, compatibility allowlist, backend exception, or test bypass.
Capability accounting
- Delete the static capability-pointer-to-evidence maps, broad passed-reference joins, and positive
*_manifest_capability_evidenceAPIs from shared, CybORG, NASim, CyberBattleSim, and PrimAITE conformance composition. - Retain manifest pointer traversal only as truthful inventory. Conformance bundles will expose every affirmative manifest leaf as an unresolved
capability_gapsentry and will never turn a broad conformance/source/probe pass into per-leaf evidence. Canonical RAES conformance reports and backend-local diagnostics remain separate bounded facts. - Remove unconditional “standard capability” grants from shared manifest plumbing. Backend modules must construct their own published RAES capability models explicitly; this keeps backend declarations local while issues 85–88 reconcile each declaration with production capture evidence.
TDD and repository-wide regression coverage
- First add repository-wide negative tests that fail on the current tree: no registered adapter may carry a static satisfaction list; emitted evaluator-summary artifacts have empty
satisfies_refs; every current evidence-requiring task is rejected before native/runtime/output effects; every affirmative manifest leaf is automatically reported as a gap; and an injected new leaf is also a gap. - Update focused CLI tests to assert the bounded validation failure and absence of output/effects for current tasks. Keep component/evaluator tests that prove the actual sanitized records and measures still work without claiming task satisfaction.
- Update conformance tests and installed-wheel probes to assert canonical report/diagnostic serialization plus unresolved capability inventory, never positive per-leaf capability evidence. Add an AST-backed repository invariant so reintroducing the retired allowlist/map APIs fails across every registered adapter.
Documentation and whole-repository fit
- Retain the architecture-preflight note and navigation entry, and update the four backend conformance guardrails where they currently endorse static pointer closure. The docs will distinguish finite conformance, source validation, emitted artifacts, manifest declarations, and semantic task satisfaction.
- Security: all rejection occurs before native import/effects/output; existing closed parsers, source/pack/participant joins, RAES validators, confined outputs, redaction, and bounded error envelopes remain active. No auth, secret, environment, argv, network, persistence, or native-output surface is added.
- Maintainability: reuse
BackendManifest,backend_manifest_payload(),ExperimentTaskModel, the canonical task/run validators, and the existing affirmative-pointer traversal. Add no second validator family or cross-simulator semantic authority. - Extensibility: future RAES field-verifiable contracts can replace the single fail-closed admission seam; a new adapter or capability leaf is automatically covered without editing an allowlist.
- Run focused claim-integrity, CLI, conformance, backend, and installed-probe tests during TDD, then the canonical requirement-free graph:
uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify -- --skip-requirement. The Conventional Commit PR title will befix: remove hardcoded capability and evidence gates; Release Please owns version/changelog changes.
- Remove
Execution obligation OBL-84-DISTRIBUTION-PROBE — Opened
Category: failing_check
Observed state: The implementation correctly rejected unverifiable task evidence before execution, but nox treated the expected exit code 3 as a distribution failure.
Impact: The canonical completion graph could not pass because its installed-wheel probes still required native validation success after semantic evidence admission became fail-closed.
Current obligation: Update every installed-wheel native validation probe to require the bounded validation exit while semantic evidence requirements remain unverifiable.Evidence
- gc_implement_mechanical verify attempt issue84-verify-1 failed at completion_gate
- nox -s distributions reproduced the failure: clean-installed cyberbattlesim validate exited 3 after fail-closed task admission
Execution obligation OBL-84-DISTRIBUTION-PROBE — Resolved
Category: failing_check
Observed state: The focused distribution session is green with all three native validation probes rejecting unverifiable evidence and every remaining package/conformance gate passing.
Impact: The distribution gate now proves both installed-package integrity and fail-closed native admission.
Current obligation: Update every installed-wheel native validation probe to require the bounded validation exit while semantic evidence requirements remain unverifiable.Evidence
- noxfile.py clean-installed cyberbattlesim, NASim, and CybORG validation probes now require exit code 3
- uv tool run --from nox[uv]==2026.4.10 nox -s distributions completed successfully in 48 seconds
Disposition: fix
Corrective action: Added success_codes=[3] to the three clean-installed native validation probes so an unexpected success remains a failure while the intended bounded rejection passes.Verification
- The CyberBattleSim probe exited 3 before pack gates, which then passed.
- The NASim probe exited 3 before conformance and pack gates, which then passed.
- The CybORG probe exited 3 before conformance, and the complete distributions session passed.
gc_codex_review — cycle 1 of 1 (pre-push) on issue #84 (branch
84-remove-hardcoded-gates)
Diff mode: inline — the complete diff was supplied in one promptCore review
Verdict:
ship-with-fixesThe production shape is largely correct: it removes adapter-local satisfaction authority, centralizes unresolved capability inventory, keeps semantic evidence claims behind the RAES-owned task/run boundary, and fails closed before runtime effects. The backend-local capability construction preserves the ownership seam without introducing a competing schema and leaves room for a future published witness contract. The material weakness is in the canonical verification graph: the installed-wheel probes now accept every validation failure with exit code 3, not specifically the intended evidence-admission rejection.
Blocking findings (1):
- [class] Installed-wheel probes accept unrelated validation regressions —
noxfile.py:616
Allowing exit code 3 as an unconditional success means these distribution probes pass for any validation failure, including missing packaged resources, broken task/participant joins, or another installed-wheel-only regression. Source-tree tests that inspect the expected diagnostic do not preserve the clean-install guarantee. Make the installed probe assert both the intended exit code and the bounded evidence-admission diagnostic so only the deliberate fail-closed outcome is accepted.
Security review
Verdict:
shipThis change is shaped correctly around a fail-closed admission seam: authored evidence requirements, manifest capability declarations, runtime artifacts, and satisfaction claims remain distinct, while unverifiable semantic requirements are rejected before native import, output reservation, or simulator effects. The shared traversal now inventories unresolved affirmative capability leaves without manufacturing evidence, and backend-specific manifest construction remains within each adapter. The change touches validation, portable artifact projection, conformance reporting, and leakage boundaries without introducing a new input, authentication, execution, filesystem, or secrets surface. I found no concrete exploitable security issue introduced by the diff.
No blocking findings.
- [class] Installed-wheel probes accept unrelated validation regressions —
gc_codex_review pre-push cycle 1 of 1 complete for issue #84 on branch '84-remove-hardcoded-gates'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Review decision record — codex cycle 1 (issue #84)
Reviewer: codex
Cycle: 1Architectural read:
Core reviewer: The production shape is largely correct: it removes adapter-local satisfaction authority, centralizes unresolved capability inventory, keeps semantic evidence claims behind the RAES-owned task/run boundary, and fails closed before runtime effects. The backend-local capability construction preserves the ownership seam without introducing a competing schema and leaves room for a future published witness contract. The material weakness is in the canonical verification graph: the installed-wheel probes now accept every validation failure with exit code 3, not specifically the intended evidence-admission rejection.
Security reviewer: This change is shaped correctly around a fail-closed admission seam: authored evidence requirements, manifest capability declarations, runtime artifacts, and satisfaction claims remain distinct, while unverifiable semantic requirements are rejected before native import, output reservation, or simulator effects. The shared traversal now inventories unresolved affirmative capability leaves without manufacturing evidence, and backend-specific manifest construction remains within each adapter. The change touches validation, portable artifact projection, conformance reporting, and leakage boundaries without introducing a new input, authentication, execution, filesystem, or secrets surface. I found no concrete exploitable security issue introduced by the diff.
Blocking findings: 1
Finding 1 —
class(3 instances)- ID:
F1 - Title: [core] Installed-wheel probes accept unrelated validation regressions
- Location:
noxfile.py:616 - Decision: fix
- Rationale: Allowing exit code 3 as an unconditional success means these distribution probes pass for any validation failure, including missing packaged resources, broken task/participant joins, or another installed-wheel-only regression. Source-tree …
- Instances:
noxfile.py:616noxfile.py:707noxfile.py:812
- ID:
gc_test_quality_review cycle 1 of 1 — issue #84
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:84-remove-hardcoded-gates
Cycle: 1 / 1
Findings: 1Finding 1 — [critical]
tests/test_claim_integrity.py::test_source_tree_contains_no_retired_static_claim_gate_identifiersProblem: The AST sweep's retired-name detection only catches the bare
_manifest_capability_evidencesuffix and a small exact-name set. It misses the_gaps-suffixed variant of the same gate (nasim_manifest_capability_evidence_gaps,cyborg_manifest_capability_evidence_gaps,cyberbattlesim_manifest_capability_evidence_gaps,primaite_manifest_capability_evidence_gaps), the base module namesmanifest_capability_evidence_gapsandpassed_probe_evidence_refsin_conformance_support.py, and the sharedcapability_evidence/capability_evidence_gapsin_gym_backend/conformance.pythat every backend's evidence gate was actually built on.
Why it matters: This test is the repository's designated backstop against exactly the regression class this PR removes (a static probe-requirement table certifying a manifest leaf). If a future change restoresnasim_manifest_capability_evidence_gaps(or the sharedgym.capability_evidence_gaps) with the old table-driven logic verbatim, this test still passes — the reintroduced hardcoded gate ships silently, which is the specific failure mode the guardrails doc calls out ("Do not retain ... a static reference reintroduced through any shared path").
Fix: Replace the suffix heuristic with a substring/regex check formanifest_capability_evidenceanywhere in the identifier (not just as a trailing suffix), and addcapability_evidence,capability_evidence_gaps, andpassed_probe_evidence_refsto_RETIRED_EXACT_IDENTIFIERS(or an equivalent pattern) so every retired name across_gym_backend/conformance.pyand_conformance_support.pyis covered, not just the four backend-prefixed_evidencenames.gc_test_quality_review cycle 1 of 1 complete for issue #84 on branch '84-remove-hardcoded-gates'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Review decision record — test-quality cycle 1 (issue #84)
Reviewer: test-quality
Cycle: 1Architectural read:
The change is shaped correctly at the architecture level: it deletes a whole family of "manufactured evidence" functions (
manifest_capability_evidence,manifest_capability_evidence_gaps,passed_probe_evidence_refs,evidence_satisfies_refs, thestandard_*_capabilitiesfactories) and replaces the claim surface with an honest, unconditional inventory (manifest_capability_gaps/_task_capture_admission_gaps) that fails closed. The test suite was updated in lockstep across all four backends plus a new repository-widetest_claim_integrity.py, and the new admission gate (_EvidenceUnverifiableFailure) is exercised end-to-end for every registered adapter via_TASKS/_BACKENDSset-containment, which is exactly the kind of extensibility guard the design doc asks for. Where this falls short is the one test explicitly built to be the last line of defense against reintroducing the retired pattern (test_source_tree_contains_no_retired_static_claim_gate_identifiers): its identifier list and suffix heuristic cover only a minority of the actual retired names, so the regression it exists to catch — a future PR quietly restoring a static evidence-gate function — would slip through undetected in most backends and in the shared_gym_backendimplementation itself.Non-blocking observation: the new early-return branch in
_task_capture_admission_gaps(if not required: return ()) has no test coverage anywhere in the diff — every current task requires evidence, so only the fail-closed branch is exercised. Given the whole point of the seam is to eventually let a fully-witnessed task pass, this pass-through path deserves at least one direct unit test constructing a task with no observation/metric evidence requirements.Blocking findings: 1
Finding 1 —
class(2 instances)- ID:
F1 - Title: (no title)
- Decision: fix
- Rationale: Addressed by next cycle
- Instances:
tests/test_claim_integrity.py:44 (_RETIRED_EXACT_IDENTIFIERS missing passed_probe_evidence_refs, capability_evidence, capability_evidence_gaps)tests/test_claim_integrity.py:170-173 (endswith("_manifest_capability_evidence") never matches *_manifest_capability_evidence_gaps in nasim/cyborg/cyberbattlesim/primaite conformance modules or manifest_capability_evidence_gaps in _conformance_support.py)
- ID:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd6aea6d6a751b8e3315103aff1ec31e136c2497e - Outcome:
already_current - Published feature head:
d9ae23f19ad2b6aa0cc890f5456715ae2c8b489c - Verified tree:
bf62689019ba5c9ca2265d684d0aa66dc8e82f03
- Source:
Execution obligation OBL-84-SONAR-DUPLICATION — Opened
Category: failing_check
Observed state: SonarCloud quality gate is ERROR solely for new-code duplication density.
Impact: PR #90 cannot pass its required SonarCloud and aggregate PR gates.
Current obligation: Remove the measured new-code duplication without restoring unconditional shared capability grants, then re-run local and remote quality gates.Evidence
- GitHub Actions run 31622582339: all code, tests, policy, docs, and distribution jobs passed; SonarCloud Scan failed the quality gate
- SonarCloud PR 90 reported new_duplicated_lines_density=6.3 above the 3 percent threshold, with zero open issues and 88.7 percent new-code coverage
- Duplicate blocks were in repeated admission error projection in cli.py and repeated cleanup/capability assembly in the NASim and CyberBattleSim manifests
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd6aea6d6a751b8e3315103aff1ec31e136c2497e - Outcome:
already_current - Published feature head:
1827251861b3c852ec94ef6bd1c912e04fda35ab - Verified tree:
67532b36594a073216ea79fbcd5d2d7d51e63d03
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd6aea6d6a751b8e3315103aff1ec31e136c2497e - Outcome:
already_current - Published feature head:
b633ec2d45469e3991541345b6098d11302f6e94 - Verified tree:
5512e039b954c1d012148720d51bbdcdabcb1000
- Source:
Execution obligation OBL-84-SONAR-DUPLICATION — Resolved
Category: failing_check
Observed state: The final analysis removed both measured duplicate blocks while keeping all backend capability values explicit.
Impact: PR #90 now passes its CI and SonarCloud quality gates.
Current obligation: Remove the measured new-code duplication without restoring unconditional shared capability grants, then re-run local and remote quality gates.Evidence
- GitHub Actions CI run 31628541613 completed successfully at commit b633ec2
- SonarCloud PR 90 quality gate is OK with new_duplicated_lines_density=0.0, new_coverage=91.2, zero open issues, and zero unreviewed hotspots
- Local make verify and make policy passed on the final repair tree before publication
Disposition: fix
Corrective action: Centralized stable CLI admission-error projection and neutral RAES capability-set model assembly; backend capability values and cleanup declarations remain explicit inputs.Verification
- Canonical local completion and policy gates passed.
- All GitHub CI jobs, CodeQL, and PR title lint passed.
- SonarCloud quality gate passed with no issues or hotspots.
Ready for review — issue #84
PR: #90
Plan: #84 (comment)Outcome
Adapters no longer certify capabilities or task evidence from static allowlists. Native runs now stop before effects when the published RAES manifest cannot verify required evidence, and all affirmative capability leaves remain visible as unresolved inventory.
Removed hardcoded capability and evidence-satisfaction gates across CybORG, NASim, CyberBattleSim, and PrimAITE; added fail-closed admission and repository-wide claim-integrity coverage.
Files changed
Added:
docs/decisions/capability-and-evidence-claim-guardrails.mdtests/test_claim_integrity.py
Modified:
docs/cyberbattlesim-researcher-command.mddocs/decisions/cyberbattlesim-conformance-guardrails.mddocs/decisions/cyberbattlesim-qualification-guardrails.mddocs/decisions/cyborg-conformance-guardrails.mddocs/decisions/nasim-conformance-guardrails.mddocs/decisions/primaite-conformance-guardrails.mddocs/nasim-researcher-command.mddocs/researcher-command.mdmkdocs.ymlnoxfile.pysrc/raes_adapters/_conformance_support.pysrc/raes_adapters/_gym_backend/conformance.pysrc/raes_adapters/_manifest_support.pysrc/raes_adapters/cli.pysrc/raes_adapters/cyberbattlesim/backend/__init__.pysrc/raes_adapters/cyberbattlesim/backend/conformance.pysrc/raes_adapters/cyberbattlesim/backend/manifest.pysrc/raes_adapters/cyberbattlesim/researcher.pysrc/raes_adapters/cyborg/__init__.pysrc/raes_adapters/cyborg/conformance.pysrc/raes_adapters/nasim/backend/__init__.pysrc/raes_adapters/nasim/backend/conformance.pysrc/raes_adapters/nasim/backend/manifest.pysrc/raes_adapters/nasim/researcher.pysrc/raes_adapters/primaite/backend/__init__.pysrc/raes_adapters/primaite/backend/conformance.pytests/test_cyberbattlesim_conformance.pytests/test_cyberbattlesim_researcher_cli.pytests/test_cyborg_conformance.pytests/test_nasim_conformance.pytests/test_nasim_researcher_cli.pytests/test_primaite_backend.pytests/test_primaite_conformance.pytests/test_researcher_cli.py
Reviews
- codex: Reviewed the complete pre-push diff; its distribution-probe finding was fixed across all installed-wheel validations, and the user directed proceeding without an over-cap cycle.
- test-quality: Reviewed changed tests; the retired-identifier family guard was expanded to cover shared and gaps-suffixed variants, then focused and canonical verification passed.
Traceability reconciliation
- Pending — requirement status transition and IMPLEMENTS/TESTS reconciliation run in Phase E once the PR merges.
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge. Ground Control reconciliation (requirement status + traceability) runs on merge (Phase E).
Final report — issue #84 complete
PR: #90
Plan: #84 (comment)Outcome
Adapters no longer certify capabilities or task evidence from static allowlists. Native runs stop before effects when published RAES contracts cannot verify required evidence, while unresolved affirmative capability leaves remain visible for remediation.
Merged PR #90 removes hardcoded capability and evidence-satisfaction gates across CybORG, NASim, CyberBattleSim, and PrimAITE, with fail-closed admission and repository-wide regression coverage.
Files changed
Added:
docs/decisions/capability-and-evidence-claim-guardrails.mdtests/test_claim_integrity.py
Modified:
docs/cyberbattlesim-researcher-command.mddocs/decisions/cyberbattlesim-conformance-guardrails.mddocs/decisions/cyberbattlesim-qualification-guardrails.mddocs/decisions/cyborg-conformance-guardrails.mddocs/decisions/nasim-conformance-guardrails.mddocs/decisions/primaite-conformance-guardrails.mddocs/nasim-researcher-command.mddocs/researcher-command.mdmkdocs.ymlnoxfile.pysrc/raes_adapters/_conformance_support.pysrc/raes_adapters/_gym_backend/conformance.pysrc/raes_adapters/_manifest_support.pysrc/raes_adapters/cli.pysrc/raes_adapters/cyberbattlesim/backend/__init__.pysrc/raes_adapters/cyberbattlesim/backend/conformance.pysrc/raes_adapters/cyberbattlesim/backend/manifest.pysrc/raes_adapters/cyberbattlesim/researcher.pysrc/raes_adapters/cyborg/__init__.pysrc/raes_adapters/cyborg/conformance.pysrc/raes_adapters/nasim/backend/__init__.pysrc/raes_adapters/nasim/backend/conformance.pysrc/raes_adapters/nasim/backend/manifest.pysrc/raes_adapters/nasim/researcher.pysrc/raes_adapters/primaite/backend/__init__.pysrc/raes_adapters/primaite/backend/conformance.pytests/test_cyberbattlesim_conformance.pytests/test_cyberbattlesim_researcher_cli.pytests/test_cyborg_conformance.pytests/test_nasim_conformance.pytests/test_nasim_researcher_cli.pytests/test_primaite_backend.pytests/test_primaite_conformance.pytests/test_researcher_cli.py
Reviews
- codex: Reviewed the complete pre-push diff; the installed-wheel validation finding was fixed across all probes, and the user authorized proceeding without an over-cap cycle.
- test-quality: Reviewed changed tests; the retired-identifier guard was expanded across the full helper family and all verification passed.
Traceability reconciliation
- IMPLEMENTS / TESTS / DOCUMENTS added: 0
- Links updated: 0
- Stale links removed: 0
Requirement-free bug fix; no in-scope requirement records or repository requirement files existed to transition or reconcile.
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge.
Objective
Remove every hardcoded capability and evidence-satisfaction gate from every adapter, even when doing so breaks an adapter until its real capture path is implemented.
Problem
#30 exposed hardcoded evidence-reference claims for
attacker-action-logandavailability-series. The action log was not emitted, yet the static claims allowed the run to appear to satisfy the authored contract.Scope
Acceptance criteria
References