Repository navigation
question: re-examine whether scoring/reward language belongs in ACES #671
Description
Open question
Should ACES carry scoring / grading language at all? Specifically the OCR-inherited SDL scoring pipeline — metrics → evaluations → tlos → goals — and the CybORG-inherited agents.reward_calculator field.
This is an open design question, not a decision. It likely needs an ADR.
Where it came from
Surfaced authoring CAGE-2 as an ACES SDL scenario (REP-003, #637). We drew a boundary that the current scoring surfaces sit awkwardly against:
ACES specifies the experiment — the environment, the participants, what they can do and perceive, their objectives, and the execution controls that shape a run within its horizon. It does not specify what a researcher does with the output of a run. RL reward-for-training, model updates, leaderboard ranking, and downstream statistical analysis are consumers of the experiment's data, not part of the experiment itself.
Discriminator: a signal is in scope only if it is used within the experiment by the participants. CAGE-2's reward fails that test — in the evaluation protocol a fixed/pre-trained policy is run and reward is accumulated as the researcher's score; it is a measurement extracted from the run and consumed by training, not a signal a participant reads and acts on during the run.
The tension
agents.reward_calculator(e.g. CAGE'sHybridAvailabilityConfidentiality) is training/scoring machinery — by the boundary above it should not be an authored ACES environment fact.metrics/evaluations/tlos/goalsare an Open-Cyber-Range training-exercise scoring pipeline. Reconstructing a graded score inside the SDL looks like the same data-use concern relocated into the language.- By contrast,
objectives(participant goals) andconditions(observable state facts like "OTService available", "no red privilege on Op_Server0") are legitimately within the horizon — they describe what participants are trying to do and the state of the run.
The question(s) to resolve
- Do the scoring sections (
metrics/evaluations/tlos/goals) andagents.reward_calculatorbelong in ACES, or are they vestigial given the experiment-vs-data-use boundary? - If objectives need success criteria, should those reference conditions / observable state rather than a score-shaped
metrics→evaluationspipeline? - If scoring/reward genuinely belongs somewhere in ACES, is that the SDL (authored scenario meaning) or an experiment/evaluator contract — and only when a participant consumes it in-run?
- What is the migration/deprecation path for existing scenarios (e.g.
techvault,paper-agent-loop) that use the scoring pipeline?
Not in scope here
Deciding the answer. This issue is to schedule the examination and produce an ADR.
Related: #637 (REP-003), ADR-069, ADR-020.
Activity
Brad-Edwards commentedon Jul 5, 2026
Downstream dependency recorded: OpenRAE/lilrae#606 is now blocked by this issue.
APTL should not continue designing a resolution for its evaluator/scoring manifest surface until ACES decides whether scoring/reward/evaluation surfaces are in scope and lands the corresponding language/contract changes. If ACES removes or narrows those surfaces, APTL should follow by removing or narrowing its declared evaluator/scoring support rather than implementing the old surface more completely.
Brad-Edwards commentedon Jul 5, 2026
🛠️ Picked up by /implement — driver Claude Code, branch
671-scoring-reward-scope, 2026-07-05T03:53:50Z.Brad-Edwards commentedon Jul 5, 2026
gc workflow phase recorded:
preflight(issue #671). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Brad-Edwards commentedon Jul 5, 2026
gc_codex_review — cycle 1 of 1 (pre-push) on issue #671 (branch
671-scoring-reward-scope)Core review
Verdict:
shipThis is shaped correctly for a proposed boundary ADR: it does not smuggle schema or runtime changes into the decision record, it makes the experiment-vs-data-use seam explicit, and it routes score/reward concerns to the existing experiment/evidence/evaluator vocabulary instead of inventing a parallel SDL mechanism. The assessment-semantics update is scoped as a compatibility guardrail while the ADR is proposed, and the research notes provide falsifiable inventory rather than becoming contract authority. I do not see it foreclosing the obvious next variation; if reviewers accept the ADR, the migration path is already staged as downstream implementation work.
No blocking findings.
Security review
Verdict:
shipThis is a documentation-only change that adds a proposed ADR, research notes, changelog text, and reference-guide guardrails around whether scoring and reward concepts belong in authored SDL or in the experiment/evaluator plane. The change fits as a decision record and supporting evidence package: it does not alter schemas, validators, runtime code, repository queries, endpoint behavior, serialization, or credential handling. The security-relevant boundary it touches is semantic rather than executable, and this diff does not introduce a concrete attack path.
No blocking findings.
Brad-Edwards commentedon Jul 5, 2026
gc_codex_review pre-push cycle 1 of 1 complete for issue #671 on branch '671-scoring-reward-scope'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Brad-Edwards commentedon Jul 5, 2026
Review decision record — codex cycle 1 (issue #671)
Reviewer: codex
Cycle: 1
Architectural read:
Blocking findings: 0 (clean run)
Brad-Edwards commentedon Jul 5, 2026
gc_test_quality_review cycle 1 of 1 — issue #671
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:
671-scoring-reward-scopeCycle: 1 / 1
Findings: 0 (clean run)
Brad-Edwards commentedon Jul 5, 2026
gc_test_quality_review cycle 1 of 1 complete for issue #671 on branch '671-scoring-reward-scope'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Brad-Edwards commentedon Jul 5, 2026
Review decision record — test-quality cycle 1 (issue #671)
Reviewer: test-quality
Cycle: 1
Blocking findings: 0 (clean run)
Brad-Edwards commentedon Jul 5, 2026
GRC screening record — issue #671
Schema:
gc.implement.grc-screening/v2Derived verdict:
not_security_relevant(computed from derived facts + the GRC coverage graph — not agent-asserted)Rationale: Docs-only change: proposed ADR-073, scoring-scope research notes, a SEM-206 assessment-semantics compatibility guardrail, README/toctree/changelog. No code, schema, contract, or runtime surface changes; no security-relevant behavior introduced. The ADR proposes (defers) future removal of SDL scoring surfaces; any such removal is separate downstream work.
Provenance:
4df4e8e5c066bbbc1d2612b066fe1a8aff0393b1→ heada1fb96enone· architecture snapshot:noneImpact set (existing coverage the change touches) — 0:
Gap set (touched security-relevant surface with no coverage — must be modeled, controlled, or dispositioned before completion) — 0:
Stale set (linked entities whose underlying code changed) — 0:
Candidate threats (GC-GRC-007, deterministic) — 0:
Candidate controls (GC-GRC-008, deterministic) — 0:
Brad-Edwards commentedon Jul 5, 2026
gc workflow phase recorded:
grc_screening(issue #671). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Brad-Edwards commentedon Jul 5, 2026
gc workflow phase recorded:
plan(issue #671). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Plan — scoring/reward language scope (proposed ADR)
Deliverable (confirmed with maintainer): a PROPOSED ADR + research notes. The issue asks to schedule the examination and produce an ADR, not to decide the answer. Direction confirmed: recommend removal (full), research-notes-first.
Approach
docs/research/scoring-scope/(mirrors the ADR-055 precedent):prior-art-and-design-criteria.md(reward hypothesis; specification gaming; experiment-database separation; the three internal boundary ADRs) andscoring-surface-inventory.md(the falsifiable per-surface map + discriminator verdict). Registered indocs/index.md.docs/decisions/adrs/adr-073-scoring-reward-language-scope.md, registered in the ADR README toctree + table (not pinned inadr-index.yaml— proposed ADRs are not pinned).docs/explain/reference/assessment-semantics.md, from the architecture preflight): treat the current pipeline as a compatibility surface, not permission to expand scoring language while the boundary is open.changelog.d/671.added.md; DOCUMENTS traceability link ADR-073 → SEM-206.Recommendation recorded in ADR-073
The OCR-inherited SDL scoring pipeline (
metrics/evaluations/tlos/goals) and the CybORGagents.reward_calculatorlabel are vestigial against the experiment-vs-data-use boundary already drawn by ADR-055 (metrics/tasks/studies in experiment-core), ADR-064 (evaluations/derived-measures), and ADR-069 §3 (reward/scoring as a backend-Evaluator projection). None is read by a participant in-horizon. Recommend: deprecate then remove those surfaces; keepobjectives+conditions; narrowobjectives.successto observable state; route graded scoring to the experiment/evaluator contracts only. Staged migration for the four study-style scenarios + two library artifacts; ADR-002 amendment; SEM-206 update; APTL OpenRAE/lilrae#606 follows by narrowing its evaluator/scoring surface.Scope
This change introduces no schema/model/scenario edits — it is doc-only. Acceptance of the ADR and the subsequent removal/migration are downstream work spawned on acceptance. Status stays proposed; the maintainer ratifies at review/merge.
Gates
GRC screening: not_security_relevant (docs-only).
nox verifygreen (incl. Sphinx docs build, ADR pin, schema manifest/drift, full test suite). Pre-push codex + test-quality reviews: clean.Add proposed ADR-073 on scoring/reward language scope
Brad-Edwards commentedon Jul 5, 2026
gc workflow phase recorded:
grc_reconciled(issue #671). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.GRC screening
gc.implement.grc-screening/v2reconciled against the final diff — 0 impacted, 0 gaps, 0 stale (reassessment tracked by GC-GRC-012).Brad-Edwards commentedon Jul 5, 2026
Ready for review — issue #671
PR: #681
Outcome
Records a proposed architecture decision (ADR-073) that ACES drop the OCR/CybORG scoring-and-reward surfaces (metrics/evaluations/tlos/goals and agents.reward_calculator) from the SDL as vestigial, keep objectives+conditions, narrow objective success to observable state, and route graded scoring to the experiment/evaluator plane. No runtime, schema, or scenario behavior changes in this PR — it is a doc-only proposed decision plus research notes and a SEM-206 compatibility guardrail; acceptance and any removal are deferred to maintainer review and downstream issues.
Files changed
Added:
docs/decisions/adrs/adr-073-scoring-reward-language-scope.mddocs/research/scoring-scope/index.mddocs/research/scoring-scope/prior-art-and-design-criteria.mddocs/research/scoring-scope/scoring-surface-inventory.mdchangelog.d/671.added.mdModified:
docs/index.mddocs/decisions/adrs/README.mddocs/explain/reference/assessment-semantics.mdReviews
Traceability reconciliation
Status
Documentation
Updated: see diff.
Brad-Edwards commentedon Jul 5, 2026
gc workflow phase recorded:
traceability_reconciled(issue #671). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.no in-scope requirements; touched-files audit clean.
Brad-Edwards commentedon Jul 5, 2026
gc workflow phase recorded:
grc_reconciled(issue #671). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.GRC screening
gc.implement.grc-screening/v2reconciled against the final diff — 0 impacted, 0 gaps, 0 stale (reassessment tracked by GC-GRC-012).Brad-Edwards commentedon Jul 5, 2026
Final report — issue #671 complete
PR: #681
Outcome
Merged ADR-073 (proposed): a recorded decision that ACES drop the OCR/CybORG scoring-and-reward surfaces (metrics/evaluations/tlos/goals, agents.reward_calculator) from the SDL as vestigial, keep objectives+conditions, narrow objective success to observable state, and route graded scoring to the experiment/evaluator plane. This PR shipped only the decision, research notes, and a SEM-206 compatibility guardrail — no runtime/schema/scenario behavior changed. Removal and migration are deferred to a downstream realization issue.
Files changed
Reviews
Traceability reconciliation
Requirement-free design-ADR run. ADR-073 DOCUMENTS SEM-206 (Assessment Semantics). No IMPLEMENTS/TESTS surface introduced; removal/migration deferred to a downstream realization issue.
Status
Documentation
Updated: see diff.