Repository navigation
question: no authoring surface for experiment-core specs (run/study/apparatus) — archival-only today #675
Description
Observation
SDL scenarios are authored as files (examples/scenarios/*.sdl.yaml), validated against a published schema, and — via agents + the MCP — are a first-class authoring surface.
The experiment-core contracts (ExperimentRunModel, ExperimentStudyModel, ExperimentApparatusContextModel, ExperimentRunAllocationPlanModel, capture spec, …) are archival / provenance OUTPUT objects emitted by a run (ExperimentRunModel docstring: "Archival provenance record for one execution"). The only in-repo instances are conformance fixtures; there is no authoring path — file or MCP tool — to specify an experiment (run count, seeds, red-variant selection, turn order, step count, termination, condition assignments) as an input artifact before execution.
Why it matters
A full CAGE-2 spec is scenario + experiment. The experiment half has no authoring home today, so "fully specify the CAGE-2 experiment" is not a supported workflow.
Question to examine
Should experiment specs get an authoring surface (agent/MCP first, optional authored file + published schema, discovery/validation test) analogous to SDL scenarios, so an experiment can be authored and validated pre-run rather than only captured post-run? (Agents + MCP are the intended authoring surface; the human CLI/GUI surfaces #85/#263 are support.)
Activity
Brad-Edwards commentedon Jul 8, 2026
🛠️ Picked up by /implement — driver Claude Code, branch
675-experiment-authoring, 2026-07-08T00:37:48Z.Brad-Edwards commentedon Jul 8, 2026
gc_codex_review — cycle 1 of 1 (pre-push) on issue #675 (branch
675-experiment-authoring)Core review
Verdict:
ship-with-fixesThis is shaped correctly at the contract boundary: it introduces a separate pre-run authoring contract instead of weakening the archival run/study/apparatus-context records, reuses the existing ContractModel/schema bundle/manifest/fixture/test/MCP vocabulary, and keeps runtime execution out of scope. The design seam is the ExperimentSpecModel/ExperimentRunPlanModel layer, with model validators carrying the cross-field invariants JSON Schema cannot express; that fits the ADR direction and does not foreclose the obvious next variation where orchestration consumes the spec and emits archival records.
Blocking findings (1):
implementations/python/packages/aces_contracts/contracts.py:5592The top-level invariant says every allocation blocking factor must resolve to a declared experiment-spec factor, but the validator only runs when
self.factorsis truthy. A spec withrun_plan.allocation.blocking_factors: ['site']and nofactorsmap will pass Pydantic validation even though the formal spec andx-aces-invariantrequire rejection. Remove theand self.factorsguard and compare againstset(self.factors)unconditionally whenever an allocation exists; add an invalid fixture for the empty/missing factors case.Security review
Verdict:
shipThis change is shaped as a contract and authoring-tool addition: it introduces a pre-run experiment-spec surface beside the existing archival experiment-core outputs, wires it into schema publication, examples, MCP discovery, and conformance tests, and keeps execution/orchestration out of scope. The main security-relevant seam is the MCP authoring boundary, where untrusted YAML enters validation and where example retrieval could have become file access; the diff uses safe YAML loading, a size cap on MCP input, and an allowlisted example map rather than user-controlled paths. I do not see a concrete exploitable weakening of an existing security control.
No blocking findings.
Brad-Edwards commentedon Jul 8, 2026
gc_codex_review pre-push cycle 1 of 1 complete for issue #675 on branch '675-experiment-authoring'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Brad-Edwards commentedon Jul 8, 2026
Review decision record — codex cycle 1 (issue #675)
Reviewer: codex
Cycle: 1
Architectural read:
Blocking findings: 1
Finding 1 —
one-offF1implementations/python/packages/aces_contracts/contracts.py:5592self.factorsis truthy. A spec withrun_plan.allocation.blocking_factors: ['site']and no…Brad-Edwards commentedon Jul 8, 2026
gc_test_quality_review cycle 1 of 1 — issue #675
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:
675-experiment-authoringCycle: 1 / 1
Findings: 0 (clean run)
Brad-Edwards commentedon Jul 8, 2026
gc_test_quality_review cycle 1 of 1 complete for issue #675 on branch '675-experiment-authoring'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Brad-Edwards commentedon Jul 8, 2026
Review decision record — test-quality cycle 1 (issue #675)
Reviewer: test-quality
Cycle: 1
Architectural read:
Blocking findings: 0 (clean run)
feat: add experiment authoring-input contract and MCP authoring surface
Brad-Edwards commentedon Jul 8, 2026
✅ Ready for review — Phase D complete (PR #711)
All automated gates are green; the only remaining step is your review + merge of PR #711.
What shipped:
experiment-authoring-input-v1(ExperimentSpecModel) — a pre-run experiment specification (task binding + run plan: seeds, episode controls, red-variant selection, condition allocation) that answers this issue's gap. It is a separate input contract that references the archival experiment-core outputs rather than weakening them, mirroringsdl-authoring-input-v1↔instantiated-scenario-v1. Includes a loader, MCP authoring tools (experiment_scaffold/experiment_validate/experiment_get_example), workedexamples/experiments/*.exp.yaml, fixtures, and a discovery test.Decision record: ADR-074 (accepted) decides the surface; ADR-055 and ADR-069 amended (cross-reference rows; pins unchanged). ADR-055 anticipated exactly this ("explicit draft/authoring surface … rather than weakening these archival contracts"), and it gives ADR-069's REP-003 a pre-run home for CAGE-2 execution-control facts.
Gates:
nox verify, including wheel/corpus packaging).Run type: requirement-free (this is a
questionissue with no requirements in scope). No requirement transition or issue→requirement link applies; the change governs a new experiment-core contract, recorded in ADR-074 and the experiment-core formal spec.Merge PR #711 when ready; I'll run Phase E (final report + issue close) after the merge.
feat: add experiment authoring-input contract and MCP authoring surfa…
Brad-Edwards commentedon Jul 9, 2026
Governance update: created EXP-736 — Experiment Authoring Input Specification (ACTIVE) to govern the new contract, with full IMPLEMENTS/TESTS traceability (contract/loader/MCP source + published schema + test modules + ADR-074 + formal spec);
check_requirement_governance.pypasses. Also filed #712 to add a detection for any published contract not tied to a requirement.Brad-Edwards commentedon Jul 10, 2026
✅ Phase E complete — merged & reconciled
PR #711 merged into
dev(merge commitac5f697).Outcome (plain English): ACES now has a pre-run authoring surface for experiments. Previously the experiment-core contracts were archival outputs only — you could record what a run produced, but not specify an experiment (seeds, turn order/step count/termination, red-variant selection, condition allocation) before running it.
experiment-authoring-input-v1closes that gap with an agent/MCP-first authoring surface (scaffold → validate → get-example), a published schema, worked examples, and a discovery test — the experiment analogue of SDL scenario authoring.Requirement: EXP-736 — Experiment Authoring Input Specification (ACTIVE, wave 2).
Traceability reconciled: EXP-736 carries IMPLEMENTS links to the changed contract/loader/MCP source + the published schema, TESTS links to the four changed test modules, and DOCUMENTS links to ADR-074 and the experiment-core formal spec. Verified locally via
check_requirement_governance.py(exit 0). (Note:gc_assert_traceability_reconciled/gc_post_final_reportcould not post their markers on this multi-project backend —gc_assert_traceability_reconciledreturns "Multiple projects exist. Specify a 'project' parameter" and exposes noprojectparam — so this is the durable reconciliation record in lieu of the tool-posted marker.)Gates at merge: CI green (full
nox verify); SonarCloud quality gate OK (new coverage 96.8%, new violations 0, ratings A); pre-push reviews codex (1 finding fixed) + test-quality (clean).Decision record: ADR-074 (accepted); ADR-055 and ADR-069 amended.
Follow-up: #712 — add a detection that every published contract traces to a governing requirement.