Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/explain/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ RAES can currently support:
- parsing and validating SDL through the Python implementation
- instantiating variables and compiling runtime models in the reference stack
- checking backend contract fixtures and conformance profiles
- browsing validated, non-normative examples, templates, and reusable patterns
- browsing non-normative examples, templates, and reusable patterns
for scenarios, workflows, participant behavior, tasks, runs, and studies
- reviewing the specifications, ADRs, and examples that define current claims

Expand All @@ -60,7 +60,7 @@ fixtures or schema authority.
| Understand the repository | `README.md`, [`docs/index.md`](../index.md), [`docs/explain/reference/canonical-reference-map.md`](reference/canonical-reference-map.md) | Read the referenced docs | The repository layout and current boundaries are understood. |
| Understand the agentic-environment lifecycle | [`docs/explain/reference/glossary.md`](reference/glossary.md), [`docs/explain/sdl/runtime-architecture.md`](sdl/runtime-architecture.md) | Follow the authored scenario, realized environment, evidence, and conformance references | RAES system concepts are distinguished from SDL and backend behavior. |
| Read a complete scenario | `examples/README.md`, `examples/scenarios/*.sdl.yaml` | `pytest tests/test_example_schema_conformance.py` | The checked examples load through the current parser boundary. |
| Start from a reusable template or pattern | `examples/library/catalog.yaml`, `examples/library/templates/`, `examples/library/patterns/` | `python tools/check_example_library.py` | The catalog covers scenario, workflow, participant behavior, task, run, and study surfaces with parser-validated template bodies. |
| Start from a reusable template or pattern | `examples/library/catalog.yaml`, `examples/library/templates/`, `examples/library/patterns/` | `python tools/check_example_library.py` | The catalog covers scenario, workflow, participant behavior, task, run, and study surfaces, with template bodies validated as SDL or as `experiment-authoring-input-v1` documents. |
| Author a small SDL file | [`docs/explain/sdl/index.md`](sdl/index.md), [`docs/explain/sdl/sections.md`](sdl/sections.md), [`docs/explain/sdl/validation.md`](sdl/validation.md) | `parse_sdl_file()` or `load_scenario()` | The file is accepted by the current SDL model and semantic validator. |
| Use an agent-facing authoring surface | `raes-mcp`, then `raes_tool_surface`, `raes_agent_guidance`, `raes_intended_use_profiles`, and [`docs/explain/sdl/language-service.md`](sdl/language-service.md) | `raes_agent_guidance`, `raes_intended_use_profiles`, `sdl_completions`, `sdl_apply_edit`, `sdl_diagnostics`, `sdl_format`, `sdl_references`, `sdl_validate`, `sdl_design_assessment`, `sdl_plan`, `sdl_claims_assessment` | The agent can choose an intended-use scope, inspect current RAES blockers, and help author, edit, dry-run, and qualify claims without repository-local code access. |
| Use variables or imports | [`docs/explain/sdl/parser.md`](sdl/parser.md), [`docs/explain/sdl/sections.md`](sdl/sections.md) | `raes sdl resolve`, `raes sdl verify-imports` | Imports and variable placeholders follow the current parser rules. |
Expand All @@ -77,7 +77,7 @@ Use the lowest level that answers the question.
| Orientation | You need to know what RAES is and is not. | README, docs index, reference map | Current repository scope and entrypoints | SDL validity, backend behavior, or experiment adequacy |
| SDL parse and validation | You have an SDL file and need current parser feedback. | `parse_sdl_file()`, `load_scenario()`, SDL parser/model/validator tests | Structural and semantic acceptance by the reference implementation | Deployment viability or general domain completeness |
| Example-backed authoring | You need a worked scenario to study or adapt. | `examples/scenarios/*.sdl.yaml`, `test_example_schema_conformance.py` | The example loads from disk without advisories under current tests | Suitability for another range, backend, exercise, or research design |
| Template and pattern authoring | You need a reusable starting shape for a scenario, workflow, participant behavior, task, run, or study. | `examples/library/catalog.yaml`, `tools/check_example_library.py` | The cataloged template body validates as current SDL and the pattern has stable metadata | New runtime semantics or first-class task, run, or study sections |
| Template and pattern authoring | You need a reusable starting shape for a scenario, workflow, participant behavior, task, run, or study. | `examples/library/catalog.yaml`, `tools/check_example_library.py` | The cataloged template body validates as current SDL or as an `experiment-authoring-input-v1` document, and the pattern has stable metadata | New runtime semantics or first-class task, run, or study sections |
| Runtime and contracts | You need processor or backend integration context. | Runtime compiler/planner, contract schemas, backend profiles, conformance fixtures | Current reference-stack and contract behavior | Production backend correctness or operational reliability |
| Specification review | You need to evaluate a semantic or authority claim. | `specs/`, ADRs, formal notes, tests | The current reasoning and normative boundary for a claim | Completed implementation when the materialized code/contracts are absent |

Expand Down Expand Up @@ -219,7 +219,7 @@ Do not use examples to claim:

## Template And Pattern Boundary

The current library provides validated authoring aids, not new runtime
The current library provides authoring aids, not new runtime
authority.

Current status:
Expand Down
2 changes: 1 addition & 1 deletion docs/explain/reference/canonical-reference-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ page is an index, not a replacement for the linked artifacts.
## Current Materialization Notes

- SDL authoring, parsing, validation, instantiation, compilation, planning,
runtime manager/control-plane APIs, published JSON schemas, and a validated
runtime manager/control-plane APIs, published JSON schemas, and a
non-normative template/pattern library are present in the repository.
- Participant-implementation manifests, evidence-capture contracts, and
provenance contract surfaces are described at the architecture level and are
Expand Down
147 changes: 130 additions & 17 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,19 +73,123 @@ The AUT-806 library is indexed by
[`library/catalog.yaml`](library/catalog.yaml). It is a versioned,
machine-readable catalog for the current non-normative authoring library.

| Surface | Template | Pattern |
|---------|----------|---------|
| Scenario | [`library/templates/scenario/minimal-validated-scenario.yaml`](library/templates/scenario/minimal-validated-scenario.yaml) | [`library/patterns/scenario-reference-integrity.yaml`](library/patterns/scenario-reference-integrity.yaml) |
| Surface | Templates | Pattern |
|---------|-----------|---------|
| Scenario | [`library/templates/scenario/minimal-validated-scenario.yaml`](library/templates/scenario/minimal-validated-scenario.yaml), [`library/templates/scenario/segmented-network-scenario.yaml`](library/templates/scenario/segmented-network-scenario.yaml), [`library/templates/scenario/parameterized-scenario.yaml`](library/templates/scenario/parameterized-scenario.yaml) | [`library/patterns/scenario-reference-integrity.yaml`](library/patterns/scenario-reference-integrity.yaml) |
| Workflow | [`library/templates/workflow/parallel-objective-workflow.yaml`](library/templates/workflow/parallel-objective-workflow.yaml) | [`library/patterns/workflow-explicit-control-graph.yaml`](library/patterns/workflow-explicit-control-graph.yaml) |
| Participant behavior | [`library/templates/participant_behavior/action-contract-observation-boundary.yaml`](library/templates/participant_behavior/action-contract-observation-boundary.yaml) | [`library/patterns/participant-behavior-contract-binding.yaml`](library/patterns/participant-behavior-contract-binding.yaml) |
| Task | [`library/templates/task/single-objective-task.yaml`](library/templates/task/single-objective-task.yaml) | [`library/patterns/task-as-objective-contract.yaml`](library/patterns/task-as-objective-contract.yaml) |
| Run | [`library/templates/run/timed-run-control.yaml`](library/templates/run/timed-run-control.yaml) | [`library/patterns/run-window-with-evidence.yaml`](library/patterns/run-window-with-evidence.yaml) |
| Study | [`library/templates/study/observational-study-protocol.yaml`](library/templates/study/observational-study-protocol.yaml) | [`library/patterns/observable-study-conditions.yaml`](library/patterns/observable-study-conditions.yaml) |
| Run | [`library/templates/run/timed-run-control.yaml`](library/templates/run/timed-run-control.yaml), [`library/templates/run/seeded-run-plan.yaml`](library/templates/run/seeded-run-plan.yaml) | [`library/patterns/run-window-with-evidence.yaml`](library/patterns/run-window-with-evidence.yaml) |
| Study | [`library/templates/study/observational-study-protocol.yaml`](library/templates/study/observational-study-protocol.yaml), [`library/templates/study/two-condition-study-design.yaml`](library/templates/study/two-condition-study-design.yaml) | [`library/patterns/observable-study-conditions.yaml`](library/patterns/observable-study-conditions.yaml) |

Each template has metadata plus a complete current-SDL `body`. The
Each template has metadata, a `validation` list of commands with their
expected results, and a complete `body`. Most bodies are current SDL; the
`seeded-run-plan` and `two-condition-study-design` bodies are
`experiment-authoring-input-v1` documents. The
`tools/check_example_library.py` policy gate validates catalog shape, stable
IDs, referenced paths, AUT-806 requirement references, and every template body
through the SDL parser and semantic validator.
IDs, referenced paths, AUT-806 requirement references, and entry metadata. It
checks every SDL template body and validated worked example with the SDL
parser and semantic validator, and every experiment template body with the
experiment authoring-input loader. It also checks that each template has a
non-empty `validation` list whose entries have non-empty `command` and
`expected` strings. The gate does not run those commands.

### Catalog entry metadata

Every `worked_examples`, `templates`, and `patterns` entry in the catalog
records `validation_status`, `intended_user`, and `limits`. SDL entries also
record `sdl_sections`, and a template whose body is not SDL records
`contract`. The policy gate rejects an entry that omits a field or breaks
these rules.

| Field | Allowed values | What the gate checks |
|-------|----------------|----------------------|
| `contract` | `sdl-yaml/v1`, the default when absent, or `experiment-authoring-input-v1` for a run or study template | The value is allowed for the entry. It selects how a `validated` body is checked and which `intended_user` the entry names. |
| `validation_status` | `validated` or `guidance` | For `validated`, the gate checks the template body or worked-example file against its contract. SDL goes through the SDL parser and semantic validator, on the worked-example file or on the template body as the `sdl-yaml/v1` loader reads it from the template file, and fails on any error or advisory they report. That loader rejects duplicate keys, merge keys, and explicit tags in the template file. `experiment-authoring-input-v1` goes through the experiment authoring-input loader, on the template body as that loader's YAML reader reads it from the template file, and fails on any error it reports. That reader rejects duplicate keys, aliases, and `<<` merge keys in the template file. A `guidance` file is not checked against a contract. Templates must be `validated`; patterns must be `guidance`. |
| `sdl_sections` | Top-level section names from [`../specs/sdl/sections.md`](../specs/sdl/sections.md), such as `nodes` or `workflows` | Present for SDL entries and absent otherwise. Each name is a current SDL section, not a metadata or composition field such as `name` or `imports`. For a `validated` entry, each listed section is present in the validated SDL. |
| `intended_user` | `sdl-author` for SDL entries, `experiment-author` for `experiment-authoring-input-v1` templates | The value matches the entry's contract. |
| `limits` | One or more short statements of what the entry does not show | The list is present and not empty. Reviewers check the wording. |

`sdl_sections` names the sections that an entry shows for its surface, so a
large worked example lists only some of the sections it uses. `validated`
means that the contract's validator accepts the file or template body. It does
not mean that a backend can realize the scenario, that anything has run, or
that references to other artifacts resolve. A `guidance` entry is reading
material, such as a prose pattern or a test module, and the gate does not
check it against a contract.

### Scenario templates

Each scenario template claims only that current SDL validation accepts its
body with no advisories.

| Template | Shows | Does not claim |
|----------|-------|----------------|
| [`minimal-validated-scenario`](library/templates/scenario/minimal-validated-scenario.yaml) | One host service, one participant, an objective whose success is an assertion over an observed proposition, and a workflow with one objective step | That a backend can realize the scenario |
| [`segmented-network-scenario`](library/templates/scenario/segmented-network-scenario.yaml) | Two switched segments, a linked host with a fixed address on each, access rules that admit only HTTPS from the user segment into the server segment, and a service feature | That a backend enforces the access rules or installs the feature |
| [`parameterized-scenario`](library/templates/scenario/parameterized-scenario.yaml) | Host operating system, size, instance count, and service port taken from declared variables with defaults and an allowed-values list | That the scenario instantiates with every supplied value; the gate does not instantiate it |

To check a template as shipped, run the catalog command in
[Validate The Examples](#validate-the-examples). To check an adapted copy, save
its `body` as `my-scenario.sdl.yaml` in the repository root and run this
command from there:

```shell
uv run --project implementations/python --frozen raes semantic validate my-scenario.sdl.yaml
```

The command prints `validate: success` and exits `0` when the SDL parser and
semantic validator accept the file. It does not report semantic-validator
advisories, such as a compute node without `resources`. The `parse_sdl_file`
check in [Validate The Examples](#validate-the-examples) asserts that there
are none.

### Task, run, and study entries

The task, run, and study surfaces keep these parts separate:

- **SDL:** the `single-objective-task`, `timed-run-control`, and
`observational-study-protocol` templates and the worked examples. SDL has no
`tasks`, `runs`, or `studies` sections, so these use objectives, workflows,
timing, and observable conditions.
- **Experiment authoring input:** the `seeded-run-plan` and
`two-condition-study-design` templates. Each body is an
`experiment-authoring-input-v1` document, the pre-run design that binds a
task to a run plan
([ADR-074](../docs/decisions/adrs/adr-074-experiment-authoring-input-contract-boundary.md)).
`seeded-run-plan` declares the run count, seed, and episode controls for one
task. `two-condition-study-design` declares a treatment factor, two compared
conditions, and the runs per condition.
- **Guidance:** the task, run, and study patterns, which are prose.
- **Outside current support:**
- An experiment task template. An experiment task is an
`experiment-task-v1` document that a run plan names in `task_ref`. The
repository has no task loader, `raes semantic validate` does not accept
the `experiment-task-v1` contract, and every task needs an artifact
reference with a checksum, size, and creation time that a reusable
template would have to invent.
- Resolving `task_ref`, and checking condition factor levels that no
binding descriptor or stratified selection stratum joins. The experiment
authoring-input loader does neither. It does check joined levels against
the declared factors and the condition assignment: binding descriptors
under `binding_semantics: explicit-required`
([ADR-094](../docs/decisions/adrs/adr-094-authoritative-cross-plane-experiment-bindings.md))
and stratified selection strata. The shipped `two-condition-study-design`
body joins none, so the loader does not check its levels.
- Admitting, scheduling, or running a plan, and the `experiment-run-v1` and
`experiment-study-v1` records that describe actual executions and
analyses.

To check an adapted copy of an experiment template, save its `body` as
`my-experiment.exp.yaml` in the repository root and run this command from
there:

```shell
uv run --project implementations/python --frozen python -c "import sys; from pathlib import Path; from raes_contracts.experiment_spec import load_experiment_spec; print(load_experiment_spec(Path(sys.argv[1])).spec_id)" my-experiment.exp.yaml
```

The command prints the document's `spec_id` and exits `0`. An invalid
document raises `ExperimentSpecValidationError` and exits `1`.

## Validate The Examples

Expand Down Expand Up @@ -137,19 +241,28 @@ Check the section reference and limitations before adapting a file:

## Template Boundary

There are no placeholder templates in this directory. Files under
No template in this directory contains incomplete SDL. Files under
`examples/scenarios/` are positive SDL examples and must load successfully from
disk. Files under `examples/library/templates/` are reusable authoring
templates with complete SDL bodies; they are validated by the example-library
policy gate.
templates with complete SDL or `experiment-authoring-input-v1` bodies; they are
validated by the example-library policy gate. An SDL template body may use
`${name}` variable placeholders, as `parameterized-scenario` does. A
placeholder's `name` follows the variable-name grammar in
[`../specs/sdl/variables-and-instantiation.md`](../specs/sdl/variables-and-instantiation.md)
(lowercase letters, digits, `-`, and `_`), and the gate rejects a body whose
placeholder names a variable that the body does not declare under `variables`.
Text such as `${Web OS}` or `${CUSTOMER_NAME}` is not a placeholder: it stays
literal text, and the gate does not report it as an undeclared variable.

Do not add invalid or incomplete SDL files under `examples/scenarios/`.
Negative-path examples belong in focused parser, model, validator, contract, or
conformance tests.

Task, run, and study templates use current SDL wrappers rather than first-class
`tasks`, `runs`, or `studies` sections. They show how to express those concepts
with objectives, workflows, timing, observable conditions, and evidence-like
references that the current implementation can validate. Graded scoring and
reward are experiment/evaluator-plane concerns (experiment-* contracts) per
ADR-073, not SDL surfaces.
The task, run, and study SDL templates use current SDL wrappers rather than
first-class `tasks`, `runs`, or `studies` sections. They show how to express
those concepts with objectives, workflows, timing, observable conditions, and
evidence-like references that the current implementation can validate. Graded
scoring and reward are experiment/evaluator-plane concerns (experiment-*
contracts) per ADR-073, not SDL surfaces. The run and study experiment
templates are pre-run designs; see
[Task, run, and study entries](#task-run-and-study-entries).
Loading
Loading