Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
841 changes: 841 additions & 0 deletions docs/design/var1-02-set-property-action-20260726.json

Large diffs are not rendered by default.

160 changes: 160 additions & 0 deletions docs/design/var1-02-set-property-action-20260726.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# VAR1-02 (SLM-425): SET_PROPERTY action -- production reachability delta

- generated_at: `2026-07-26T19:54:14Z`
- gated on: `docs/design/var1-01-set-property-probe-20260725.md` (SLM-424/VAR1-01)
- hypothetical: `false` -- this is the REAL, production action space, checkpoint
format, and analyzer, not a what-if probe.
- mode: `extended`, max_edits: 8, node_budget: 15 (see "Why node_budget=15" below)
- seed: `root = Stack([], "column")`

> Reachability is space coverage, not model quality: no quality claim follows
> from these proofs alone. No training, evaluation, or promotion ran as part
> of this issue.

## What changed

`ACTION_SET_PROPERTY` (index 11; `N_ACTIONS` 11 -> 12) rebinds an existing
container's declared `rest` (its enum/direction argument) to another value of
the SAME pack-declared domain (`DslPack.component_property_domains`) -- root
included, since (unlike `REMOVE_CONTAINER`) it never removes or re-mints a
node. `CHECKPOINT_FORMAT` bumped 2 -> 3; a format-2 checkpoint fails closed on
load pointing at `checkpoint_migrate.migrate_tree_edit_checkpoint`, which
already upgrades any `format_version < CHECKPOINT_FORMAT` generically (no new
function needed -- see the code changes list below). The tree-edit variant's
`VariantContractV1.kernel_ops` now declares `openui.set_property`. The SLM-299
analyzer's `_check_invariants` now recognizes the real action (pack-domain
bounded, forced on in extended mode) separately from VAR1-01's hypothetical
`set_property_action()` (which may use a wider what-if domain and still
defers to BFS exactly as before) -- every VAR1-01 probe test continues to pass
unchanged.

## Reachable fraction by suite: four points of comparison

| suite | SLM-305 baseline (node_budget=120, older corpora) | VAR1-01 Arm A (before, node_budget=15, current corpora) | VAR1-01 Arm B (`set_property_action()` what-if, node_budget=15) | VAR1-02 production (this run, node_budget=15) | gate: production <= Arm B |
| --- | --- | --- | --- | --- | --- |
| train | 0.0 | corpus_unavailable | corpus_unavailable | corpus_unavailable | n/a |
| smoke | 0.0 | 0.0 | 0.0 | 0.0 | ok (0.0 <= 0.0) |
| held_out | 0.0 | 0.0 | 0.0 | 0.0 | ok (0.0 <= 0.0) |
| adversarial | 0.0 | 0.0 | 0.5 | 0.333333 | ok (0.333333 <= 0.5) |
| ood | 0.0 | 0.0 | 0.0 | 0.0 | ok (0.0 <= 0.0) |
| rico | 0.0 | 0.0 | no_decided_cases | 0.0 | ok (Arm B had no decided cases to exceed) |

The SLM-305 baseline (`docs/design/iter-slm305-edit-language-20260724.md`) is
**not** a same-budget, same-corpus comparison: it ran at `node_budget=120`
against older corpora (rico had 6 records, not 35; adversarial had no
`adv_deep_nest_01` UNKNOWN_BUDGET case; `train`'s corpus existed as a
committed fixture, now a gitignored generated artifact) -- see VAR1-01's own
note on this. VAR1-01's **Arm A_baseline** reproduces the pre-VAR1-02
production action set at `node_budget=15` against the exact same corpora this
run used, so it is the honest apples-to-apples "before"; **Arm B** is the
probe's what-if prediction this issue is gated on.

### Why node_budget=15, not the issue's suggested 120

A single-arm run at `node_budget=120` was attempted first and did not finish
inside a 170s wall-clock cap (`MAX_RUN_MINUTES=3`); a timed-out run is never
evidence per AGENTS.md. `node_budget=15` (VAR1-01's own precedent) completed
in 41.6s. This is a materially weaker probe on the `rico` suite in particular
(see the discrepancy note below); a longer, separately-run job at a larger
budget outside this session's cap would be needed to fully resolve it.

## Per-suite verdict flips vs VAR1-01 Arm A (the same-corpus "before")

- **smoke / held_out / ood**: no flips (0 each) -- these suites' gaps are all
`unsupported_component` (component-inventory gap, out of this issue's
scope; VAR0-03/SLM-426 already closed the analogous inventory gap
elsewhere).
- **adversarial**: 1 flip.
- `adv_empty_prompt_01`: `PROVEN_UNREACHABLE` (`needs_direction_change`) ->
`PROVEN_REACHABLE` (`reached`, `edit_lower_bound=2`,
`path=[ADD, SET_PROPERTY]`) -- **CONFIRMED**, and matches VAR1-01 Arm B's
confirmed flip on this exact case, same path shape, same 2-edit bound.
- `adv_many_buttons_01` stays `PROVEN_UNREACHABLE` (`needs_direction_change`)
-- its target needs a `"row"` rest, which the pack does not declare in
`component_property_domains` (only `', "column"'` and `""`), so
`ACTION_SET_PROPERTY` genuinely cannot produce it. Production's
pack-domain-bounded invariant proves this case decisively; VAR1-01 Arm B
(unconditional invariant skip, unbounded what-if domain) left it as
`UNKNOWN_BUDGET` instead. **More decisive, not less correct.**
- `adv_deep_nest_01` stays `UNKNOWN_BUDGET` in both (budget-bound,
unaffected by this change).
- **rico**: 34 flips, all `PROVEN_UNREACHABLE` (`needs_direction_change`) ->
`UNKNOWN_BUDGET` (`budget`). See the discrepancy note below -- this is an
honest, disclosed side effect, not a reachability regression.

## Honest-verdict

- **Claim class: `capability`** (space coverage only). Not claimed: ship
readiness, promotion of any checkpoint, a champion-model change, or that
improved reachability implies improved output quality. No training,
evaluation, or promotion ran as part of this issue.
- **What is confirmed:** `adv_empty_prompt_01` flips to `PROVEN_REACHABLE`,
reproducing VAR1-01 Arm B's confirmed positive exactly (same case, same
2-edit path shape: `ADD` then `SET_PROPERTY`).
- **Gate check (issue requirement: production <= Arm B per suite, else red
flag):** every suite lands at or below its Arm B counterpart (see the table
above) -- **no red flag.** `adversarial`'s 0.333333 <= 0.5 is the only
suite with a numeric Arm B value below 1.0; the apparent "regression" from
0.5 to 0.333333 is explained by production **deciding** one more case
(`adv_many_buttons_01`, proven genuinely unreachable) than Arm B's cruder
unconditional-skip invariant could -- a strictly more decisive result, not
a worse one.
- **Discrepancy disclosed honestly (not picked for looking favorable):**
`rico`'s reachable_fraction did **not** improve (stays `0.0`), and its
decided-case count collapsed: VAR1-01 Arm A decided all 35/35 cases
(`PROVEN_UNREACHABLE`, `needs_direction_change`); this production run
decides only 1/35 (`rico_eval_test_56`, whose target needs a `"row"` rest
outside the pack's domain -- still decisively unreachable), and the other
34 become `UNKNOWN_BUDGET`. This is **not** a reachability regression: most
`rico` targets omit a direction argument entirely (`root =
Stack([...])`, rest `""`), which the pack's own `container_rests` domain
already permits -- so the OLD hard "root rest must equal seed's" check was
never actually sound once a real rest-mutation action exists; it was
masking the true (deeper, structural) reachability question behind a
cheap-but-now-recognized-as-overly-strict proof. Confirming or refuting the
REST of each transformation (typically several `ADD_CONTAINER`/
`INSERT_SUBTREE`/`BIND_PLACEHOLDER` steps to rebuild multiple
`Card`+leaf subtrees) needs a deeper search than `node_budget=15,
max_edits=8` can decide. Per AGENTS.md I14, `UNKNOWN_BUDGET` is
inconclusive and never evidence of unreachability; it is reported
separately here, not folded into a falsely-precise `0.0`. A longer,
separately-run job at a larger `node_budget` (outside this session's
`MAX_RUN_MINUTES=3` cap) is required to resolve those 34 cases either way.

## Files changed

- `src/slm_training/models/tree_edit_diffusion.py` -- `ACTION_SET_PROPERTY`,
`N_ACTIONS=12`, `CHECKPOINT_FORMAT=3`, `EditDomain.component_property_domains`
/ `.property_names`, `TreeEditSpace.apply` branch (root-inclusive, real
parser re-validated), `sample_mutation` inverse-edit generation,
`_enumerate_edits` decode-time scoring.
- `src/slm_training/models/checkpoint_migrate.py` -- docstring only;
`migrate_tree_edit_checkpoint` already upgrades any
`source_format < CHECKPOINT_FORMAT` generically (shape-driven, not
hardcoded to format 1), so format 2 -> 3 needed no new code path.
- `src/slm_training/harnesses/experiments/slm299_edit_reachability.py` --
`_enumerate_children` gains the real `ACTION_SET_PROPERTY` transitions;
`_check_invariants` distinguishes the real (pack-domain-bounded) capability
from VAR1-01's hypothetical (potentially wider-domain) one via
`hypothetical_set_property_domain`; `analyze_reachability` forces
`set_property` on in extended mode alongside the existing `container_add`
forcing.
- `src/slm_training/dsl/variants.py` -- `tree_edit_diffusion` variant's
`kernel_ops` now includes `openui.set_property`; regenerated
`src/slm_training/resources/variant_registry.json`.
- `tests/test_models/test_tree_edit_diffusion.py` -- new tests: apply/inverse
round-trip on root and a non-root container, illegal-value rejection via
the real parser (not just an index bounds-check), format-2 fail-closed
load, and format-2->3 migration with bit-identical logits on the
pre-existing 11 action rows.
- `src/slm_training/resources/versions.json` -- bumped
`harness.experiments.slm299_edit_reachability` (v5->v6), `dsl.variants`
(v2->v3), `model.twotower` (v255->v256, watches
`checkpoint_migrate.py`).
- This doc pair.

## Out of scope (per issue)

- Training, evaluating, promoting, or syncing any checkpoint.
- Widening the component inventory (VAR0-03/SLM-426, already done).
- Any claim that improved reachability implies improved output quality.
3 changes: 2 additions & 1 deletion src/slm_training/dsl/variants.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,8 @@ def build_variant_contracts() -> tuple[VariantContractV1, ...]:
pack_id="openui",
action_alphabet_id="tree_edit_diffusion.edit_actions",
action_alphabet_fingerprint=_alphabet_fingerprint(_tree_edit_action_names()),
kernel_ops=(),
# SLM-425 (VAR1-02): ACTION_SET_PROPERTY realizes this kernel op.
kernel_ops=("openui.set_property",),
seed_policy_id="tree_edit_diffusion.minimal_valid_program_seed",
inventory_source="pack",
source_path=_TREE_EDIT_DIFFUSION_SOURCE,
Expand Down
100 changes: 77 additions & 23 deletions src/slm_training/harnesses/experiments/slm299_edit_reachability.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
ACTION_REPLACE,
ACTION_REPLACE_STATEMENT,
ACTION_REPLACE_SUBTREE,
ACTION_SET_PROPERTY,
MAX_SLOTS,
Edit,
Statement,
Expand Down Expand Up @@ -179,10 +180,14 @@ def set_property_action(
) -> ExtraAction:
"""VAR1-01 hypothetical: rebind an existing container's ``rest`` (its
enum/direction argument) in place, without minting or removing any
statement. No real action in the deployed edit space edits ``rest`` on an
existing node — see ``_check_invariants``'s ``needs_direction_change``
reason. This is a *what-if* probe of a property-mutation action class,
never a production action; see VAR1-01 (SLM-424).
statement, drawing from ``rests`` — which may be WIDER than the real
``TreeEditSpace.container_rests`` domain (the default includes ``"row"``,
which the pack does not declare). SLM-425 (VAR1-02) added a REAL
``ACTION_SET_PROPERTY`` gated on this probe's confirmed result, but its
domain is exactly the pack-declared ``container_rests`` — narrower than
this what-if's default. This factory (and the capability it grants) stays
useful for exploring domains wider than the pack currently declares; see
VAR1-01 (SLM-424) and VAR1-02 (SLM-425).
"""

def _generate(
Expand Down Expand Up @@ -375,6 +380,7 @@ def _check_invariants(
capabilities: frozenset[str] = frozenset(),
*,
extended: bool = False,
hypothetical_set_property_domain: bool = False,
) -> str | None:
"""Structural impossibility proofs over the EXACT action set.

Expand All @@ -386,7 +392,17 @@ def _check_invariants(
leaves only to
inventory slots, and the V0.5 statement actions mint only canonical
templates — the invariant reasons fire only when the corresponding REAL
action of the analyzed mode is absent.
action of the analyzed mode is absent. SLM-425 (VAR1-02): the extended
space also has a REAL ``ACTION_SET_PROPERTY`` that rebinds any existing
container's (root included) ``rest`` to another value of
``space.container_rests`` — the pack-declared domain, narrower than a
hypothetical ``set_property_action()``'s (default, wider) ``rests``.
``hypothetical_set_property_domain`` distinguishes the two: it is True
only when the ``set_property`` capability comes from an *extra_action*
(VAR1-01 what-if, unbounded domain — the cheap proof below cannot rule
those out generically, so it is skipped entirely, as before); it is False
when the capability is the real, pack-domain-bounded action (forced on
in extended mode), where the cheap proof stays precise.
"""
seed_containers = [s for s in seed if s.has_list]
target_containers = [s for s in target if s.has_list]
Expand Down Expand Up @@ -431,28 +447,38 @@ def _check_invariants(
if normalized not in inventory:
return REASON_NEEDS_SLOT_REBIND

# REPLACE preserves the container's raw enum/direction arg text (rest);
# no real action edits it. Containers minted by the container-creating
# actions carry exactly one of CONTAINER_RESTS. Every target container
# rest must therefore be a seed rest or (with container_add) a candidate
# mint rest; without any container-creating action the multisets must
# match exactly.
if "set_property" in capabilities:
# A rest-mutation action can produce any target rest on any
# container, root included -- skip both rest checks below entirely.
# REPLACE preserves the container's raw enum/direction arg text (rest).
# Containers minted by the container-creating actions carry exactly one
# of CONTAINER_RESTS; SET_PROPERTY (SLM-425/VAR1-02) rewrites an
# existing container's rest to exactly one of CONTAINER_RESTS too, root
# included. Every target container rest must therefore be a seed rest or
# a candidate mint/set rest; without either capability the multisets
# must match exactly.
if "set_property" in capabilities and hypothetical_set_property_domain:
# VAR1-01 what-if only: an ExtraAction may draw from ANY domain (its
# own ``rests``, unbounded by the pack), so no cheap proof is
# derivable generically here -- defer entirely to BFS (legacy
# behavior, unchanged since VAR1-01).
return None

seed_rests = sorted(s.rest for s in seed_containers)
target_rests = sorted(s.rest for s in target_containers)
# Root can never be removed or re-minted, and REPLACE preserves rest:
# the target root's rest must equal the seed root's rest in every mode.
seed_root_rest = next(
(s.rest for s in seed_containers if s.name == "root"), None
)
for stmt in target_containers:
if stmt.name == "root" and stmt.rest != seed_root_rest:
return REASON_NEEDS_DIRECTION_CHANGE
if "container_add" in capabilities:
if "set_property" not in capabilities:
# No action edits an existing container's rest in place: the root
# can never be removed or re-minted, and REPLACE preserves rest, so
# the target root's rest must equal the seed root's rest.
for stmt in target_containers:
if stmt.name == "root" and stmt.rest != seed_root_rest:
return REASON_NEEDS_DIRECTION_CHANGE
if "container_add" in capabilities or "set_property" in capabilities:
# Either a fresh container can be minted with any of
# space.container_rests, or an existing one's rest (root included)
# can be rewritten to any of space.container_rests -- so every
# target rest must be drawn from the seed's own rests or that real
# domain.
allowed = set(seed_rests) | set(space.container_rests)
if any(rest not in allowed for rest in target_rests):
return REASON_NEEDS_DIRECTION_CHANGE
Expand Down Expand Up @@ -599,6 +625,23 @@ def _replace_subtree_ok(stmt: Statement) -> bool:
},
)
)
# SLM-425 (VAR1-02): rebind this container's rest in place (root
# included) to another value of the same pack-declared domain --
# the property-mutation action class VAR1-01 proved missing.
for rest_idx in range(len(space.container_rests)):
edit = Edit(ACTION_SET_PROPERTY, stmt_idx, target=rest_idx)
nxt = space.apply(statements, edit, inventory, pre)
if nxt is not None:
children.append(
(
nxt,
{
"action": "SET_PROPERTY",
"stmt": stmt_idx,
"rest": space.container_rests[rest_idx],
},
)
)
else:
for slot_idx in range(n_slots):
edit = Edit(ACTION_BIND_PLACEHOLDER, stmt_idx, slot=slot_idx)
Expand Down Expand Up @@ -714,16 +757,27 @@ def analyze_reachability(
details=details,
)

capabilities = frozenset().union(
extra_capabilities = frozenset().union(
*(a.capabilities for a in extra_actions)
) if extra_actions else frozenset()
capabilities = extra_capabilities
if mode == "extended":
# ADD_CONTAINER / INSERT_SUBTREE are REAL actions in the extended
# space, so the container_add invariant must not fire; the synthetic
# add_container_action what-if lane is retired for this space.
capabilities = capabilities | frozenset({"container_add"})
# SET_PROPERTY (SLM-425/VAR1-02) is likewise REAL in the extended
# space (pack-domain-bounded); see ``_check_invariants``'s
# ``hypothetical_set_property_domain`` for how that differs from an
# extra_action's (potentially wider) what-if domain.
capabilities = capabilities | frozenset({"container_add", "set_property"})
fired = _check_invariants(
target, seed, inventory, space, capabilities, extended=(mode == "extended")
target,
seed,
inventory,
space,
capabilities,
extended=(mode == "extended"),
hypothetical_set_property_domain="set_property" in extra_capabilities,
)
if fired is not None:
return ReachabilityCase(
Expand Down
19 changes: 12 additions & 7 deletions src/slm_training/models/checkpoint_migrate.py
Original file line number Diff line number Diff line change
Expand Up @@ -301,13 +301,18 @@ def migrate_tree_edit_checkpoint(
output_checkpoint: Path | str,
device: str = "cpu",
) -> dict:
"""Warm-start a tree-edit-diffusion format-1 checkpoint to format 2.

SLM-305 grew ``action_head`` from 4 to ``N_ACTIONS`` rows (extended edit
language). All same-shape tensors are copied verbatim; the old action-head
rows are preserved in the first rows of the new head and the new action
rows stay randomly initialized. A ``.migrate.json`` report is written next
to the output checkpoint.
"""Warm-start an older tree-edit-diffusion checkpoint to the current format.

Format-agnostic: it upgrades any checkpoint whose
``format_version < TreeEditDiffusionModel.CHECKPOINT_FORMAT`` (format 1
or format 2 today) to the live format by shape, not by a hardcoded source
version. SLM-305 grew ``action_head`` from 4 to 11 rows (extended edit
language, format 1 -> 2); SLM-425 (VAR1-02) grew it again from 11 to 12
rows for ``ACTION_SET_PROPERTY`` (format 2 -> 3). All same-shape tensors
are copied verbatim; the old action-head rows are preserved in the first
rows of the new head (bit-identical logits on those rows) and the new
action row(s) stay randomly initialized. A ``.migrate.json`` report is
written next to the output checkpoint.
"""
from slm_training.models.tree_edit_diffusion import (
TreeEditDiffusionConfig,
Expand Down
Loading
Loading