SLM-418: convert replay preference rows to PreferencePair (DSH5-10)

docs/design/dsh5-10-replay-preference-rows.md

@@ -1,6 +1,6 @@
 # DSH5-10: replay-grounded preference rows from undo/redo history (SLM-418)
 
-**Status:** partial slice, in progress (fifth increment).
+**Status:** partial slice, in progress (sixth increment).
 **Claim class:** `wiring`.
 **Honest verdict:** not yet dispositioned -- this PR extends a scoped
 subset, not the full issue.
@@ -13,13 +13,18 @@ SFT/preference training against four context-view baselines, and (4)
 held-out measurement of action/operator/argument/reference/branch accuracy,
 calibration, and CAP0/CAP1/CAP2 retention.
 
-This slice adds the seventh and final named pattern, **pronoun-focus-
-followup** (see "Fifth slice (v6)" below), bringing extraction coverage to
-7 of 7. All seven named patterns from the issue's own list now extract and
-replay-verify. What remains is the issue's separate, still fully unattempted
-training/measurement scope: SFT/preference training against the
-DSH3-selected policy/control heads, the four-baseline comparison, held-out
-benefit measurement, and turn-depth/context-view ablations.
+The fifth slice added the seventh and final named pattern, **pronoun-focus-
+followup**, bringing extraction coverage to 7 of 7 -- all seven named
+patterns from the issue's own list now extract and replay-verify. This
+slice (see "Sixth slice (v7)" below) takes the smallest next step of the
+issue's still-unattempted scope: it converts an already-extracted row into
+this repo's *existing* preference-training example format
+(`PreferencePair`), so a future slice can feed rows into the existing
+`slm preference` harness without inventing a second corpus shape. It does
+**not** attempt SFT/preference training against the DSH3-selected
+policy/control heads, the four-baseline comparison, held-out benefit
+measurement, or turn-depth/context-view ablations -- all of that remains
+exactly as unattempted as before this slice.
 
 ## What this PR delivers
 
@@ -288,10 +293,95 @@ training/evaluation work enumerated above.
   snapshot is untouched, staying immutable point-in-time evidence from
   before this slice landed.
 
+## Sixth slice (v7)
+
+* Added `preference_pair_from_replay_row` and `preference_pairs_from_trace`
+  to `src/slm_training/dsl/operators/replay_preference.py` -- pure
+  data-conversion functions, not another extraction pattern. They take an
+  already-extracted `OperatorReplayPreferenceRowV1` and materialize it into
+  `PreferencePair` (`slm_training.harnesses.preference`), the exact
+  `prompt`/`chosen`/`rejected`/`design_md`/`chosen_score`/`rejected_score`/
+  `meta` schema the existing `slm preference build-pairs`/`train` harness
+  (`.agents/skills/autotrain/references/preference.md`,
+  `src/slm_training/harnesses/preference/train.py`) already reads and
+  writes -- no second corpus shape is invented.
+* **`prompt`** is `trace.node(row.input_state_id).state.source`: the exact,
+  already pack-authorized OpenUI source of the state the choice was made
+  from. This is the one piece of real context a row carries; a
+  `ConversationTraceV1`'s turns are AST operations, never user utterances,
+  so there is no natural-language instruction anywhere upstream to recover.
+  Using the state source as `prompt` is an honest, if unusual, reuse of the
+  field -- `pair.meta["schema"] == "operator_replay_preference_pair/v1"`
+  lets a caller tell these pairs apart from `build_pairs_from_candidates`
+  output, whose `prompt` is a design-task instruction. No instruction text
+  is fabricated.
+* **`chosen`/`rejected`** are `row.chosen_action`/`row.rejected_action`
+  verbatim -- the row's own legal-set action tokens (e.g. `"undo"`,
+  `"checkout:<state>"`, `"merge:<pair>"`, or a serialized `OPERATOR <id>
+  ..."` action). This slice deliberately does **not** attempt to replay the
+  rejected side to an alternate full OpenUI program: for an unchosen
+  operator action, no such program was ever materialized in the trace (only
+  the chosen action was actually applied), and reconstructing one would
+  require independently re-running `OperatorLibraryV1.apply` against a
+  legal-set candidate outside what extraction already computed -- real
+  additional work, out of scope for this slice's stated size. `chosen`/
+  `rejected` here are therefore two *action tokens* being preferred, not two
+  full-program renderings of one prompt the way `build_pairs_from_candidates`
+  pairs are; the doc says so explicitly rather than letting the shared field
+  names imply otherwise.
+* `composite_reward` (and its `grammar_score`/`placeholder_score`/
+  `layout_metrics` components) is deliberately **never called** on these
+  tokens. Scoring `"undo"` or a serialized `OPERATOR ...` action as though it
+  were OpenUI source would silently manufacture a meaningless number (e.g.
+  `grammar_score` would legitimately return `0.0` for non-program text, but
+  that `0.0` would look like a real quality signal to a downstream reader).
+  `chosen_score`/`rejected_score` are therefore left at `PreferencePair`'s
+  own `0.0` default, and the honest distinction lives in `meta` instead.
+* **Honest skip, not fabrication.** `preference_pairs_from_trace` looks up
+  each row's `input_state_id` via `trace.node(...)`; when it is not a member
+  of that trace's own `state_nodes` (the exact case for a `MERGE_SUCCESS`
+  row, which is grounded on a `BranchEditV1` tip rather than any single
+  trace -- see "Fourth slice (v5)" above), the row is skipped and recorded
+  in a separate `skipped` list (row + reason) rather than the function
+  fabricating a prompt or raising. `preference_pair_from_replay_row` also
+  refuses (`ValueError`) a blank/empty `input_state_source` rather than
+  emit a pair with a fabricated prompt.
+  `test_preference_pairs_from_trace_skips_a_row_whose_input_state_is_foreign_to_this_trace`
+  and
+  `test_preference_pair_from_replay_row_refuses_to_fabricate_a_blank_prompt`
+  (`tests/test_dsl/test_replay_preference.py`) prove both paths directly. A
+  `MERGE_SUCCESS` row itself still converts correctly --
+  `test_preference_pair_from_replay_row_converts_merge_success_row` calls
+  `preference_pair_from_replay_row` directly with
+  `left.output_node.state.source` supplied explicitly, exactly as the
+  function's docstring describes for that case.
+* Six more regression tests cover one row of each of the other six
+  extracted relation types converting correctly end to end through
+  `preference_pairs_from_trace`: `EDIT_THEN_UNDO`/`PARTIAL_ROLLBACK` (one
+  trace produces both), `UNDO_THEN_REDO`, `CHECKOUT_ANOTHER_STATE`,
+  `FORK_THEN_CHOOSE_ONE_BRANCH`, and `PRONOUN_FOCUS_FOLLOWUP` -- 7 of 7
+  named patterns are exercised by this slice's tests (six via the trace
+  path, one, `MERGE_SUCCESS`, via the direct path).
+* **Explicitly still not attempted:** no `PreferencePair` produced by this
+  slice has been written to a corpus file, fed to `slm preference
+  build-pairs`/`train`, or trained against; no composite-reward score is
+  computed for any row; no replay of a rejected operator action to an
+  alternate rendering is attempted; the DSH3-selected policy/control heads
+  (`TypedOperatorPolicyScorer`) and `ObjectiveView`/`DecisionStateV2`
+  materializers in `decision_events_v2.py` are untouched. This is
+  data-conversion wiring only, same honesty tier as every prior slice in
+  this doc.
+* `dsl.operators.replay_preference` bumped v6 -> v7 in
+  `src/slm_training/resources/versions.json`; `dsl.operators.contracts`
+  gets a `no-bump:` history entry for the two new re-exports from
+  `operators/__init__.py`.
+
 ## Reproducibility
 
 ```bash
 NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py
 ```
 
-Result (this PR, real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean.
+Result (fifth slice, v6): real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean.
+
+Result (this PR, sixth slice, v7): same command, same freshly built `.venv` (Python 3.12) plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` -- test file grew by 8 tests (7 conversion-coverage tests + 1 refusal test) in the same `tests/test_dsl/test_replay_preference.py`, no new test module needed: `69 passed`. Also verified: `ruff check` on every changed file (`src/slm_training/dsl/operators/replay_preference.py`, `src/slm_training/dsl/operators/__init__.py`, `tests/test_dsl/test_replay_preference.py`) -- `All checks passed!`; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (vs 5f94b925a121; 4 changed file(s), 2 component(s) touched)`; `python -m scripts.repo_policy` -- `repo-policy: ok (tracked + untracked)`; `python -m scripts.verify_decode_invariants` -- exits clean (`0`).

src/slm_training/dsl/operators/__init__.py

@@ -165,6 +165,8 @@
     ReplayPreferenceRelation,
     extract_merge_preference_row,
     extract_replay_preference_rows,
+    preference_pair_from_replay_row,
+    preference_pairs_from_trace,
 )
 from slm_training.dsl.operators.topology import (
     CONTRACT_SUBTREE,
@@ -457,6 +459,8 @@
     "prepare_operator_action",
     "prepare_operator_transaction",
     "preference_cost",
+    "preference_pair_from_replay_row",
+    "preference_pairs_from_trace",
     "redo_conversation",
     "replay_collapsed_instruction",
     "replay_conversation_trace",

src/slm_training/dsl/operators/replay_preference.py

@@ -34,16 +34,29 @@
 ``merge_conversation_branches`` has already confirmed it succeeds, so a
 conflicting merge is never mistakenly added to any ranking denominator; see
 ``test_merge_conflict_never_yields_a_preference_row``.
+
+Sixth slice (v7): :func:`preference_pair_from_replay_row` and
+:func:`preference_pairs_from_trace` convert an already-extracted row into
+this repo's existing ``PreferencePair`` shape (``slm_training.harnesses.
+preference``, the same ``prompt``/``chosen``/``rejected`` schema ``slm
+preference build-pairs``/``train`` already reads and writes). This is data-
+conversion wiring only: it does not train, checkpoint, or score against the
+DSH3-selected policy/control heads (``TypedOperatorPolicyScorer``) or the
+``ObjectiveView`` materializers in ``decision_events_v2.py`` -- see
+``docs/design/dsh5-10-replay-preference-rows.md``'s "Sixth slice (v7)" for
+why those remain untouched and what the conversion honestly can and cannot
+do.
 """
 
 from __future__ import annotations
 
-from collections.abc import Callable
+from collections.abc import Callable, Sequence
 from dataclasses import dataclass
 from enum import Enum
 
 from slm_training.dsl.operators.conversation import (
     ConversationOperation,
+    ConversationTraceError,
     ConversationTraceV1,
 )
 from slm_training.dsl.operators.contracts import (
@@ -63,6 +76,7 @@
 from slm_training.dsl.operators.registry import OperatorLibraryV1, OperatorStateV1
 from slm_training.dsl.pack import DslPack
 from slm_training.harness_core.versioning import build_version_stamp
+from slm_training.harnesses.preference import PreferencePair
 
 ProvenanceFactory = Callable[[OperatorStateV1], ApplicationProvenanceV1]
 
@@ -531,3 +545,91 @@ def extract_merge_preference_row(
         correction_reason="user_merged_diverged_branches",
         legal_set_fingerprint=legal_set.fingerprint,
     )
+
+
+def preference_pair_from_replay_row(
+    row: OperatorReplayPreferenceRowV1,
+    *,
+    input_state_source: str,
+) -> PreferencePair:
+    """Materialize one row into this repo's existing ``PreferencePair`` shape.
+
+    ``prompt`` is the exact, already pack-authorized OpenUI source of
+    ``row.input_state_id`` -- never a fabricated natural-language instruction.
+    No such instruction exists anywhere in a ``ConversationTraceV1`` (its
+    turns are AST operations, not user utterances), so this honestly reuses
+    the one piece of real context the row is grounded in: the rendered state
+    the choice was made from. This is a real, if unusual, use of the
+    ``prompt`` field -- callers reading ``meta.schema`` can tell these pairs
+    apart from ``build_pairs_from_candidates`` output, whose ``prompt`` is a
+    design-task instruction.
+
+    ``chosen``/``rejected`` are the row's own recorded ``chosen_action``/
+    ``rejected_action`` legal-set tokens verbatim (e.g. ``"undo"``,
+    ``"checkout:<state>"``, ``"merge:<pair>"``, or a serialized ``OPERATOR
+    <id> ...`` action) -- never a replayed or reconstructed OpenUI program
+    for the rejected side. Resolving a full alternate rendering for an
+    unchosen operator action would require independently re-running
+    ``OperatorLibraryV1.apply`` against a legal-set candidate that was never
+    actually applied in this trace; that replay is not attempted here, so
+    this slice does not claim ``chosen``/``rejected`` are two alternative
+    full-program candidates the way ``build_pairs_from_candidates`` pairs
+    are. ``composite_reward`` is deliberately never called on these tokens:
+    scoring an action token as if it were OpenUI source (e.g.
+    ``grammar_score("undo")``) would silently manufacture a meaningless
+    number, so ``chosen_score``/``rejected_score`` stay at the ``0.0``
+    default and the honest distinction is recorded in ``meta`` instead.
+
+    Raises ``ValueError`` if ``input_state_source`` is empty -- the one
+    piece of real context this conversion requires -- rather than emitting a
+    pair with a fabricated or blank prompt.
+    """
+    if not input_state_source or not input_state_source.strip():
+        raise ValueError(
+            "replay preference row conversion requires a non-empty input "
+            "state source; refusing to fabricate a prompt"
+        )
+    return PreferencePair(
+        prompt=input_state_source,
+        chosen=row.chosen_action,
+        rejected=row.rejected_action,
+        design_md=None,
+        chosen_score=0.0,
+        rejected_score=0.0,
+        meta={
+            "schema": "operator_replay_preference_pair/v1",
+            "semantic_relation": row.semantic_relation.value,
+            "correction_reason": row.correction_reason,
+            "input_state_id": row.input_state_id,
+            "chosen_output_state_id": row.chosen_output_state_id,
+            "legal_set_fingerprint": row.legal_set_fingerprint,
+        },
+    )
+
+
+def preference_pairs_from_trace(
+    trace: ConversationTraceV1,
+    rows: Sequence[OperatorReplayPreferenceRowV1],
+) -> tuple[tuple[PreferencePair, ...], tuple[dict, ...]]:
+    """Convert every row grounded in ``trace`` to a ``PreferencePair``.
+
+    Returns ``(pairs, skipped)``. A row is skipped -- never fabricated a
+    prompt -- when ``row.input_state_id`` is not one of ``trace``'s own
+    state nodes, e.g. a ``MERGE_SUCCESS`` row (whose input state lives on a
+    ``BranchEditV1`` tip, not in any single trace's ``state_nodes``; convert
+    those with :func:`preference_pair_from_replay_row` directly, passing
+    ``left.output_node.state.source``). Each ``skipped`` entry carries the
+    row (as ``to_dict()``) and the honest reason it could not be converted.
+    """
+    pairs: list[PreferencePair] = []
+    skipped: list[dict] = []
+    for row in rows:
+        try:
+            node = trace.node(row.input_state_id)
+        except ConversationTraceError as exc:
+            skipped.append({"row": row.to_dict(), "reason": str(exc)})
+            continue
+        pairs.append(
+            preference_pair_from_replay_row(row, input_state_source=node.state.source)
+        )
+    return tuple(pairs), tuple(skipped)

src/slm_training/resources/versions.json

@@ -646,6 +646,11 @@
         "tests/test_dsl/test_operator_contracts.py"
       ],
       "history": [
+        {
+          "version": "v16",
+          "date": "2026-07-27",
+          "note": "no-bump: SLM-418 (DSH5-10) sixth slice re-exports preference_pair_from_replay_row and preference_pairs_from_trace from operators/__init__.py (dsl.operators.replay_preference bumped to v7); no change to contracts.py itself"
+        },
         {
           "version": "v16",
           "date": "2026-07-27",
@@ -1091,13 +1096,18 @@
       ]
     },
     "dsl.operators.replay_preference": {
-      "version": "v6",
+      "version": "v7",
       "kind": "harness",
       "paths": [
         "src/slm_training/dsl/operators/replay_preference.py",
         "tests/test_dsl/test_replay_preference.py"
       ],
       "history": [
+        {
+          "version": "v7",
+          "date": "2026-07-27",
+          "note": "SLM-418 (DSH5-10) sixth slice: add preference_pair_from_replay_row and preference_pairs_from_trace, converting an already-extracted OperatorReplayPreferenceRowV1 into this repo's existing PreferencePair shape (slm_training.harnesses.preference, the prompt/chosen/rejected schema slm preference build-pairs/train already reads and writes). prompt is the row's exact pack-authorized input-state OpenUI source (the only real context available -- ConversationTraceV1 turns carry no natural-language instruction); chosen/rejected are the row's own recorded action tokens verbatim, never a replayed alternate OpenUI rendering for the rejected side and never scored via composite_reward (which would silently manufacture a meaningless number for a non-program token). A row whose input_state_id is not a member of the given trace (e.g. a MERGE_SUCCESS row, grounded on a BranchEditV1 tip rather than any one trace's state_nodes) is skipped and recorded, never fabricated a prompt. Data-conversion wiring only -- no model, checkpoint, training run, four-baseline comparison, or held-out measurement is added; the DSH3-selected policy/control heads and ObjectiveView materializers remain untouched"
+        },
         {
           "version": "v6",
           "date": "2026-07-27",