Skip to content

refactor(rrsi): fidelity closure — the wiring consumes the pinned K/L_t fixtures - #333

Merged
Birfy merged 1 commit into
Birfy:mainfrom
HugoXK:feat/rrsi-fidelity
Oct 17, 2026
Merged

Birfy merged 1 commit into
Birfy:mainfrom
HugoXK:feat/rrsi-fidelity

Conversation

@HugoXK

@HugoXK HugoXK commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator

The problem

The RRSI wiring (examples/rrsi/rrsi_policies.py, merged #254) re-derived two things the pinned upstream fixtures already provide: component attribution used a first-path-segment heuristic (skills/x/SKILL.md -> "skills") instead of the pinned components vocabulary (K / K_STR / normalize evidence rules -> "skill"), and the per-edit ledger was an in-memory list instead of the pinned history.History (durable JSONL L_t, merged #253). The fidelity claim in docs/algo-rrsi.md said the port is pinned; the wiring was not.

What this PR changes

  • Component attribution through the pinned vocabulary. RRsiState.components_of now classifies each op via components.normalize / classify_diff: skills/x/SKILL.md is a skill edit, ToolRegistry(...) is client_tool, a pure prose edit is prompt. Novelty and pruning now count the components RRSI actually counts.
  • The per-edit ledger is history.History (durable) when history_path= is given, with an in-memory twin for scratch runs and tests, pinned to the same summaries by a parity test so the twin cannot silently drift.
  • RRsiProposalPolicy reads History.render / prune_set (with the actionable accepted edits to remove) / stall_flag.
  • RRsiAcceptance scores novelty over History.incumbent_component_counts and records every decision via History.append_candidate.
  • Checkpoint separation. The aggregator checkpoint now carries only the search state (round, S*, incumbent cost, progress); the per-edit evidence is the History's own file, so a resumed run reads the same evidence the loop conditioned on.
  • stalled() uses the pinned history.stall_flag, which requires a full window of progress points (t >= w) — a stricter, correcter reading than the old tail-slice heuristic; the integration tests were updated to that semantics.

Fidelity / deviations

Behavior Before Now
skills/x/SKILL.md "skills" (path segment) "skill" (upstream K_STR)
ToolRegistry(...) path segment "client_tool"
pure prose edit path segment "prompt"
per-edit ledger in-memory list in the checkpoint history.History JSONL (durable)
stall tail-slice heuristic pinned stall_flag (t >= w)
resume ledger lost History's own file re-read on resume

Verification

Item Result
Upstream vocabulary classifies skills/tool/memory/prompt correctly 2 tests
Memory and file History backends agree (yield_g/prune_set/render/counts) 1 test
Durable History survives reopen and resume 1 test
Checkpoint carries search state; History stays durable 1 test
A genuinely novel structural component carries a within-band candidate 1 test
Existing integration/history/selection/schedule suite 107 passed
Changelog fragment valid exit 0

Files

5 files changed, +435, -159

Modified:

  • examples/rrsi/rrsi_policies.py — upstream_components, _InMemoryHistory, History-backed RRsiState/RRsiProposalPolicy/RRsiAcceptance, build_rrsi_bundle(history_path=)
  • tests/test_rrsi_integration.py — updated to the upstream vocabulary and History API (intent preserved)
  • docs/algo-rrsi.md — the wiring now consumes the pinned fixtures

New:

  • tests/test_rrsi_fidelity.py (+~150) — 6 fidelity tests
  • changelog.d/331-rrsi-fidelity.md — fragment

…_t fixtures

Component attribution goes through the pinned upstream vocabulary
(components.normalize/classify_diff: skills/ -> skill, ToolRegistry ->
client_tool, pure prose -> prompt) instead of the first-path-segment
heuristic. The per-edit ledger is history.History (durable JSONL) when
build_rrsi_bundle(history_path=...) is given, with an in-memory twin pinned to
the same summaries by a parity test.

- RRsiProposalPolicy reads History.render / prune_set (with actionable accepted
  edits) / stall_flag;
- RRsiAcceptance scores novelty over History.incumbent_component_counts and
  records via append_candidate;
- the aggregator checkpoint now carries only the search state (round, S*,
  incumbent cost, progress); the per-edit evidence is the History's own file,
  so a resumed run reads the same evidence the loop conditioned on;
- stalled() uses the pinned history.stall_flag, which requires a full window of
  progress points (t >= w) — a stricter, correcter reading than the old
  tail-slice heuristic, and the integration tests were updated to that.

Six new fidelity tests pin the vocabulary, memory/file parity, durable resume
and a genuinely novel structural component carrying a within-band candidate.

@Birfy Birfy left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE

This is the right fidelity closure: the wiring stops re-deriving what components / history already pin, and the docs/tests match the claim.

Correctness

  • upstream_components → normalize(None, key\nvalue) is the right call site. Spot-checked skills/… → skill, ToolRegistry → client_tool, prose → prompt; novelty/K_STR now see the vocabulary Algorithm 2 actually counts.
  • judge(..., history.incumbent_component_counts()) is a real fix vs the old attempts Counter. _novelty / components.novelty want accepted structural counts on the incumbent; feeding proposal attempts suppressed novelty after a rejected try.
  • Checkpoint / History split is sound: aggregator payload keeps search state (round / S* / cost / progress); the JSONL is the durable ledger. Resume with the same history_path re-reads the evidence the loop conditioned on.
  • stalled() → stall_flag(progress, len-1, w, δ) matches the pinned t >= w guard; integration tests correctly moved to a full window.

Design fit

  • Reusing History / components instead of a parallel ledger is the port convention the module docstring promised.
  • _InMemoryHistory as a scratch twin + parity test is a good compromise for tests that do not want a path.
  • build_rrsi_bundle(history_path=...) is the right seam; docs/algo-rrsi.md updated accordingly.

Tests / verification (local)

Fresh clone at f79e936. 107 RRSI tests green (test_rrsi_fidelity 6, integration 22, history 31, selection 29, schedule 19). CI: build green; test matrix 3.9–3.13 still pending at review time.

Docs / changelog

Fidelity table and wiring section in docs/algo-rrsi.md match the code. Fragment is present and readable.

Non-blocking follow-ups

  1. Changelog slug 331-rrsi-fidelity.md — open issue #331 is the unrelated Calibrator min_labels/min_per_stratum validation bug. Prefer renaming the fragment to track this PR (e.g. 333-…) or an issue you open for the fidelity closure, so the next release notes are not confused with #331.
  2. Parity is incomplete on rounded fields. File History.append_candidate rounds delta_S/delta_C/S/C; _InMemoryHistory does not (and skips _finite). Clean values in test_memory_and_file_history_agree pass, but e.g. delta_S=0.123456789 / C=100.44 makes render diverge. Mirror the rounding (and ideally _finite) in the twin, or assert only the summary methods that stay equal.
  3. Nits: unused import json in tests/test_rrsi_fidelity.py; unused field import in rrsi_policies.py; mid-module from .components import K as _K_ALL could sit with the other imports.
  4. Outcome spelling: wiring writes lowercase "accepted"/"rejected" while MEASURED_OUTCOMES documents ACCEPTED/REJECTED/LOST. Harmless today (append does not validate), but aligning would reduce future drift.

No merge from this review (routine policy).

Birfy commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Two additional P2 findings from independent source review at f79e9361c13db355f11b1d99a95577eb2f91da8b, beyond the existing follow-ups. Both concern the opt-in RRSI example bundle. The scenarios below are source-derived and proposed regressions, not locally executed reproductions (project execution remains on hold).

  1. Default checkpoint/restore loses the history used for decisions. checkpoint / restore now omit all history even when history_path=None selects the in-memory backend. A fresh default bundle restores S*/round/progress but loses accepted component counts, pruning evidence and proposal history. The existing end-to-end resume test uses this default and only checks S*. Proposed regression: record an accepted skill, round-trip the default state, and compare the next flat-score/flat-cost skill candidate with uninterrupted execution (w_n=1). The resumed state should not award novelty again. Please preserve in-memory history in the checkpoint, or automatically arrange durable history for checkpointed runs; the explicit same-file path case does not cover this default.

  2. Raw replacement text bypasses the prose-only classifier. upstream_components passes key + '\n' + value to normalize, while text_only recognizes changed lines only with +/- markers. A SingleSlot prompt replacement such as Do not mention ToolRegistry in the final answer therefore matches client_tool, although no tool machinery was added. That false structural tag supplies novelty to a flat candidate. Proposed regression: create this prompt via SingleSlot.to_diff, require component prompt, and require no novelty-based acceptance at equal score/cost. Please supply actual edit/context semantics rather than treating structural words in replacement prose as structural edits.

@Birfy
Birfy enabled auto-merge (squash) October 17, 2026 04:14
@Birfy
Birfy merged commit f413952 into Birfy:main Oct 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants