Skip to content

fix(research): RSI correction round — token matching, artifact-derived reports, lineage and provider validation - #1

Merged
rustfuture merged 3 commits into
mainfrom
fix/research-correction-round
Sep 15, 2026
Merged

rustfuture merged 3 commits into
mainfrom
fix/research-correction-round

Conversation

@rustfuture

Copy link
Copy Markdown
Owner

Why

A prior audit found the reported multi-seed numbers did not match results/multi_seed_results.json.
Root cause: README.md carried a hand-written results table that no committed artifact ever
produced; report.md was generated but nothing checked it, and the README was never regenerated.

Re-running the committed v1 code reproduces first_run.json, history.jsonl, metrics.csv
byte-for-byte and the multi-seed/ablation numbers modulo runtime_seconds, so the artifact was
sound and the prose was stale.

What changed

  • Experiment definition (versioned). Keywords now match whole tokens, so the positive keyword
    safe no longer fires inside unsafe and clear inside unclear. experiment_version = v2-token-match, schema_version = 2. v1 artifacts preserved verbatim in results/archive-v1/.
  • Reporting. rsi_framework.reporting renders results/report.md and the README results block
    from results/*.json. --check and test_report_numbers_match_artifacts fail on drift; CI runs
    the check.
  • Counting. "Accepted versions including the baseline" is now reported separately from
    "accepted new changes".
  • Provenance. results/provenance.json records commit, dirty paths, config hash, dataset hash,
    splits, run command and interpreter.
  • Lineage. History rows carry a unique event_id distinct from the policy version. A repeated
    policy hash in a later generation no longer overwrites earlier records; missing parents are
    reported as orphans and cycles are detected. The recorded parent_event_id is always the accepted
    event that held the active policy.
  • Provider injection. run_experiment(..., candidate_generator=...); JsonFileProposalProvider
    reads externally produced proposals; validate_proposal rejects no-ops, multi-edit proposals,
    illegal tokens, keywords outside the vocabulary and out-of-bounds bias. OpenWeightColabL4Stub
    is documented as an unwired boundary; no LLM was run.
  • Ablations. Added ablation_random_selection (genuine score-free control); ablation_no_selection
    relabelled as dev-blind greedy selection; each mode records the mechanism it changes and runs with
    the same seed, dataset hash and budget.
  • Terminology. rollback_regressionrejected_dev_regression (the rule rejects a candidate;
    no applied state is reverted). Legacy label retained as a v1 alias.
  • Claim scope. Byte equality limited to the four runtime-free artifacts; leakage test described as
    a scoped invariance check; held-out n=8 stated wherever outcomes appear; multi-seed runs described
    as replays of one shared synthetic pool.

Verification

  • python3 -m unittest discover -s tests -v → 23 tests, OK (also on python3.11).
  • python3 -m rsi_framework.reporting --check → consistent.
  • cmp on first_run.json, history.jsonl, metrics.csv, progress.svg after an independent run → byte-identical.
  • Re-run benchmarks → multi_seed_results.json / ablation_results.json equal modulo runtime_seconds.

Pending

  • Real open-weight LLM integration is not done. The injection point and validation gate are
    implemented and tested with a file-backed provider; the model evaluation is explicitly pending.

…ts, provider validation

Correction round for the RSI harness.

- Match keywords as whole tokens, not substrings, so 'safe' no longer fires
  inside 'unsafe' or 'clear' inside 'unclear'. Version the experiment as
  v2-token-match and archive v1 artifacts under results/archive-v1/.
- Render report.md and the README results block from results/*.json so the
  stale hand-written numbers (0.812 +/- 0.054 train, 1.00 +/- 0.00 rejections)
  cannot recur; add a report/artifact consistency test and CI check.
- Separate accepted versions including the baseline from accepted new changes.
- Record provenance (commit, dirty paths, config hash, dataset hash, command).
- Key lineage on a unique event_id so a repeated policy hash in a later
  generation no longer overwrites records; report orphans and detect cycles;
  keep the recorded parent on the accepted event only.
- Inject the candidate generator and validate every proposal; add
  JsonFileProposalProvider and reject malformed/out-of-policy proposals.
- Add ablation_random_selection as a score-free control; relabel
  ablation_no_selection as dev-blind greedy selection; record mechanisms.
- Rename rollback_regression to rejected_dev_regression and document that the
  rule rejects a candidate rather than reverting applied state.
- Scope the byte-equality, leakage and significance claims to what was measured.
Adds the multi-seed list, config seed and dataset pool description to
results/provenance.json so a run is reproducible from the artifact alone,
plus a provenance unit test.
@rustfuture
rustfuture merged commit 471b627 into main Sep 15, 2026
1 check passed
@rustfuture
rustfuture deleted the fix/research-correction-round branch September 16, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant