fix(research): RSI correction round — token matching, artifact-derived reports, lineage and provider validation - #1
Merged
Conversation
…ts, provider validation Correction round for the RSI harness. - Match keywords as whole tokens, not substrings, so 'safe' no longer fires inside 'unsafe' or 'clear' inside 'unclear'. Version the experiment as v2-token-match and archive v1 artifacts under results/archive-v1/. - Render report.md and the README results block from results/*.json so the stale hand-written numbers (0.812 +/- 0.054 train, 1.00 +/- 0.00 rejections) cannot recur; add a report/artifact consistency test and CI check. - Separate accepted versions including the baseline from accepted new changes. - Record provenance (commit, dirty paths, config hash, dataset hash, command). - Key lineage on a unique event_id so a repeated policy hash in a later generation no longer overwrites records; report orphans and detect cycles; keep the recorded parent on the accepted event only. - Inject the candidate generator and validate every proposal; add JsonFileProposalProvider and reject malformed/out-of-policy proposals. - Add ablation_random_selection as a score-free control; relabel ablation_no_selection as dev-blind greedy selection; record mechanisms. - Rename rollback_regression to rejected_dev_regression and document that the rule rejects a candidate rather than reverting applied state. - Scope the byte-equality, leakage and significance claims to what was measured.
Adds the multi-seed list, config seed and dataset pool description to results/provenance.json so a run is reproducible from the artifact alone, plus a provenance unit test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A prior audit found the reported multi-seed numbers did not match
results/multi_seed_results.json.Root cause:
README.mdcarried a hand-written results table that no committed artifact everproduced;
report.mdwas generated but nothing checked it, and the README was never regenerated.Re-running the committed v1 code reproduces
first_run.json,history.jsonl,metrics.csvbyte-for-byte and the multi-seed/ablation numbers modulo
runtime_seconds, so the artifact wassound and the prose was stale.
What changed
safeno longer fires insideunsafeandclearinsideunclear.experiment_version = v2-token-match,schema_version = 2. v1 artifacts preserved verbatim inresults/archive-v1/.rsi_framework.reportingrendersresults/report.mdand the README results blockfrom
results/*.json.--checkandtest_report_numbers_match_artifactsfail on drift; CI runsthe check.
"accepted new changes".
results/provenance.jsonrecords commit, dirty paths, config hash, dataset hash,splits, run command and interpreter.
event_iddistinct from the policyversion. A repeatedpolicy hash in a later generation no longer overwrites earlier records; missing parents are
reported as orphans and cycles are detected. The recorded
parent_event_idis always the acceptedevent that held the active policy.
run_experiment(..., candidate_generator=...);JsonFileProposalProviderreads externally produced proposals;
validate_proposalrejects no-ops, multi-edit proposals,illegal tokens, keywords outside the vocabulary and out-of-bounds bias.
OpenWeightColabL4Stubis documented as an unwired boundary; no LLM was run.
ablation_random_selection(genuine score-free control);ablation_no_selectionrelabelled as dev-blind greedy selection; each mode records the mechanism it changes and runs with
the same seed, dataset hash and budget.
rollback_regression→rejected_dev_regression(the rule rejects a candidate;no applied state is reverted). Legacy label retained as a v1 alias.
a scoped invariance check; held-out n=8 stated wherever outcomes appear; multi-seed runs described
as replays of one shared synthetic pool.
Verification
python3 -m unittest discover -s tests -v→ 23 tests, OK (also on python3.11).python3 -m rsi_framework.reporting --check→ consistent.cmponfirst_run.json,history.jsonl,metrics.csv,progress.svgafter an independent run → byte-identical.multi_seed_results.json/ablation_results.jsonequal moduloruntime_seconds.Pending
implemented and tested with a file-backed provider; the model evaluation is explicitly pending.