ml round-2 arm: second draw at v0.27.0, closeness to the reviewed text - #299
Merged
Conversation
…eness to the reviewed text Archives functions-v0.27.0.md and adds the seed/arm-1/arm-2 closeness table (exact-line match 11% → 21% → 19%; whole-prose ratio 0.576 → 0.650 → 0.638), the arm-2 signatures, and four findings: the ex5 negative example works, set did not recur, refer calqued on both draws, list-item -ഉം regressed once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Archives a second Malayalam functions regeneration run at the released engine (v0.27.0) alongside the existing round-2 arm, and documents quantitative “closeness” metrics versus the editor-reviewed text to help bound model variance and assess rule effectiveness.
Changes:
- Extend the experiment README with “second draw” run metadata, closeness metrics, and observed signature/regression notes.
- Add the new archived output file
functions-v0.27.0.mdfor the v0.27.0 draw.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| experiments/ml-benchmark/arms/2026-09-01-round2-rules-sonnet5/README.md | Documents the second draw and adds closeness/signature metrics vs the reviewed text. |
| experiments/ml-benchmark/arms/2026-09-01-round2-rules-sonnet5/functions-v0.27.0.md | Adds the archived v0.27.0 regeneration output used for the comparison. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ഈ function define ചെയ്തു കഴിഞ്ഞു, ഇനി നമുക്ക് ഇതിനെ *call* ചെയ്ത് പ്രതീക്ഷിച്ചത് പോലെ പ്രവർത്തിക്കുന്നുണ്ടോ എന്ന് check ചെയ്യാം: | ||
|
|
||
| ```{code-cell} python3 | ||
| f(1) |
This was referenced Sep 1, 2026
Merged
mmcky
added a commit
that referenced
this pull request
Sep 3, 2026
* chore: release v0.28.0 Release prep per the AGENTS.md checklist (steps 1–4): [Unreleased] promoted to [0.28.0] - 2026-09-03 with an entry added for #299 (experiment archive, no runtime change); version bumped in package.json and package-lock.json; STATE.md in-flight updated (W1 #259 retargets to v0.29.0); dist and dist-action rebuilt so the bundle reports engineVersion 0.28.0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * dev: STATE.md in-flight entry for the v0.28.0 cut; W1 retargets to v0.29.0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Keep an empty [Unreleased] header above the released section; record the convention in the release checklist Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Archives a second regeneration of
functionsat the released engine (125801c,claude-sonnet-5,--localize none) beside the round-2 arm, per the DISPOSITIONS.md process rule, and adds a closeness measure against the editor's reviewed text (mlmain@c1200fa): exact-line match 11% (seed) → 21% (arm 1) → 19% (arm 2); whole-proseSequenceMatcherratio 0.576 → 0.650 → 0.638. The engine closed roughly a third of the seed → reviewed distance and the two draws agree to within two lines.Findings from the second draw: the ex5 negative example added in #297 works (the recursion exercise is Malayalam again);
setdid not recur;refer→സൂചിപ്പിക്കുന്നുrecurred on both draws in the same sentence, so the round-1 pin is not holding there — a rule-example candidate; list-item-ഉംregressed once; terminal punctuation remains the weak class (17 bare endings), unchanged case for the deterministic post-processing step (#260). Docs/experiment archive only — no engine change. Tracked on #296.🤖 Generated with Claude Code