ml: stop the exercise rule priming invented directives — v0.29.1's gate failed scenario 17 (latent since v0.28.0) - #317
Merged
Merged
Conversation
… has
v0.29.1's gate failed scenario 17 on the .ml lane twice: the model wrapped a
plain '## Exercises' list in {exercise-start} and structural parity refused
the file. Twelve draws per arm: 5/12 refused at v0.29.0 (latent since
v0.28.0), 11/12 at v0.29.1, 0/24 with the scope sentence. Action path on the
failing fixture PR: delivered 2 of 2. v0.29.1 stays tagged and unreleased;
this ships as v0.29.2.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Release-status documentation is internally inconsistent (STATE/CHANGELOG) and should be clarified before shipping the patch release notes.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
This PR updates the Malayalam (ml) language rules to explicitly scope the “exercise-verbatim” rule to directives already present in the source, preventing the model from inventing {exercise-*} directives around plain ## Exercises sections (fixing the scenario-17 .ml gate failure that blocked the v0.29.1 release).
Changes:
- Extend the
mlexercise-verbatim rule to forbid introducing any new exercise directives/fences/labels when the source uses a plain## Exercisesheading + list. - Add a targeted Jest assertion to lock in the new scoping language while keeping the rule count at 28.
- Rebuild
dist-action/and document the incident/fix inCHANGELOG.mdand.dev/notes.
| File | Description |
|---|---|
src/language-config.ts |
Adds the scoping sentence to the Malayalam exercise-verbatim rule to prevent directive invention. |
src/__tests__/language-config.test.ts |
Adds regression coverage asserting the rule explicitly forbids adding directives and keeps rule count unchanged. |
dist-action/index.js |
Updates the bundled Action output to reflect the source rule change. |
CHANGELOG.md |
Documents the fix under [Unreleased]. |
.dev/STATE.md |
Records the v0.29.1 gate failure context and that the repair ships as v0.29.2. |
.dev/log/2026-09-21-v0291-gate-scenario17.md |
Adds a detailed incident log entry for the gate failure and resolution. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ead as if it shipped Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Sep 23, 2026
Directory-rename detection moved the six .dev/ files main added since the branch point (two decision records, three log entries, the #319 arm references) into .qe/dev/. Conflicts resolved: - .qe/dev/STATE.md: main's re-verified page (#330), with its one .dev/ path mention updated and the #314 In-flight bullet replaced by a Recently-landed entry for the move itself. - src/language-config.ts: main's ml rules (#315/#317) with the four decision-record path comments pointed at .qe/dev/decisions/. - dist-action/index.js: rebuilt; differs from main only by those comments. Also carried through: rate-check.sh (#323, merged after this branch) wrote its logs to .dev/scratch/rate-check/, which would recreate an in-tree scratch folder that is no longer git-ignored. Logs now go to ${TMPDIR:-/tmp}/action-translation-rate-check/<stamp>/ per D-2026-09-21-no-scratch-in-tree; its README line updated to match. npm test 1590/1590, lint and format:check clean, check-dev-refs: 168 .qe/ references resolve. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

v0.29.1's §4a gate came back 83 of 84. The miss is scenario 17 (
new-document-toc) on the.mllane, and a re-run against the same tagged bytes failed identically — so the floating tags were not moved and no GitHub release was published.v0andv0.29still point at v0.29.0 (a6fda54). This PR is the repair; it ships as v0.29.2.What failed
The fixture
game-theory.mdwrites its exercises as a plain## Exercisesheading and a numbered list, with no directives. The model wrapped them in{exercise-start}…{exercise-end}. The structural-parity guard did its job and refused the file (directives 3 → 5), which left the sync PR carrying a_toc.ymlentry for a lecture it had not delivered — the #222 / #289 class. The primer is themlexercise-verbatim rule: it names every exercise directive at length and never says they must already be in the source.It is a latent defect, not only a regression
Two draws were not enough to tell, and nor were six per arm (they read 1/6 against 5/6 and looked like a clean regression). Twelve draws per arm on the fixture, refusals by the parity guard:
The defect has been there since the rule arrived in v0.28.0: the v0.28.0, v0.28.1 and v0.29.0 gates each passed scenario 17 on a single draw with roughly a 40% chance of failing. The #315 rules aggravate it to about 90%. The crossing places that in the rules rather than the glossary; why longer comma and plural rules should have this effect is not known, and the log says so rather than guessing.
The change
One sentence added to the exercise rule — still 28 rules. It says the rule concerns directives the source already contains, that no directive, fence or label may be added, and that a plain
## Exercisesheading and list keeps its English text and its source structure. A first wording told the model to translate that list; it never did at either version (0 of 19 written draws), and the editor's ruling covers all exercise content, so the wording was changed to ask for the behaviour the ruling wants. Reverting #315 was not considered: those are the editor's answers, and the defect predates them.Harness
Local
dist-actionbundle through the action path, on the fixture PR that failed (test-translation-sync#907): delivered 2 of 2 — test-translation-sync.ml#221, #222, each withgame-theory.md, its state file and_toc.yml. The tagged v0.29.1 failed 2 of 2 on the same PR.Worth carrying forward
A single-draw gate cannot see a 40% defect; it passed three releases. Retries or N draws on the new-document scenarios belong with #260 / W1 (#259). And a rule that lists markup should say "already in the source".
Suite 1590 green (+2); lint,
format:check,check-dev-refsclean;dist-action/rebuilt.🤖 Generated with Claude Code