Skip to content

ml: stop the exercise rule priming invented directives — v0.29.1's gate failed scenario 17 (latent since v0.28.0) - #317

Merged
mmcky merged 2 commits into
mainfrom
ml-no-invented-directives
Sep 21, 2026
Merged

mmcky merged 2 commits into
mainfrom
ml-no-invented-directives

Conversation

@mmcky

@mmcky mmcky commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

v0.29.1's §4a gate came back 83 of 84. The miss is scenario 17 (new-document-toc) on the .ml lane, and a re-run against the same tagged bytes failed identically — so the floating tags were not moved and no GitHub release was published. v0 and v0.29 still point at v0.29.0 (a6fda54). This PR is the repair; it ships as v0.29.2.

What failed

The fixture game-theory.md writes its exercises as a plain ## Exercises heading and a numbered list, with no directives. The model wrapped them in {exercise-start} … {exercise-end}. The structural-parity guard did its job and refused the file (directives 3 → 5), which left the sync PR carrying a _toc.yml entry for a lecture it had not delivered — the #222 / #289 class. The primer is the ml exercise-verbatim rule: it names every exercise directive at length and never says they must already be in the source.

It is a latent defect, not only a regression

Two draws were not enough to tell, and nor were six per arm (they read 1/6 against 5/6 and looked like a clean regression). Twelve draws per arm on the fixture, refusals by the parity guard:

Rules Glossary Refused
v0.29.0 v0.6.0 5 / 12
v0.29.0 v0.7.0 8 / 12
v0.29.1 v0.6.0 11 / 12
v0.29.1 v0.7.0 11 / 12
v0.29.1 + this PR v0.7.0 0 / 24 (two wordings, twelve each)

The defect has been there since the rule arrived in v0.28.0: the v0.28.0, v0.28.1 and v0.29.0 gates each passed scenario 17 on a single draw with roughly a 40% chance of failing. The #315 rules aggravate it to about 90%. The crossing places that in the rules rather than the glossary; why longer comma and plural rules should have this effect is not known, and the log says so rather than guessing.

The change

One sentence added to the exercise rule — still 28 rules. It says the rule concerns directives the source already contains, that no directive, fence or label may be added, and that a plain ## Exercises heading and list keeps its English text and its source structure. A first wording told the model to translate that list; it never did at either version (0 of 19 written draws), and the editor's ruling covers all exercise content, so the wording was changed to ask for the behaviour the ruling wants. Reverting #315 was not considered: those are the editor's answers, and the defect predates them.

Harness

Local dist-action bundle through the action path, on the fixture PR that failed (test-translation-sync#907): delivered 2 of 2 — test-translation-sync.ml#221, #222, each with game-theory.md, its state file and _toc.yml. The tagged v0.29.1 failed 2 of 2 on the same PR.

Worth carrying forward

A single-draw gate cannot see a 40% defect; it passed three releases. Retries or N draws on the new-document scenarios belong with #260 / W1 (#259). And a rule that lists markup should say "already in the source".

Suite 1590 green (+2); lint, format:check, check-dev-refs clean; dist-action/ rebuilt.

🤖 Generated with Claude Code

… has

v0.29.1's gate failed scenario 17 on the .ml lane twice: the model wrapped a
plain '## Exercises' list in {exercise-start} and structural parity refused
the file. Twelve draws per arm: 5/12 refused at v0.29.0 (latent since
v0.28.0), 11/12 at v0.29.1, 0/24 with the scope sentence. Action path on the
failing fixture PR: delivered 2 of 2. v0.29.1 stays tagged and unreleased;
this ships as v0.29.2.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 21, 2026 11:53
@mmcky mmcky mentioned this pull request Sep 21, 2026
7 tasks

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Release-status documentation is internally inconsistent (STATE/CHANGELOG) and should be clarified before shipping the patch release notes.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 Low severity

Open (2)
What changed in this PR

This PR updates the Malayalam (ml) language rules to explicitly scope the “exercise-verbatim” rule to directives already present in the source, preventing the model from inventing {exercise-*} directives around plain ## Exercises sections (fixing the scenario-17 .ml gate failure that blocked the v0.29.1 release).

Changes:

  • Extend the ml exercise-verbatim rule to forbid introducing any new exercise directives/fences/labels when the source uses a plain ## Exercises heading + list.
  • Add a targeted Jest assertion to lock in the new scoping language while keeping the rule count at 28.
  • Rebuild dist-action/ and document the incident/fix in CHANGELOG.md and .dev/ notes.
File Description
src/​language-config.ts Adds the scoping sentence to the Malayalam exercise-verbatim rule to prevent directive invention.
src/​__tests__/​language-config.test.ts Adds regression coverage asserting the rule explicitly forbids adding directives and keeps rule count unchanged.
dist-action/​index.js Updates the bundled Action output to reflect the source rule change.
CHANGELOG.md Documents the fix under [Unreleased].
.dev/​STATE.md Records the v0.29.1 gate failure context and that the repair ships as v0.29.2.
.dev/​log/​2026-09-21-v0291-gate-scenario17.md Adds a detailed incident log entry for the gate failure and resolution.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .dev/STATE.md
Comment thread CHANGELOG.md Outdated
…ead as if it shipped

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mmcky
mmcky merged commit 62504c1 into main Sep 21, 2026
1 check passed
@mmcky
mmcky deleted the ml-no-invented-directives branch September 21, 2026 12:24
@mmcky mmcky mentioned this pull request Sep 21, 2026
7 tasks
mmcky added a commit that referenced this pull request Sep 21, 2026
Patch: the ml exercise-rule scope sentence (#317) that v0.29.1's failed gate
called for, carrying the editor's round-3 answers (#315) that v0.29.1 never
delivered. No source change in this commit; dist-action/ rebuilds
byte-identical.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
mmcky added a commit that referenced this pull request Sep 23, 2026
Directory-rename detection moved the six .dev/ files main added since the
branch point (two decision records, three log entries, the #319 arm
references) into .qe/dev/. Conflicts resolved:

- .qe/dev/STATE.md: main's re-verified page (#330), with its one .dev/
  path mention updated and the #314 In-flight bullet replaced by a
  Recently-landed entry for the move itself.
- src/language-config.ts: main's ml rules (#315/#317) with the four
  decision-record path comments pointed at .qe/dev/decisions/.
- dist-action/index.js: rebuilt; differs from main only by those comments.

Also carried through: rate-check.sh (#323, merged after this branch) wrote
its logs to .dev/scratch/rate-check/, which would recreate an in-tree
scratch folder that is no longer git-ignored. Logs now go to
${TMPDIR:-/tmp}/action-translation-rate-check/<stamp>/ per
D-2026-09-21-no-scratch-in-tree; its README line updated to match.

npm test 1590/1590, lint and format:check clean, check-dev-refs: 168 .qe/
references resolve.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants