#148 is closed, but it deliberately left its own triggering defect in place. Recording that here so it does not get lost behind a closed issue.
What v0.23.0 did and did not do
#148 was filed because the four diffChecks gate the auto-merge recommendation absolutely while being model output — a confidently wrong boolean is indistinguishable, to the gate, from a real structural failure. It fired on QuantEcon/lecture-python.zh-cn#199, which was routed to editor on positionCorrect with findings asserting the source's code change had never reached the target. It had.
v0.23.0 (#152) made structurePreserved and headingMapCorrect deterministic, and tagged all four with provenance in diffCheckSources. positionCorrect — the check that actually misfired — is still model-asserted. It now gates through a minor/diff-check finding rather than the boolean, so it is attributable rather than eliminated. The #199-class false gate can recur; it will just be visible as a model opinion when it does.
That was a deliberate scope choice, not an oversight: deterministic positionCorrect requires attributing each source change to a target section across a translated document, which is novel logic with no ground-truth corpus. Building it badly replaces model false-gates with engine false-gates, which look authoritative and are harder to spot.
What has changed since, that makes this decidable
The measurement now exists. diffCheckSources lets shadow-mode data separate model-asserted gates from deterministic ones, so the precision of positionCorrect can be reported as its own number instead of blending into the base rate. That number is the input this decision was waiting on.
Do not build this before the measurement exists to justify it. If model-asserted positionCorrect turns out to be right 95% of the time on real sync PRs, the deterministic version is not worth its false-gate risk. If it is the coin-flip that n=2 suggested, it is.
A specific hypothesis worth testing first, cheaply
Recorded on #148 and still untested. positionCorrect is specified in the prompt as "Do changes appear in the same sections as source? Section order should match." — prose-oriented. #199's diff is entirely code (a code-cell edit plus .translate frontmatter). There is no prose movement to assess, so the criterion is being applied outside the shape it was written for.
Testable without building anything: run the reviewer over a set of code-only sync diffs and measure how often positionCorrect comes back false. If code-only diffs are a systematic weak spot, the cheapest fix is a prompt change scoped to that case — not a deterministic reimplementation.
Sketch, if it is built
The pieces exist. MystParser.parseSections gives the section tree on both sides; buildHeadingMap/extractHeadingMap give the source-heading → target-heading correspondence; the source before/after diff gives which source sections changed. The check would be: for each changed source section, did the corresponding target section (via the heading map) also change?
The hard parts are the ones that would generate false gates:
- a source change that legitimately produces no target change (whitespace, a comment the translation had already localised)
- a target change that is correct but lands in a different section because the translation restructured legitimately
- resync PRs, where the whole document is rewritten and per-section attribution is meaningless —
evaluateDiff already special-cases these
- new or deleted sections, where the correspondence does not exist
Any of these getting it wrong produces an engine false-gate, which is worse than the model one because it carries the authority of deterministic in diffCheckSources.
Related
#148 is closed, but it deliberately left its own triggering defect in place. Recording that here so it does not get lost behind a closed issue.
What v0.23.0 did and did not do
#148 was filed because the four
diffChecksgate the auto-merge recommendation absolutely while being model output — a confidently wrong boolean is indistinguishable, to the gate, from a real structural failure. It fired on QuantEcon/lecture-python.zh-cn#199, which was routed toeditoronpositionCorrectwith findings asserting the source's code change had never reached the target. It had.v0.23.0 (#152) made
structurePreservedandheadingMapCorrectdeterministic, and tagged all four with provenance indiffCheckSources.positionCorrect— the check that actually misfired — is still model-asserted. It now gates through aminor/diff-checkfinding rather than the boolean, so it is attributable rather than eliminated. The #199-class false gate can recur; it will just be visible as a model opinion when it does.That was a deliberate scope choice, not an oversight: deterministic
positionCorrectrequires attributing each source change to a target section across a translated document, which is novel logic with no ground-truth corpus. Building it badly replaces model false-gates with engine false-gates, which look authoritative and are harder to spot.What has changed since, that makes this decidable
The measurement now exists.
diffCheckSourceslets shadow-mode data separate model-asserted gates from deterministic ones, so the precision ofpositionCorrectcan be reported as its own number instead of blending into the base rate. That number is the input this decision was waiting on.Do not build this before the measurement exists to justify it. If model-asserted
positionCorrectturns out to be right 95% of the time on real sync PRs, the deterministic version is not worth its false-gate risk. If it is the coin-flip that n=2 suggested, it is.A specific hypothesis worth testing first, cheaply
Recorded on #148 and still untested.
positionCorrectis specified in the prompt as "Do changes appear in the same sections as source? Section order should match." — prose-oriented. #199's diff is entirely code (a code-cell edit plus.translatefrontmatter). There is no prose movement to assess, so the criterion is being applied outside the shape it was written for.Testable without building anything: run the reviewer over a set of code-only sync diffs and measure how often
positionCorrectcomes back false. If code-only diffs are a systematic weak spot, the cheapest fix is a prompt change scoped to that case — not a deterministic reimplementation.Sketch, if it is built
The pieces exist.
MystParser.parseSectionsgives the section tree on both sides;buildHeadingMap/extractHeadingMapgive the source-heading → target-heading correspondence; the source before/after diff gives which source sections changed. The check would be: for each changed source section, did the corresponding target section (via the heading map) also change?The hard parts are the ones that would generate false gates:
evaluateDiffalready special-cases theseAny of these getting it wrong produces an engine false-gate, which is worse than the model one because it carries the authority of
deterministicindiffCheckSources.Related