Observed
On QuantEcon/lecture-python-programming.fr#12 (review run 2026-07-17, workflow tracking @v0 → v0.17.0), the review comment rendered Formatting: undefined/10, Overall: NaN/10, and Verdict: FAIL — while the review text itself was positive ("accurate, fluent, and well-formatted") and Diff Quality scored 10/10. The diff contained only three URL swaps inside code cells, so a FAIL verdict was clearly wrong.
Root cause
evaluateTranslation consumes the model's JSON without validating the four criterion scores (src/reviewer.ts:816-821):
const result: any = await this.callWithRetry(prompt, MAX_TOKENS.review, 'evaluateTranslation');
const score =
result.accuracy * 0.35 +
result.fluency * 0.25 +
result.terminology * 0.25 +
result.formatting * 0.15;
When the model omits one criterion (here formatting), undefined * 0.15 yields NaN, which propagates through overallScore = translationQuality.score * 0.7 + diffQuality.score * 0.3 (src/reviewer.ts:667) and then falls through both verdict thresholds (src/reviewer.ts:669-675) — NaN >= 8 and NaN >= 6 are both false — landing on FAIL.
So a partially-formed model response silently converts into the harshest verdict, and the report renders undefined/10 and NaN/10 rather than failing loudly.
Suggested fix
Validate the score object after callWithRetry and retry when any of the four criteria is missing or non-numeric, mirroring the existing max_tokens-truncation retry (src/reviewer.ts:337). As a belt-and-braces fallback, either renormalise the weights over the criteria that are present or throw rather than letting NaN reach the verdict comparison — an incomplete review should be an error or a retry, never an automatic FAIL.
Impact
Low frequency but misleading: a FAIL verdict on a clean sync PR sends a human to look for translation defects that do not exist, and erodes trust in PASS/FAIL as a merge signal. Found while reviewing the programming-series sync PRs across editions.
Observed
On QuantEcon/lecture-python-programming.fr#12 (review run 2026-07-17, workflow tracking
@v0→ v0.17.0), the review comment rendered Formatting: undefined/10, Overall: NaN/10, and Verdict: FAIL — while the review text itself was positive ("accurate, fluent, and well-formatted") and Diff Quality scored 10/10. The diff contained only three URL swaps inside code cells, so a FAIL verdict was clearly wrong.Root cause
evaluateTranslationconsumes the model's JSON without validating the four criterion scores (src/reviewer.ts:816-821):When the model omits one criterion (here
formatting),undefined * 0.15yieldsNaN, which propagates throughoverallScore = translationQuality.score * 0.7 + diffQuality.score * 0.3(src/reviewer.ts:667) and then falls through both verdict thresholds (src/reviewer.ts:669-675) —NaN >= 8andNaN >= 6are both false — landing on FAIL.So a partially-formed model response silently converts into the harshest verdict, and the report renders
undefined/10andNaN/10rather than failing loudly.Suggested fix
Validate the score object after
callWithRetryand retry when any of the four criteria is missing or non-numeric, mirroring the existing max_tokens-truncation retry (src/reviewer.ts:337). As a belt-and-braces fallback, either renormalise the weights over the criteria that are present or throw rather than letting NaN reach the verdict comparison — an incomplete review should be an error or a retry, never an automatic FAIL.Impact
Low frequency but misleading: a FAIL verdict on a clean sync PR sends a human to look for translation defects that do not exist, and erodes trust in PASS/FAIL as a merge signal. Found while reviewing the programming-series sync PRs across editions.