diff --git a/.dev/ARCHITECTURE.md b/.dev/ARCHITECTURE.md new file mode 100644 index 00000000..7c387251 --- /dev/null +++ b/.dev/ARCHITECTURE.md @@ -0,0 +1,216 @@ +# ARCHITECTURE: assessment, open questions, decisions + +**Created**: 2026-07-05 (from the deep technical review). Living document — iterate on the +open questions here *before* anything becomes a PLAN.md phase; when a question settles, +remove it here and record the outcome as a file in [decisions/](decisions/). + +--- + +## 1. System snapshot (current state) + +Two surfaces share one translation engine: + +- **GitHub Action** (`src/`, bundled to `dist-action/`): `sync` (merged source PR → section + diff → translate changed sections → PR in target repo), `review` (AI quality score on + translation PRs), `rebase` (regenerate sibling translation PRs after one merges, with a + section-level translation cache). +- **`translate` CLI** (`src/cli/`): repo lifecycle — `init` (bulk bootstrap), `forward` + (whole-file RESYNC drift recovery), `backward` (find target-side improvements worth + backporting), `review` (interactive ink UI → GitHub issues), plus `status`/`doctor`/ + `headingmap`/`setup` diagnostics and scaffolding. + +**The pipeline hangs off heading text.** `MystParser` splits on `^#{2,6}` regexes; sections +are identified by heading-text slugs; the diff detector, target matching, heading-maps +(`translation:` frontmatter), and the rebase cache all key on those slugs, with positional +fallback when lookups miss. + +**State lives in three uncoordinated channels**: `translation:` frontmatter heading-maps +(per file), `.translate/` YAML state (per target repo), and a machine-readable +`translation-sync-metadata` JSON block in PR bodies (sole input to rebase mode). + +**Deployment topology**: one English source repo per lecture series; one target repo per +(series × language) (`lecture-python-programming` → `.zh-cn`, `.fa`); the action runs in the +source repo (sync) and target repos (review, rebase) with a cross-repo PAT. Production: +zh-cn + fa; ml in flight (PR #71), fr/ja glossaries drafted (#68/#69). + +Known structural weaknesses (evidence in the 2026-07-05 review; fixes tracked in PLAN.md): +no fence awareness in the parser, no parse→reconstruct guarantee, a validation gate that +cannot fire, truncation undetectable, PR-body state human-editable, duplicated +retry/parse/section logic drifting across five sites. + +--- + +## 2. Recommendations (R1–R7) + +### R1. Fence-aware tokenizer now; mystmd AST later; round-trip invariant forever +The recurring one-construct-at-a-time bug family (#5, #6/#40, #49, #50/#54, #65) is the +empirical proof that regex line-parsing of MyST is the wrong foundation. Sequence: +(a) PLAN Phase 2 — fence tracking + anchor ownership + `reconstruct(parse(doc)) === doc` +test; (b) evaluate adopting **mystmd's TypeScript parser** for splitting/validation — same +language as this codebase, real AST, and the same dialect as the future builder (see Q2). +Constructs spanning section boundaries (`exercise-start`/`end`) need pair-integrity checks +regardless of parser. + +### R2. Consolidate state into `.translate/`; demote the PR body to a summary +The PR-body channel is the weakest state store (64 KB cap, human-editable — already broke +review mode and needed a CRLF shim; goes stale after rebase force-pushes) yet it is the +*sole* input to rebase. +Move rebase-relevant state (source SHAs, per-file types, target base SHA, cache hints) into +`.translate/` **on the PR branch** — bot-controlled, size-unbounded, versioned with the +content it describes. Keep the PR-body block as a small versioned *read-only summary* for +external consumers (issue #66) — freeze the contract only after this split (see FUTURE.md +idea 3). + +### R3. One core; one LLM client; one parser +Layering is inverted in one spot: the action imports from `src/cli/` +(`translate-state`, CLI types) — state is core domain. Extract `src/core/` (or flatten): +state module, one shared Claude-call helper (retry + `stop_reason` + JSON extraction — +replaces 5 drifting copies), one section parser (delete `reviewer.ts`'s private one). +The drift is not hypothetical: CRLF, pagination, and `overloaded`-retry fixes each landed in +one copy and missed the others. Tracked as PLAN Phase 6. + +### R4. Give state a target-content identity +`.translate/state` records only `source-sha`, so "has the target changed since sync?" and +"did the sync PR merge?" are inexpressible — the direct cause of the backward-skip and +forward-state bugs (PLAN Phase 3). Add `target-sha`/content hash and a lifecycle status. +This is also the substrate auto-merge and the reviewer web app need. + +### R5. Heading identity, not just heading matching +A reworded heading is indistinguishable from delete+add, which re-translates from scratch and +**discards accumulated human refinements** — the project's most expensive asset. Ascending +options: fuzzy candidate-pairing on delete+add pairs; using `(label)=` anchors as stable +section IDs where present (QuantEcon lectures are label-rich — and this pairs naturally with +R1's anchor-ownership fix); path-independent heading-map keys so a parent rename doesn't +invalidate every descendant. + +### R6. Decompose `index.ts`; type the GitHub layer; atomic commits +`index.ts` is entrypoint + content-fetch layer + entire rebase pipeline (1,095 lines, 0% +coverage, unreachable from tests/CLI). `octokit: any` everywhere forfeited the typing that +would have flagged the pagination bugs. Per-file `createOrUpdateFileContents` makes N commits +per PR and creates SHA races — a single tree+commit via the Git Data API is atomic and faster. + +### R7. Prompts as versioned artifacts +Policy text is duplicated verbatim across six prompt sites with hand-maintained (already +colliding) rule numbering; language rules split between code and glossary JSON; no snapshot +tests, so prompt drift is invisible in review. Centralize fragments, snapshot-test assembled +prompts, and version them so quality changes are attributable. Prompt caching (FUTURE.md +idea 7) falls out of the same restructuring: static prefix first, `cache_control` breakpoint. + +--- + +## 3. Open questions + +### Q1. Repo topology: separate repos per language vs mono-repo + +**Context.** Today: one repo per (series × language). Under consideration elsewhere in +QuantEcon: reorganizing the English lectures into a central mono-repo. Question: should +translations also consolidate (languages as folders in the source repo), or does the +separate-repo design remain right — especially looking ahead to the mystmd builder? + +**What the current design buys (forces for separate repos):** +- **Per-language community autonomy.** Native-speaker reviewers get real repo permissions, + their own issue tracker and review queue. GitHub permissions are repo-granular; in a + mono-repo this degrades to CODEOWNERS + branch protection, and every language shares one + PR/notification firehose with English editorial work. +- **CI and deploy isolation.** Lecture builds execute code and are slow/heavy + (jupyter-book/sphinx today). Per-language repos build and deploy independently; a mono-repo + multiplies build matrix and queue contention on every merge, and language sites deploy + 1:1 from their repo today (Pages/domains). +- **Machine-PR noise containment.** Sync generates high-volume bot PRs; the #63 conflict + storms stayed contained in target repos rather than polluting the source repo. +- **Repo size**: content × N languages plus notebooks/images in one repo. + +**What the boundary costs (forces for consolidation):** +- Nearly every hard engineering problem in this codebase is a **compensation for the repo + boundary**: cross-repo state channels (PR-body metadata, R2), heading-maps as cross-repo + correspondence tables, rebase mode itself (#63), cross-repo PATs and the machine-user + question (#61), the rebase trust boundary, `setup`'s workflow scaffolding. Same-repo + translations would make a translation update an ordinary PR — atomic with its source + change if desired, no PAT, no body-embedded state. +- Cross-cutting refactors (file renames, TOC restructures) currently require N coordinated + PRs; drift between repos is structural (that's what `status`/`forward` exist to repair). + +**Options considered:** +- **A. Status quo** — repo per (series × language). +- **B. Per-series mono-repo** — languages as folders (`lectures/`, `lectures.zh-cn/`, …) in + each source repo. Maximal atomicity; worst CI/permissions/noise trade-offs. +- **C. One translations mono-repo** — all languages, separate from English source. Single + PAT target and state home, English editorial stays clean; still cross-repo, still + conflict-prone, and mixes language communities with each other. +- **D. Central English mono-repo + one repo per language** (all series inside each) — the + natural companion to the planned lectures mono-repo. Repo count drops from series×languages + to languages; sync fan-out becomes 1 source → N language repos; per-language community, CI, + and deployment isolation all survive. + +**Current position (2026-07-05).** The separate-repo boundary is carried by two forces that +are *not* engineering conveniences: per-language community autonomy and heavy executed +builds. Both remain real under mystmd (builds get faster, but execution and per-language +deployment remain). So: **keep the language-repo boundary; evolve toward D as the English +mono-repo lands.** Meanwhile, invest in making sync state topology-agnostic (R2 + R4: state +travels in `.translate/` next to the content, not in PR bodies or workflow config) — that +makes the action indifferent to where the target lives, keeps a future consolidation cheap, +and pays off immediately regardless. + +**Revisit triggers**: mystmd grows a first-class i18n/multi-language site story; review +latency stays the bottleneck even after auto-merge (FUTURE idea 1) — the strongest argument +for same-repo translations is eliminating the PR-merge round-trip entirely; per-language +repo count exceeding ops capacity (secrets rotation, workflow version pinning across ~N×6 +repos); or the D migration itself (which forces the sync config to become a language matrix — +design that config once, for D). + +### Q2. Builder migration: `jupyter-book<2` + `quantecon-book-theme` → `mystmd` + `quantecon-theme.mystmd` + +**Principle**: the action translates MyST *source*, not built output — so it is largely +builder-agnostic by design. The exceptions are exactly where it will break: + +- **TOC**: mystmd replaces `_toc.yml` with `myst.yml` (project `toc:`). The action hardcodes + `_toc.yml` (`classifyChangedFiles`) and the dead `toc-file` input doesn't help; the CLI's + `init` (`parseTocLectures`) and `setup` workflow `paths` filters parse/point at `_toc.yml` + too. Migration work: support both TOC formats behind one abstraction, auto-detected. + (PLAN Phase 4 wires the input; the abstraction belongs to the migration.) +- **Frontmatter**: the `translation:` block (title + heading-map) rides in page frontmatter. + mystmd *validates* frontmatter and warns on unknown keys — verify the key survives cleanly + (or is configurable) on a pilot repo. If it's noisy, that **accelerates R2**: move + heading-maps into `.translate/` and out of the published source entirely (the issue-#3 + design fork, revisited). +- **Dialect**: heading/fence/math/code-cell syntax is shared, so the parser keeps working; + but sphinx-era constructs in lecture content (`{tableofcontents}`, sphinx-only roles, + substitutions) will be rewritten during migration — expect a wave of "new construct" + parser edge cases (R1's round-trip test is the safety net; add fixtures from the first + migrated repo). +- **Opportunities**: mystmd is **TypeScript** — its parser can back R1 (real AST, same + dialect as the builder) and the structural lint (FUTURE idea 4) could literally be + `mystmd` parse + custom rules; mystmd's link/xref checking would have caught the #65 + anchor damage at build time in target repos. The docs site already runs mystmd, so team + familiarity exists. + +**Suggested sequencing**: don't couple the action to either builder; land R1 + TOC +abstraction first; pilot sync on the first mystmd-migrated series before migrating the +production language repos. + +### Q3. Whole-file vs section-by-section forward translation (carried from the 2026-03 plan, "Phase 9") + +Backward Stage 2's move to whole-file evaluation gave ~6× fewer API calls *and* better +results (182→32 calls on a 51-file repo; more high-confidence findings, less noise — see +decisions/D-2026-03-05-whole-file-backward-eval.md and `experiments/forward/`; full plan in git history: +`dev-notes/PLAN.md`). Should forward sync (`translator.ts`) follow? + +- **For**: cross-section terminology consistency; fewer calls; no section-reconstruction + bug class (a large slice of PLAN Phase 2 exists because of split/rejoin). +- **Against**: loses section-level caching (UPDATE mode re-translates everything on any + change — cost and *churn*: unchanged sections get retranslated, discarding human edits, + which violates the prime directive of preserving target refinements); all-or-nothing error + recovery; token limits on long lectures (the truncation findings make this worse, not + better). +- **Likely landing zone (hybrid)**: whole-file for `init`/NEW files and CLI `forward` RESYNC + (already whole-file); section-based for UPDATE where preserving unchanged target content + is the point. Prompt caching (FUTURE idea 7) weakens the cost argument for whole-file; + R5 (stable section identity) weakens the bug-class argument. Decide after Phase 2 ships + and with real cost numbers. + +--- + +## 4. Decisions + +Settled questions are recorded as one file each in [decisions/](decisions/) (append-only). When +a question above closes, remove it from this file and add the decision file. diff --git a/.dev/FUTURE.md b/.dev/FUTURE.md new file mode 100644 index 00000000..8579a3ec --- /dev/null +++ b/.dev/FUTURE.md @@ -0,0 +1,342 @@ +# FUTURE: Feature ideas + +**Created**: 2026-07-05. Each idea is documented well enough to revisit and iterate on later. +When an idea is scheduled for building, expand it into phases and move it to +[PLAN.md](PLAN.md); if rejected, record why in a [decisions/](decisions/) file and delete the +section here. + +Template per idea: **Status / Summary / Motivation & evidence / Design sketch / +Open questions / Effort / References**. + +--- + +## 1. Auto-merge with quality threshold + editor digest + +**Status**: Designed (spec below is the surviving Stage 2 of the issue-#63 fix); prerequisite +(rebase-on-merge) shipped in v0.15.0 and is deployed. + +**Summary**: Automatically merge translation PRs whose review score clears a configurable +threshold, and replace per-PR review with a periodic digest issue that editors audit. + +**Motivation & evidence**: Review latency is the program's binding constraint — it *caused* +the #63 conflict pile-ups (62% of PRs conflicted on lecture-python-programming.fa), and the +#63 analysis explicitly recommended auto-merge + digest as the workflow fix; only the +conflict-mechanics half (rebase-on-merge) was built. Most translation PRs score well on +review (~80–90% per the #63 discussion). Without rebase-on-merge auto-merge was a race +condition; with it shipped, auto-merge is now safe and purely a productivity feature. + +**Design sketch** (from the #63 design record — git history: `dev-notes/FIX-ISSUE-63.md`; +decision context: decisions/D-2026-04-01-rebase-on-merge.md): + +- New sync-mode inputs, off by default: + `auto-merge: true`, `auto-merge-quality-threshold: 9` (1–10), + `auto-merge-labels: "auto-merged"`, `auto-merge-digest: weekly|monthly|none`, + `auto-merge-digest-assignees: ` +- Flow: sync creates PR → review mode runs automatically → score ≥ threshold **and** no + structural issues → approve + merge + label `auto-merged`; below threshold → stays open, + label `needs-review` +- Digest: a scheduled `mode: digest` run opens a periodic issue — count of auto-merged PRs, + table of PR / source PR / files / scores / reviewer warnings, assigned to editors, + labelled `translation-digest` +- Safeguards: opt-in per repo, conservative default threshold, structural-issue veto + regardless of score, label tracking for audit, digest assignees for accountability + +**Open questions**: Should the structural veto be the Phase-2 `validateMyST`/myst-lint result +(strongly suggested — don't auto-merge what a deterministic check can't pass)? Does auto-merge +need the machine-user identity (#61) first so merges aren't attributed to a person? Threshold +calibration — use the 24-PR human evaluation set (#4) as the baseline? + +**Effort**: M (auto-merge) + S (digest mode). + +**References**: issue #63 (analysis + recommendation), decisions/D-2026-04-01-rebase-on-merge.md, +PLAN.md Phase 4 (review-mode fixes are prerequisites — per-file evaluation, NaN guards). + +--- + +## 2. Per-language model & translation-policy configuration + +**Status**: Forced by the language scale-out; partially designed in issue #70. + +**Summary**: Let each target language specify its own model (e.g. Opus for low-resource +pairs), glossary policy (translate / transliterate / keep-English), and prompt rules — +instead of today's single global `claude-model` input and uniform glossary semantics. + +**Motivation & evidence**: en→ml is a genuine low-resource generation cliff (issue #70: +GPT-4 chrF 28.4 vs specialist NMT ~66); the native-reviewer decision was a +**keep-English-dominant** policy carried by `language-config.ts` prompt rules, with a +per-language Opus default named as a follow-up in PR #71. Meanwhile fr (#68) and ja (#69) +glossaries are blocked partly because a glossary alone doesn't enable a language — the +`LANGUAGE_CONFIGS` entry is the real switch, which is exactly the kind of per-language +config this idea makes first-class. `VALID_MODEL_PATTERNS` staleness (PLAN Phase 4) is the +same problem from another angle. + +**Design sketch**: +- Extend `LanguageConfig` with optional `defaultModel`, and resolution order: + action input `claude-model` (explicit) > language default > global default +- Add `claude-opus-4-8`/current-generation IDs to the allowlist or drop pattern validation +- Per-language glossary policy field (the deferred `treatment` schema from #70) only if the + ml calibration shows prompt rules alone are insufficient — v1 stays zero-schema-change +- Document per-language cost implications (Opus vs Sonnet) in the language-config docs page + +**Open questions**: Is model choice per-language or per-(language × mode) — review could stay +on a cheaper model than translation? Where does the reviewer's model come from (today it's +hardcoded separately, `src/reviewer.ts:28`)? + +**Effort**: S–M. + +**References**: issues #70, PR #71 follow-ups, PRs #68/#69 Copilot comments, +memory: ml keep-English policy. + +--- + +## 3. `translation-sync-metadata` as a versioned public contract (issue #66) + +**Status**: Requested, undesigned beyond the issue; internal consumer (rebase mode) already +depends on it. + +**Summary**: Promote the machine-readable JSON block in translation-PR bodies to a stable, +versioned, documented contract that downstream tooling can build on. + +**Motivation & evidence**: QuantEcon/action-weekly-report (QuantEcon/meta#313) wants to roll +up "N upstream changes → M languages" from these blocks. The schema already exists +(`src/pr-creator.ts:44-54`) but has no `schemaVersion`, no docs page, and ad-hoc optional-field +evolution (`targetBaseSha` backfill, `type` defaulting). + +**Design sketch**: add `schemaVersion: 1`; write `docs/developer/translation-sync-metadata.md` +(fields, invariants, the `action-translation` label and `translation-sync-` branch prefix as +stable identifiers, breaking-change policy = bump major); keep parser tolerant of unknown +fields. + +**Open questions**: Settle **ARCHITECTURE.md Q3 first** — if rebase state moves into +`.translate/` on the PR branch, the PR-body block becomes a *read-only summary* for external +consumers, which is a much safer thing to freeze. Also sequence after the rebase-mode +input-validation hardening (PLAN Phase 1.5) so the documented contract rests on validated +inputs. + +**Effort**: S. + +**References**: issue #66, the #63 design record for the original schema (git history: +`dev-notes/FIX-ISSUE-63.md`), ARCHITECTURE.md R2/Q3. + +--- + +## 4. Deterministic MyST structural lint + +**Status**: Repeatedly promised (issues #4, #5 → QuantEcon/meta#268), never built. +PLAN Phase 2 implements the minimal in-process version; this idea is the full tool. + +**Summary**: A deterministic (no-LLM) structural checker for translated MyST documents, +run between Claude output and commit, and available standalone (`translate lint`). + +**Motivation & evidence**: The recurring bug family — one unseen MyST construct at a time +getting silently mangled (#5 malformed headings, #6/#40 pre-title anchors, #49 CJK spacing, +#50/#54 roles in heading-maps, #65 dropped anchors) — plus the review finding that +`validateMyST` validates nothing. Every one of these would have been a loud failure with a +structural gate. + +**Design sketch**: checks = balanced code fences; `$$` pairing; directive open/close pairing +(`exercise-start`/`exercise-end` etc.); label-anchor set equality between source and +translation; heading count/level shape vs source; code-cell count equality; frontmatter +schema. Emit machine-readable findings (reuse the CLI report shapes). Wire as: (a) the sync +pipeline gate, (b) the structural veto for auto-merge (idea 1), (c) a CLI command for target +repos' CI. + +**Open questions**: Build on **mystmd's own TypeScript parser** rather than our regex layer? +mystmd is TS/JS, gives a real AST, and matches the future builder (ARCHITECTURE.md Q2) — +strong synergy, but adds a dependency and pins us to its dialect. Should target repos run it +in their own CI (catching hand-edit breakage too)? + +**Effort**: M standalone; S if PLAN Phase 2's checks are just exposed as a command. + +**References**: QuantEcon/meta#268, issues #4/#5/#65, ARCHITECTURE.md R1/Q2. + +--- + +## 5. Correction-capture review CLI (issue #55) + +**Status**: Proposed with real reviewer evidence; no design decisions yet. + +**Summary**: `translate review` extension (or sibling command) for interactive accept/edit/flag +review of the three element types AI translation most often gets wrong — code comments, +figure/axis labels, `\text{}` inside math — writing corrections back to the target and feeding +flagged terms into the glossary. + +**Motivation & evidence**: Originated from HumphreyYang's reviewer findings; these elements +are invisible in prose-level review but break rendered lectures. Captured corrections are +also the raw material for the feedback loop (idea 6 shares this goal for non-developers). + +**Design sketch**: extract reviewable elements per file (parser already isolates code cells +and math); walk them in the existing ink review UI; on accept/edit, patch the target file; +on flag, append to a glossary-candidates file (draft-then-native-review workflow, as used for +ml/fr/ja). Reuse `review-session.ts` state machine. + +**Open questions**: scope — target-repo-local edits vs PRs? How do corrections feed prompts +(few-shot examples per language?) vs glossary terms? Priority relative to idea 6 (recommend: +this first — smaller, developer-audience, same data model). + +**Effort**: L. + +**References**: issue #55, issue #4 (reviewer findings), memory: glossary draft-then-review +workflow. + +--- + +## 6. Reviewer web app — side-by-side annotation (issue #56) + +**Status**: Idea + companion RA-project sketch; MVP unscoped. + +**Summary**: Dual-pane source/target web editor for **non-developer** native-speaker +reviewers: annotate, correct, and submit — the app turns submissions into PRs via the GitHub +API and logs structured corrections as a training/eval signal. + +**Motivation & evidence**: Per-language native reviewers are the established pattern +(HumphreyYang for zh-cn/fa; Adisankar for ml), but they currently need GitHub + local +tooling. Review capacity is the bottleneck (see idea 1); lowering the barrier for reviewers +scales it. The RA-projects design (docs/projects + memory) already frames this as a +gamified data-collection opportunity. + +**Design sketch (MVP)**: read-only paired rendering first (source | target, section-aligned +via heading-maps); then inline target editing producing a single PR per session; corrections +logged as `{file, section, before, after, category}` JSONL. Auth via GitHub OAuth; +static-hosted SPA + minimal API. Defer: scoring, leaderboards, prompt feedback. + +**Open questions**: hosting/ownership (QuantEcon org infra?); does it read heading-maps from +`.translate/` (argues for ARCHITECTURE Q3 consolidation); relationship to idea 5's data model +(should share the corrections schema). + +**Effort**: XL — treat as an RA project with its own plan. + +**References**: issue #56, memory: RA projects design, issue #55. + +--- + +## 7. Prompt caching + real token counting + +**Status**: Unexplored; pure cost/latency win. + +**Summary**: Restructure prompts so the static prefix (system rules + language rules + +glossary — easily several thousand tokens) carries a `cache_control` breakpoint, and replace +the chars/4 heuristic with the `count_tokens` API. + +**Motivation & evidence**: A sync run issues dozens of sequential calls per language with an +identical prefix; cached input tokens are ~10× cheaper. `estimateOutputTokens` +(`src/translator.ts:40-64`) guesses chars/4 with a 2000-token buffer and drives both the +32768 "API maximum" rejection and `max_tokens` sizing — miscalibration causes both H2-class +truncation and false "document too large" rejections. + +**Design sketch**: order prompts static-first (rules + glossary, then document content); +add `cache_control: {type: 'ephemeral'}` on the static block; measure hit rates in action +logs; use `count_tokens` before full-document calls to pick `max_tokens` and split decisions. +Do after PLAN Phase 6's shared client exists (one place to implement). + +**Open questions**: none blocking — measure and ship. + +**Effort**: S (after Phase 6). + +**References**: PLAN Phase 6, review finding on `checkDocumentSize` (PLAN Phase 4). + +--- + +## 8. Config-driven CLI defaults from `.translate/config.yml` + +**Status**: Half-built; the config file exists precisely for this and is mostly unread. + +**Summary**: Make `translate` commands resolve `target-language`, `docs-folder`, glossary +path, and source repo from the target repo's `.translate/config.yml`, so per-repo flags +become unnecessary. + +**Motivation & evidence**: `config.yml` was designed "so CLI flags don't need to be repeated" +(`src/cli/types.ts:289-297`), but only `source-language` is resolved from it; `-l` silently +defaults to `zh-cn` and `-d` to `lectures` — a live trap for the new ml/fr/ja repos where a +forgotten flag targets the wrong language. + +**Design sketch**: resolution order flag > config > error-if-ambiguous (not silent default); +`translate doctor` validates config completeness; `setup`/`init` write complete configs. + +**Effort**: S. + +**References**: review CLI findings; PLAN Phase 3 (state semantics). + +--- + +## 9. Cross-model (GPT) second reviewer (issue #2 remainder) + +**Status**: Mostly superseded — Claude review mode shipped v0.7.0; the unbuilt remainder is +specifically a *second, non-Anthropic* opinion. + +**Summary**: Optional second review comment from a non-Claude model on translation PRs, as an +independent-perspective check on the Claude reviewer. + +**Motivation & evidence**: mmcky's GPT-5 evaluation matrix over 16 test PRs was useful +(issue #2 discussion); an independent model catches shared-blind-spot errors, relevant once +auto-merge (idea 1) raises the stakes of a single reviewer. + +**Design sketch**: `reviewer-model-2` input + second API key; post as a separate comment or a +combined table; disagreement above a delta flags `needs-review` even if the primary score +passes. + +**Open questions**: worth a second API dependency and secret in every workflow? Cheaper +alternative: two diverse Claude prompts/models (e.g. Opus adversarial pass). Decide after +auto-merge lands and real disagreement data exists. + +**Effort**: M. + +**References**: issue #2. + +--- + +## 10. Scheduled backward analysis (carried from previous plan, "Phase 8") + +**Status**: Designed at task level in the 2026-03 plan (git history: `dev-notes/PLAN.md`); +unscheduled. + +**Summary**: Monthly GitHub Actions workflow running `translate backward` + `status` per +target repo, storing the report as an artifact and notifying maintainers (tracking-issue +comment or Slack); maintainers run `translate review` locally on the downloaded report. + +**Motivation & evidence**: Backward analysis only has value if it runs; today it requires a +maintainer remembering to run it. Auto-PR creation was deliberately scoped out (human review +via `review` stays). + +**Effort**: S — blocked on PLAN Phase 3 (backward's skip predicate is currently inverted, so +scheduled runs would silently analyze nothing). + +**References**: 2026-03 plan Phase 8 (git history: `dev-notes/PLAN.md`); PLAN Phase 3. + +--- + +## 11. Python/`rich` CLI rewrite (carried from previous plan) + +**Status**: Documented fallback, explicitly conditional — not planned. + +**Summary**: If ink rendering proves insufficient for MyST review (math, directives, +side-by-side diffs), port the entire CLI (~3,600 lines) to Python with `rich`/`textual`, +publish to PyPI; the action stays Node. + +**Trigger conditions** (from the original write-up): ink rendering gaps actually blocking +reviewers; a team decision to maintain the CLI in Python long-term; stable CLI interfaces and +JSON schemas (now true). Revisit only if idea 5's richer review UI hits ink's limits. + +**Effort**: XL. + +**References**: 2026-03 plan, "Future: Python Rewrite with rich" (git history: +`dev-notes/PLAN.md`; includes the module inventory). Decision context: +decisions/D-2026-03-04-ink-over-rich-cli.md. + +--- + +## 12. Smaller carried-forward backlogs + +Kept as one section; promote items individually if they become real. + +- **Review-command UX polish** (2026-03 plan): scroll viewport; truncate long before/after + blocks with expand; syntax highlighting (cli-highlight); MyST-aware card rendering; + word-level inline diff. +- **Prompt-tuning pass** (2026-03 plan): Stage-1 triage precision (flagging rate ~67% vs + target 5–10% — high recall, poor precision); Stage-2 noise reduction; RESYNC preservation + quality; re-run the validation set after each change. Pairs with prompt versioning + (ARCHITECTURE.md R7) so tuning is measurable. +- **Error-handling hardening** (2026-03 plan): missing source/target files; API timeout/rate + limit; invalid heading-map; oversized documents; graceful degradation with warnings. +- **Digest of dropped-anchor damage** (issue #65 follow-up): one-off scan of past translation + output for silently dropped `(label)=` anchors across zh-cn/fa repos. diff --git a/.dev/PLAN.md b/.dev/PLAN.md new file mode 100644 index 00000000..41f0b466 --- /dev/null +++ b/.dev/PLAN.md @@ -0,0 +1,379 @@ +# PLAN: Maintenance & Hardening + +**Created**: 2026-07-05 (from the deep technical review of all source, tests, CI, docs, issues, and PRs) +**Baseline**: v0.15.0 on `main`; PR #71 (Malayalam) open as draft +**Predecessor**: the 2026-02→03 resync-CLI plan — complete; distilled into +[decisions/](decisions/), full text in git history (`dev-notes/PLAN.md`) + +How to use this plan: work phases in order — each phase is independently shippable and ends +with a release or a verifiable checkpoint. Tick tasks as they land; when the whole plan +completes, distill outcomes into [decisions/](decisions/) and start the next plan fresh +(git keeps the history). Feature work lives in [FUTURE.md](FUTURE.md); design questions in +[ARCHITECTURE.md](ARCHITECTURE.md). + +Severity tags: **[H]** produces wrong output or breaks a workflow, **[M]** wrong under +realistic conditions, **[L]** quality/robustness. + +--- + +## Phase 1 — Patch release v0.15.1 (small, high-confidence fixes) + +Everything here is a contained fix with an obvious correct behaviour. Ship as one patch release. + +### 1.1 GitHub API pagination +- [ ] **[H]** `runSync` truncates PRs with >30 changed files — add `octokit.paginate` to + `pulls.listFiles` in `src/index.ts:573` (the rebase path at `src/index.ts:153` already + does this correctly; copy that pattern) +- [ ] **[H]** Same unpaginated `listFiles` in review mode: `src/reviewer.ts:448` and `src/reviewer.ts:368` +- [ ] **[L]** `pulls.list` capped at 100 in rebase (`src/index.ts:159`) — paginate so sibling + PRs beyond 100 can be rebased +- [ ] **[L]** `issues.listComments` unpaginated in `postReviewComment` (`src/reviewer.ts:1028`) — + with >30 comments the existing review comment isn't found and duplicates accumulate + +### 1.2 Small correctness fixes (action) +- [ ] **[H]** CRLF: `parseSourcePRNumber` requires `\n` (`src/reviewer.ts:340`) — GitHub + normalizes edited PR bodies to `\r\n`, permanently breaking review mode for that PR. + Use `\r?\n` (as `src/pr-creator.ts:417` already does) +- [ ] **[M]** `runRebase` posts "♻️ Automatically rebased" and counts success even when the + early returns at `src/index.ts:375` and `src/index.ts:447` mean nothing was pushed — + return a status and only comment/count on an actual push +- [ ] **[M]** Glossary terms missing the target language render as `"term" → "undefined"` in + prompts (`src/translator.ts:664`) — skip such terms (and log) +- [ ] **[L]** `\translate-resync zh` (unsupported code) proceeds with **all** languages + (`src/inputs.ts:281`) — fail closed with an explanatory comment instead +- [ ] **[M]** Primary sync path uses `context.sha` (`src/index.ts:544`); for `pull_request: + closed` events this can be a stale synthetic merge-ref SHA. Use `pr.merge_commit_sha` + as the resync path already does (`src/index.ts:548-563`) +- [ ] **[L]** Guard `response.content[0]` before reading `.text` (`src/translator.ts:279` et al., + `src/reviewer.ts:297`) — empty/refusal responses currently throw a bare `TypeError` + +### 1.3 Truncation detection (minimal version) +- [ ] **[H]** Check `response.stop_reason` after every Anthropic call and **fail the file** on + `max_tokens` instead of committing truncated output. Sites: `src/translator.ts:279, 361, + 433, 512, 622`; `src/reviewer.ts:755, 863`; CLI: `src/cli/backward-evaluator.ts:594`, + `src/cli/document-comparator.ts:246`, `src/cli/forward-triage.ts:237`. + (The full shared-client refactor is Phase 6; this is the per-site guard.) + +### 1.4 Dependencies & packaging +- [ ] **[H]** `npm audit fix` — clears the two high-severity `ws` advisories (via `ink`, CLI-only) +- [ ] **[H]** Bump `@actions/core` (1.11 → 3.x) and `@actions/github` (6 → 9.x) and rebuild — + the committed `dist-action/index.js` currently **ships a vulnerable `undici`** +- [ ] Bump `@anthropic-ai/sdk` (0.78 → current 0.110.x) and re-run the retry tests +- [ ] Fix `package-lock.json` self-version still saying `0.8.0` (lines 3, 9) +- [ ] Add an `engines` field to `package.json` (node >= 20 now; see Phase 5.6 for node24) +- [ ] Fix stale `.gitignore:6` comment ("ncc CJS bundle" — the build is esbuild) + +### 1.5 Rebase-mode input validation (security — pulled forward from Phase 4) +- [ ] **[H]** Harden rebase mode's handling of PR-embedded metadata: cross-check it against the + workflow's own configuration and require the expected automation identity before acting. + Specifics deliberately omitted here per the `.dev/` public-content rule. Land this before + issue #66 documents the metadata as a public contract. + +### 1.6 Release chores +- [ ] Fix `CHANGELOG.md:10` date: `[0.15.0] - 2025-07-14` → `2026-04-14` +- [ ] Add `[Unreleased]` entries (Malayalam commit `d5b216e` shipped without one) +- [ ] Release v0.15.1, rebuild `dist-action/`, move the `v0` / `v0.15` tags + +**Done when**: v0.15.1 tagged; `npm audit --omit=dev` reports 0 high/critical; a >30-file test +PR syncs completely. + +--- + +## Phase 2 — Parser & validation correctness (v0.16) + +The silent-corruption class. These interact, so they ship together with the round-trip test as +the gate. See ARCHITECTURE.md R1 for the longer-term parser direction (mystmd AST). + +- [ ] **[H]** Make `parseSections` fence-aware (`src/parser.ts:64-118`): `##` lines inside + ```` ``` ````/`{code-cell}` blocks are currently parsed as section headings — phantom + diffs, code sent to Claude as prose, corrupted cells on reconstruction. Fence tracking + already exists in the same file for pre-title scanning (`src/parser.ts:273-286`) +- [ ] **[H]** Issue **#65** — label-anchor ownership: `(label)=` immediately above `## Heading` + is stored at the tail of the *previous* section (`src/parser.ts:113-117`), so anchors + vanish when the previous section is deleted/skipped (broke the zh-cn build). Anchors must + bind to the following heading and be preserved verbatim through translation +- [ ] **[M]** The anchor-adjacency fixup regex (`src/file-processor.ts:564`) doesn't match + non-ASCII labels — fix alongside #65 +- [ ] **[H]** Replace the no-op `validateMyST` (`src/parser.ts:223-233` — `parseSections` never + throws, so the gates at `src/sync-orchestrator.ts:340, 410` can never fire) with real + structural checks: balanced fences, `$$` pairs, `{exercise-start}`/`{exercise-end}` + pairing, heading-count sanity, anchor preservation. (FUTURE.md "MyST structural lint" is + the fuller version; implement the checks behind one function so it can swap in) +- [ ] **[H]** Add the **round-trip invariant test**: for every fixture, + `reconstruct(parse(doc))` must equal `doc` byte-for-byte for unchanged documents +- [ ] **[M]** `mergeSubsectionsWithTargetTranslations` keeps the *old* target content for every + positional match when Claude's structure mismatches, discarding the fresh translations + from that same call (`src/file-processor.ts:352-399`) — and the run reports success +- [ ] **[M]** Updated sections always get the *old* target heading re-attached + (`src/file-processor.ts:319-326`), so reworded English headings never propagate — translate + and use the new heading (heading-map keyed) +- [ ] **[M]** `updateHeadingMap` doesn't preserve existing mappings (contradicting its docstring) + and its deletion pass is dead code (`src/heading-map.ts:82-166`); the unused `titleHeading` + param should go too +- [ ] **[M]** Position fallback mis-fires when one section is added *and* another deleted + (counts equal but positions shifted) — `src/file-processor.ts:193, 254, 742-749` +- [ ] **[M]** Duplicate heading slugs (two `## Exercises`) corrupt matching in three places: + change attachment (`src/file-processor.ts:184`), deletion detection + (`src/diff-detector.ts:123, 140-148`), and the rebase cache keys + (`src/file-processor.ts:90-91`) — disambiguate IDs (e.g. suffix by occurrence index) +- [ ] **[M]** `${sha}^` old-content fetch is wrong for **rebase-merged** multi-commit PRs + (`src/index.ts:776`, `src/index.ts:314`) — earlier commits' changes are silently treated + as unchanged. Fetch the PR's base SHA instead, or detect and reject rebase-merge events +- [ ] **[L]** `doctor` counts `^## ` inside code fences (`src/cli/commands/doctor.ts:173-177`) — + reuse the fence-aware parser instead of a private regex +- [ ] **[L]** Heading replacement uses `sourceSub.heading.replace(/^(#+\s+).*/,` with an + interpolated string (`src/file-processor.ts:387`) — headings containing `$1`/`$&` corrupt; + use a function replacer +- [ ] **[L]** Prompt rule-numbering collisions when custom instructions are appended + (`src/translator.ts:395, 408-412, 459, 472-480`) — number rules programmatically + +**Done when**: round-trip test green over all fixtures incl. new code-cell-with-`##` and +anchor-before-heading fixtures; #65 closed; a deliberately truncated/malformed model response +fails the file loudly. + +--- + +## Phase 3 — CLI state & lifecycle correctness + +The `.translate/` state model needs one schema addition (target SHA) plus predicate fixes. +See ARCHITECTURE.md R4 for the design rationale. + +- [ ] **[schema]** Record `target-sha` (or a content hash) alongside `source-sha` in + `.translate/state/.yml`, written at the same moments state is written today +- [ ] **[H]** `backward` bulk skips files whose *source* is unchanged + (`src/cli/commands/backward.ts:469-483` via `isSourceChanged`, + `src/cli/translate-state.ts:232-247`) — but backward hunts for *target-side* edits, so it + skips exactly the files it exists to analyze. Skip on "target unchanged since sync" instead +- [ ] **[H]** `forward --github` writes new state into the working tree the moment the PR is + *opened* (`src/cli/commands/forward.ts:228-246`), masking the file as in-sync even if the + PR is closed unmerged, and leaving the repo dirty. Commit state **on the PR branch** so it + lands iff the PR merges +- [ ] **[H]** `forward` bulk filters on primary status only (`src/cli/commands/forward.ts:286-289`); + a file that is OUTDATED **and** MISSING_HEADINGMAP is never resynced (priority ordering at + `src/cli/commands/status.ts:256-257`). Filter on `e.flags.includes('OUTDATED')` +- [ ] **[M]** Auth failures poison `--resume`: errored files enter the done-set + (`src/cli/commands/backward.ts:494-515`), so an expired `ANTHROPIC_API_KEY` marks every + file permanently done. Abort the run on non-retryable auth errors; retry errored files on + resume; validate checkpoints with the already-written-but-unused `ProgressCheckpointSchema` + (`src/cli/schema.ts:192-261`) instead of raw `JSON.parse` (`backward.ts:377-387`) +- [ ] **[M]** `backward` bulk analyzes the source∪target **union** (`backward.ts:417-428`), so + every untranslated file errors ("TARGET file not found") — use the intersection +- [ ] **[M]** `TARGET_HAS_ADDITIONS` files are destructively resynced after only a console + warning (`src/cli/commands/forward.ts:127-135`; the RESYNC prompt orders removal of + non-source content, `src/translator.ts:575`) and then reported as "skipped (i18n only)" + (`forward.ts:359-384`). Gate behind `--force` in bulk mode and report honestly +- [ ] **[M]** `forward --test` mutates real repos — writes `[TEST RESYNC]` content over the + target (`forward.ts:142-146, 223`) and in `--github` mode pushes and opens real PRs. + Test mode must be side-effect-free +- [ ] **[M]** `--json` + `--resume` loses completed reports from the aggregate (sidecar path + mismatch, `backward.ts:339-342` vs `521-532, 727-733`) +- [ ] **[M]** `setup` emits broken templates: hardcoded action version `'0.9.0'` + (`src/cli/commands/setup.ts:304, 351` — use `getToolVersion()`), TOC path in the workflow + `paths` filter is repo-root `_toc.yml` instead of `/_toc.yml` (`setup.ts:139`), + and no `permissions:` block in either generated workflow +- [ ] **[M]** Triage keyword fallback classifies "not in sync" as IN_SYNC + (`src/cli/document-comparator.ts:151-154`) — guard the negation like + `src/cli/forward-triage.ts:139` does +- [ ] **[M]** Cross-repo commit timeline sorted lexically on local-time `%ai` strings + (`src/cli/git-metadata.ts:174-177`) — wrong by up to a day across timezones, and this + ordering exists to prevent LLM directional errors. Sort on epoch +- [ ] **[M]** `init -f cobweb.md` can select `extended_cobweb.md` (`src/cli/commands/init.ts:433-445`) — + exact match must win before substring match +- [ ] **[L]** Non-recursive discovery (`src/cli/commands/status.ts:103-112`) makes nested + lectures invisible to status/backward/forward/headingmap/doctor, though state and init + support nesting — recurse +- [ ] **[L]** Non-atomic state writes (`src/cli/translate-state.ts:118-125, 173-179`) — write + temp + rename; add a `doctor` check for corrupt state YAML (readers currently coerce it + silently to "no state", `translate-state.ts:164-166`) +- [ ] **[L]** Glossary loading is cwd-dependent (`forward.ts:399-418`, `init.ts:72-92`) — resolve + relative to the tool install/repo and add `--glossary` to `forward`; dedupe the two loaders +- [ ] **[L]** `status --write-state` overwrites existing `source-language` config with the CLI + default (`status.ts:350-355`) — status should use `resolveSourceLanguage` like backward/forward +- [ ] **[L]** `-f ../../x.md` path traversal writes state outside the repo + (`translate-state.ts:136-138`) — apply init's guards (`init.ts:269-272`) everywhere +- [ ] **[L]** Validate `--write-state`/`--check-sync` incompatibility before the repo scan + (`status.ts:312-318`), not after + +**Done when**: a bootstrapped repo with a hand-edited target file is *found* by `backward`; +closing a forward PR unmerged leaves status reporting OUTDATED; `forward --test` leaves both +repos byte-identical. + +--- + +## Phase 4 — Security & robustness (action) + +- [ ] **[M]** Refresh (or re-embed) PR-body metadata after a rebase rewrites the branch — + `targetBaseSha` goes stale after the first rebase and degrades cache decisions +- [ ] **[M]** `fetchFileContent` silently returns `""` for files >1 MB (contents API returns + `encoding: "none"`; `'content' in data` is still true, `src/index.ts:743`) — detect and + error (or fetch via blob API) +- [ ] **[M]** `toc-file` input is dead: documented (`action.yml:45-48`), parsed + (`src/inputs.ts:61`), but `classifyChangedFiles` hardcodes `_toc.yml` + (`src/sync-orchestrator.ts:183-206`). Wire it — this also matters for the mystmd migration + (ARCHITECTURE.md Q2) +- [ ] **[M]** Review mode concatenates all files into one blob and pairs sections positionally + across it (`src/reviewer.ts:528-573`, `:214-235`) — any per-file section-count difference + (which sync *deliberately produces* via skipped sections) misaligns everything after it. + Evaluate per file +- [ ] **[M]** Review JSON: raise/handle the 1500-token cap (`src/reviewer.ts:755, 863` — + truncated JSON → 3 identical retries → review fails) and validate numeric fields + (`reviewer.ts:756-761` — missing fields yield `NaN/10` scores and a spurious FAIL) +- [ ] **[L]** `VALID_MODEL_PATTERNS` is stale (`src/inputs.ts:9-20`) — every newer valid model + ID warns; either validate against the Models API or drop the check (interacts with + FUTURE.md "Per-language model configuration") +- [ ] **[L]** `checkDocumentSize` hardcodes 32768 as "API maximum" (`src/translator.ts:70-85, + 501, 609`) and the resync variant passes source+target combined length (`:604`), rejecting + documents at ~half the real threshold — recompute against actual model limits +- [ ] **[L]** Language-code case drift: `validateLanguageCode` lowercases but glossary filename + and term lookup use the raw code (`src/language-config.ts:126`, + `src/sync-orchestrator.ts:112`, `src/translator.ts:666`) — normalize once at input parsing +- [ ] **[L]** Rebase force-push races: stale blob SHA after `git.updateRef` (`src/index.ts:464-473`) + → unretried 409; two near-simultaneous merges rebase the same branch concurrently. Add + retry-on-409 and document the `concurrency` group as required in the workflow template + +**Done when**: review mode scores a 3-file PR with different per-file section counts correctly; +oversized files error instead of reading as empty. + +--- + +## Phase 5 — Test debt & CI + +- [ ] **[H]** Delete or rewrite `src/__tests__/translator.test.ts` — ~30 tests never import + `../translator` (they assert on locally-declared literals, e.g. `expect(true).toBe(true)` + at lines 46/58, re-derived expectations at 276–305). Replace with real tests of prompt + assembly, marker handling, and size limits, following the excellent + `translator-retry.test.ts` pattern +- [ ] Add tests for `src/index.ts` (currently **0% of 1,095 lines**): mode dispatch, event + validation, `runRebase`/`rebaseSinglePR` (incl. the no-op early-return paths), + `fetchAllFileContents`, failure-issue creation — inject a fake octokit (the CLI's + `GhRunner`/`GitRunner` fakes show the pattern, `src/cli/__tests__/forward-pr-creator.test.ts:36-41`) +- [ ] Add tests for the `TranslationReviewer` class (`src/reviewer.ts:255-1061`, 34% coverage) — + verdict computation, NaN handling, comment generation +- [ ] Add octokit-fake tests for `createTranslationPR` (`src/pr-creator.ts:90`) +- [ ] Fix fresh-clone `npm test` (build-dependent `cli-smoke.test.ts:21-27`) — chain a build or + skip-with-notice +- [ ] Make the 6 fixture-gated skips visible (`schema.test.ts:463-482`, `review.test.ts:314-333` + silently skip in CI) — commit minimal fixtures or fail loudly when fixtures are expected +- [ ] CI additions (`.github/workflows/ci.yml`): `prettier --check`, coverage threshold + (statements ≥ 66% to start, ratcheting), `npm audit --omit=dev` gate +- [ ] **node20 → node24** in one change: `action.yml:92` (`using`), `ci.yml:21`, + `build-action.mjs:23` (esbuild target), `@types/node`, `engines` — Node 20 passed EOL + April 2026 +- [ ] Carried from the previous plan (still unchecked there): backward+review workflow test on + `lecture-python-intro` ↔ `lecture-intro.zh-cn`; review → Issue creation end-to-end + (non-dry-run); Stage-1 triage recall validation (≥95%) + +**Done when**: coverage report runs in CI with a threshold; test count reflects only real tests; +action runs on node24. + +--- + +## Phase 6 — Consolidation refactors (kill the duplication that keeps re-creating bugs) + +Evidence this matters: CRLF was fixed in `pr-creator` but not `reviewer`; pagination in rebase +but not sync; `overloaded` retry in the translator but not the reviewer or any CLI copy. + +- [ ] **One shared Claude call helper** (retry + backoff + `retry-after` + `stop_reason` check + + JSON extraction): replaces `translator.callWithRetry` (`src/translator.ts:153-190`), + `reviewer.callWithRetry` (`src/reviewer.ts:281-330`), and the three CLI copies + (`document-comparator.ts:241-286`, `backward-evaluator.ts:277-315, 589-627`, + `forward-triage.ts:232-277`); also stop stacking on the SDK's built-in 2 retries + (worst case today: 9 attempts) +- [ ] **Move `.translate/` state out of `src/cli/`** into core — the action imports backwards + from the CLI today (`src/index.ts:8`, `src/sync-orchestrator.ts:20-21`) (ARCHITECTURE.md R3) +- [ ] **One section parser**: `reviewer.ts` ships its own `extractPreamble`/`extractSections`/ + `headingToId` (`src/reviewer.ts:87-152`) with different rules from `MystParser` — delete it +- [ ] Dedupe: glossary loaders (action `src/index.ts:61-78` vs `src/sync-orchestrator.ts:105-139`; + CLI `forward.ts` vs `init.ts`), `GhRunner`/`GitRunner` (3 copies with incompatible + signatures), the docs-folder normalization blocks (`src/inputs.ts:57, 115, 146`), the + `languageNames` map (`src/reviewer.ts:650-659` vs `language-config.ts`), and the 3 copied + batch-concurrency loops (`backward.ts:611`, `forward.ts:314`, `init.ts:582` — replace the + head-of-line-blocking batcher with a worker pool) +- [ ] Honor the `Logger` abstraction (`src/sync-orchestrator.ts:31-35`) — `FileProcessor`, + `DiffDetector`, `TranslationService` import `@actions/core` directly; `heading-map.ts` + uses bare `console` +- [ ] Type octokit (`ReturnType` — already used at `src/reviewer.ts:257`) + everywhere `octokit: any` appears +- [ ] Extract constants: `max_tokens` values, branch prefix `'translation-sync-'` (4 literals), + label `'test-translation'`, token-estimation factors +- [ ] Dead-code sweep: `evaluateSection` path (`backward-evaluator.ts:34-315`), + `computeSummaryStats`, `ResyncSectionResult` plumbing (`src/cli/types.ts:226-242`), + `matchSections`' ignored `_headingMap` param, `SectionChange.position`, + `MystParser.findSectionByPosition`, `if (!mode)` after required-input read + (`src/inputs.ts:40-43`), diff-detector `_preamble` change (never consumed), + the dead Ctrl+C handler (`src/cli/components/ReviewSession.tsx:69-74`) +- [ ] Decide `localization-rules.ts`: it's never invoked from the action path (translations get + no code-cell localization while prompts insist it be *preserved*) — wire it in or document + the asymmetry +- [ ] Consolidate the two version mechanisms (`getToolVersion` walks vs `createRequire`, + `src/cli/translate-state.ts:37-81` vs `src/cli/index.ts:30-32`) into one build-time constant +- [ ] Break up `processSectionBased` (~420 lines, `src/file-processor.ts:53-473`) — extract the + nested closures so the Phase 2 merge fixes are testable +- [ ] Simplify `parseTranslatedSubsections` wrapper-line arithmetic (`src/file-processor.ts:630-676`) + by parsing the raw fragment directly + +**Done when**: one retry implementation, one section parser, one state module; grep for +`new Anthropic(` finds one construction site. + +--- + +## Phase 7 — Docs & repo currency + +- [ ] **[H]** `examples/README.md`: both sync examples use `secrets.GITHUB_TOKEN`, which cannot + push to a different repo — switch to the PAT pattern (README/quickstart are already + correct); also fix stale `@v0.11` pins, pre-v0.6.3 label defaults, the old project name, + and the unconfigured `ja` example +- [ ] Add the six tutorials (plus `developer/legacy-tools.md`) to `docs/myst.yml` toc — they 404 + on the live site today while `docs/index.md:63-70` links them +- [ ] Refresh `docs/index.md:84-87` (says v0.8.0, 873 tests; contradicts README) +- [ ] **Document rebase mode** in `docs/user/action-reference.md` (inputs, triggers, + `translation-sync-` branch convention, cache behaviour, failure comments) — v0.15.0's + headline feature is absent from the docs site; add rebase troubleshooting to the FAQ +- [ ] Issue **#66**: add `schemaVersion: 1` to the metadata interface (`src/pr-creator.ts:44-54`) + and write the contract docs page (see FUTURE.md idea 3 for the full scope; do after + Phase 4's trust fix) +- [ ] `glossary/README.md`: list `fa.json` and `ml.json`, document the draft-glossary + (`0.1.0-draft`) + native-reviewer workflow, and fix the "translate all terms" recipe that + contradicts the ml keep-English policy; update `docs/user/glossary.md` (phantom top-level + `language` field at line 37; omits ml), `docs/user/language-config.md`, FAQ +- [ ] Retire `src/cli/README.md` (documents the old `resync` binary, 1 of 8 commands) — replace + with a pointer to `docs/user/cli-reference.md` +- [ ] `.github/copilot-instructions.md`: reconcile 1005 vs "1001 tests", add ml +- [ ] Fix `tool-test-action-on-github/test-action-on-github.sh:25` header ("9 PRs" → 26) +- [ ] Document `.translate/` state files appearing in translation PRs (action side) in + `docs/user/action-reference.md` +- [ ] Organization decisions (each small, do deliberately): + - [ ] `experiments/` → `.dev/experiments/` (single historical experiment write-up) + - [ ] `docs/projects/` (internal RA planning, excluded from the site toc) → `.dev/projects/` + or add to the toc deliberately + - [ ] `presentations/`: stop committing generated artifacts (`.pdf`, built `.html`) or archive + the directory; content is stale (issue #7 — old project name) + - [ ] Normalize version-pin style across docs (`@v0` vs `@v0.15` vs `@v0.15.0`) + +**Done when**: the deployed docs site has no broken toc links, documents all three modes, and a +new-language adopter can go from zero to a working target repo following only published docs. + +--- + +## Phase 8 — Issue-tracker gardening & ops + +- [ ] Close **#4** (all six findings fixed in v0.6.1; residual tracked in QuantEcon/meta#268) +- [ ] Close **#6** (implemented + tested; point to #65 for the translator-side remainder) +- [ ] Close **#48** (all four referenced PRs closed unmerged; superseded by #63/#64) +- [ ] Close **#1** (decision "accept, monitor" — recorded in + decisions/D-2025-10-01-accept-llm-translation-improvements.md; close with a pointer) +- [ ] Close **#3** (superseded by #51/#52 `translation:` frontmatter + `.translate/`; fold any + residue into #66) +- [ ] Close or retitle **#2** to the narrow "cross-model (GPT) reviewer" remainder + (FUTURE.md idea 9) +- [ ] Finish **#53** — legacy `heading-map:` fallback removal (`src/heading-map.ts:41`); the + self-imposed v0.15.0 deadline has passed and target repos are migrated +- [ ] **#61** — create the `quantecon-services` PAT, grant repo access, rotate secrets in the + ~6 workflow repos (pure ops; recent translation PRs still author as `mmcky`) +- [ ] **#7** — resolve alongside the `presentations/` decision in Phase 7 +- [ ] For PRs **#68** (fr) / **#69** (ja): before merge, add the missing `LANGUAGE_CONFIGS` + entries (Copilot's catch — a glossary alone does not enable a language), resolve the + flagged term-choice judgment calls with native speakers, rebase whichever lands second + +**Done when**: open-issue list contains only live work; translation PRs author as +`quantecon-services`. diff --git a/.dev/README.md b/.dev/README.md new file mode 100644 index 00000000..5c16bf1d --- /dev/null +++ b/.dev/README.md @@ -0,0 +1,59 @@ +# `.dev/` — project notes + +Working notes for this repository — **state, decisions, and design ideas** — maintained +jointly by humans and coding agents. Nothing here is published (not in the docs site, npm +package, or action bundle). Git holds the project's history; `.dev/` holds the curated, +current picture: distill, supersede, or delete. + +`action-translation` is the **pilot repo** for the QuantEcon `.dev/` convention — +spec and rationale: QuantEcon/QuantEcon.manual#103; pilot: QuantEcon/action-translation#73. + +## Layout + +``` +.dev/ +├── STATE.md # where things stand: in flight / blocked / next (~1 page); +│ # first line "verified: YYYY-MM-DD" +├── PLAN.md # current roadmap (not its history) +├── ARCHITECTURE.md # optional living doc: design deliberation, open questions +├── FUTURE.md # optional living doc: uncommitted feature ideas +├── decisions/ # D-YYYY-MM-DD-.md — one settled decision per file +├── log/ # YYYY-MM-DD-.md — short dated session notes +└── scratch/ # gitignored working files (the repo's scratch location) +``` + +**Lifecycle by location**: anything at the `.dev/` root is *living* (edited in place, always +current); anything under `decisions/` or `log/` is *append-only* (entries never edited; stale +log files may be deleted once distilled — deletion ≠ editing). Settled architecture graduates +from `ARCHITECTURE.md` to contributor docs; committed ideas graduate from `FUTURE.md` to +`PLAN.md`. + +**No YAML schema, no CI gates.** Structure lives in filenames, git, and three plain-text +conventions: + +1. `verified: YYYY-MM-DD` as STATE.md's first line — trust the file less as that date ages. +2. A supersession note at the top of an old decision file, pointing to its replacement + (decision files are otherwise never edited; date+slug filenames avoid id races between + parallel agents). +3. Inline `#promote` tags marking cross-repo findings for the future org knowledge vault — + everything stays one `grep -rn "#promote" .dev/` away. + +Decision files are a few lines each: **context / decision / consequences** (+ refs). + +## Maintenance + +An occasional **"tidy `.dev/`" session** — an agent reads the folder, flags contradictions and +staleness, proposes pruning; a human approves the PR. Run when returning after a gap, or +roughly monthly. Humans curate STATE.md/PLAN.md and approve decisions and pruning; agents +write logs, file decisions, and run tidy passes. If tidy sessions repeatedly surface the same +mechanical problems, that's the evidence to script a check — not before. + +## Content rules + +- `.dev/` is **public**: no credentials, no unpatched-vulnerability specifics (track those in + security advisories until fixed). +- Absolute dates only ("2026-07-05", never "last week") — these files outlive sessions. +- The agent contract lives in [`AGENTS.md`](../AGENTS.md). + +For user-facing and contributor documentation, see [`docs/`](../docs/) and +[`CONTRIBUTING.md`](../CONTRIBUTING.md). diff --git a/.dev/STATE.md b/.dev/STATE.md new file mode 100644 index 00000000..9a5eeea5 --- /dev/null +++ b/.dev/STATE.md @@ -0,0 +1,43 @@ +verified: 2026-07-14 + +# STATE + +Where things stand, ~1 page. Read this first; trust it less as the `verified:` date ages. +Roadmap detail lives in [PLAN.md](PLAN.md), not here. + +## In flight + +- **PR #72** — `.dev/` notes convention pilot (this folder; spec QuantEcon/QuantEcon.manual#103, + pilot QuantEcon/action-translation#73). Seeds `.dev/` from the 2026-07-05 deep technical review. +- **PR #71** — Malayalam (`ml`) language support, draft; awaiting a native-reviewer calibration + batch. Glossary PR **#69** (ja) open, awaiting native review + a `LANGUAGE_CONFIGS` entry. + +## Recently landed + +- **PR #68** (fr) merged 2026-07-14 — French glossary + `LANGUAGE_CONFIGS` entry; `fr` is now + a usable target language (native-speaker review fixes applied pre-merge). + +## Blocked + +- Nothing hard-blocked. Language PRs wait on native-speaker review (external cadence). + +## Next + +- **PLAN Phase 1** — v0.15.1 patch: API pagination, CRLF PR-body parse fix, truncation guards, + rebase-mode input-validation hardening (security — pulled forward), dependency security bumps, + CHANGELOG date fix. See [PLAN.md](PLAN.md). + +## Health & context + +- Released **v0.15.0**; `main` clean. Production targets: `zh-cn`, `fa`. `fr` enabled + (glossary + config landed, no production repo yet); `ml`/`ja` in flight. +- Highest-priority known bug: issue **#65** — translator drops `(label)=` anchors before + headings; broke a zh-cn build once already (PLAN Phase 2). +- Test suite green (~5s); note ~30 tests in `translator.test.ts` don't exercise the module + (PLAN Phase 5). + +## Map + +[PLAN.md](PLAN.md) roadmap · [FUTURE.md](FUTURE.md) feature ideas · +[ARCHITECTURE.md](ARCHITECTURE.md) design questions · [decisions/](decisions/) settled calls · +[log/](log/) session notes · [README.md](README.md) the convention. diff --git a/.dev/decisions/D-2025-10-01-accept-llm-translation-improvements.md b/.dev/decisions/D-2025-10-01-accept-llm-translation-improvements.md new file mode 100644 index 00000000..4e03f206 --- /dev/null +++ b/.dev/decisions/D-2025-10-01-accept-llm-translation-improvements.md @@ -0,0 +1,13 @@ +# LLM improvements to unchanged translations: accept and monitor + +**Context**: During section updates Claude sometimes improves *unchanged* target content +(e.g. fixing the Leontief transliteration) — technically out of scope for the diff. + +**Decision**: Accept such improvements rather than constraining the prompt; monitor. A +strict-preservation flag remains an option if review cost grows (field evidence of that cost +noted in the #63 discussion). + +**Consequences**: Reviewers may see edits outside the source diff in translation PRs; issue #1 +closes with a pointer here (PLAN Phase 8). + +**Refs**: issue #1. diff --git a/.dev/decisions/D-2026-03-01-heading-maps-in-frontmatter.md b/.dev/decisions/D-2026-03-01-heading-maps-in-frontmatter.md new file mode 100644 index 00000000..a4093a23 --- /dev/null +++ b/.dev/decisions/D-2026-03-01-heading-maps-in-frontmatter.md @@ -0,0 +1,13 @@ +# Heading-maps in `translation:` frontmatter, not `_translation.yml` + +**Context**: Cross-language section matching needs a persistent heading correspondence; the +fork was per-file frontmatter vs a centralized `_translation.yml` (issue #3). + +**Decision**: Per-file frontmatter block (`translation: {title, headings}`) for v0.x — +metadata travels with the file through renames and PRs, no cross-file sync problem. Legacy +`heading-map:` format deprecated (removal: issue #53 / PLAN Phase 8). + +**Consequences**: ARCHITECTURE.md Q2 may reopen the storage location under mystmd (frontmatter +validation), toward `.translate/` (recommendation R2). + +**Refs**: issues #3/#51, PR #52. diff --git a/.dev/decisions/D-2026-03-04-ink-over-rich-cli.md b/.dev/decisions/D-2026-03-04-ink-over-rich-cli.md new file mode 100644 index 00000000..3e1477d3 --- /dev/null +++ b/.dev/decisions/D-2026-03-04-ink-over-rich-cli.md @@ -0,0 +1,12 @@ +# `ink` over Python `rich` for the interactive review UI + +**Context**: The `review` command needed interactive terminal UI; `rich`/`textual` (Python) +render MyST better, but would split the project across two runtimes. + +**Decision**: Stay single-language TypeScript with ink v4; a full Python/`rich` rewrite is +documented as a conditional fallback (FUTURE.md idea 11). + +**Consequences**: Unified codebase and direct imports of the core engine; accept weaker +terminal rendering until it demonstrably blocks reviewers. + +**Refs**: 2026-03 plan Phase 3a (git history: `dev-notes/PLAN.md`). diff --git a/.dev/decisions/D-2026-03-05-whole-file-backward-eval.md b/.dev/decisions/D-2026-03-05-whole-file-backward-eval.md new file mode 100644 index 00000000..5242fc3d --- /dev/null +++ b/.dev/decisions/D-2026-03-05-whole-file-backward-eval.md @@ -0,0 +1,13 @@ +# Whole-file evaluation for backward Stage 2 + +**Context**: Backward analysis originally planned one LLM call per section, mirroring forward +sync. + +**Decision**: Evaluate one whole file per call. Experiment on a 51-file repo: 182 → 32 API +calls with *better* results (more high-confidence findings, less noise) — cross-section +context reduces false positives. + +**Consequences**: Raised the open question of whether forward sync should follow +(ARCHITECTURE.md Q3); experiment data in `experiments/forward/`. + +**Refs**: 2026-03 plan Phase 3b (git history: `dev-notes/PLAN.md`). diff --git a/.dev/decisions/D-2026-03-06-cli-rename-translate.md b/.dev/decisions/D-2026-03-06-cli-rename-translate.md new file mode 100644 index 00000000..2a765478 --- /dev/null +++ b/.dev/decisions/D-2026-03-06-cli-rename-translate.md @@ -0,0 +1,11 @@ +# CLI renamed `resync` → `translate`; `init` command added + +**Context**: The CLI outgrew its resync origin — it manages the whole lifecycle of translated +lecture repos. + +**Decision**: Rename the binary to `translate`; add `init` to bulk-bootstrap a target repo. + +**Consequences**: Eight-command lifecycle CLI (`status`/`backward`/`review`/`forward`/`init`/ +`setup`/`doctor`/`headingmap`). + +**Refs**: PR #23; 2026-03 plan Phase 5 (git history: `dev-notes/PLAN.md`). diff --git a/.dev/decisions/D-2026-04-01-rebase-on-merge.md b/.dev/decisions/D-2026-04-01-rebase-on-merge.md new file mode 100644 index 00000000..b430ea08 --- /dev/null +++ b/.dev/decisions/D-2026-04-01-rebase-on-merge.md @@ -0,0 +1,16 @@ +# Rebase-on-merge over queue/batching for sync-PR conflicts + +**Context**: Issue #63 — 62% of translation PRs on lecture-python-programming.fa hit merge +conflicts. Translation PRs are full-file snapshots, so 3-way merge is structurally impossible +when siblings touch the same file. + +**Decision**: A `rebase` action mode: when a translation PR merges, regenerate open sibling +PRs against the new main, with a section-level translation cache making the common case +zero-API-cost. Rejected: sequential merge queue (latency, complexity); batching (loses per-PR +provenance); auto-merge alone (sidesteps the root cause; race-prone without rebase). + +**Consequences**: Shipped v0.14–v0.15.0. Auto-merge + editor digest (the design's "Stage 2") +deliberately deferred — now FUTURE.md idea 1. Regeneration is idempotent; merge order doesn't +matter. + +**Refs**: issue #63, PR #64; full design in git history: `dev-notes/FIX-ISSUE-63.md`. diff --git a/.dev/decisions/D-2026-06-01-malayalam-keep-english-policy.md b/.dev/decisions/D-2026-06-01-malayalam-keep-english-policy.md new file mode 100644 index 00000000..393f7952 --- /dev/null +++ b/.dev/decisions/D-2026-06-01-malayalam-keep-english-policy.md @@ -0,0 +1,17 @@ +# Malayalam: keep-English-dominant policy + +**Context**: en→ml is a low-resource generation cliff (issue #70: GPT-4 chrF 28.4 vs +specialist NMT ~66), and Malayalam-speaking economists read technical prose with English +terms embedded. Native reviewer: Adisankar Manoj Thanuja. Relevant to any future low-resource +language across QuantEcon translation repos. #promote + +**Decision**: For `ml`, technical terms stay in English with Malayalam grammatical inflection +around them (`economy-യിലെ`, `bond-ന്റെ`); only everyday connective words are translated. +Policy carried by `language-config.ts` prompt rules; the per-term glossary `treatment` field +deferred (zero-schema-change v1). + +**Consequences**: `ml.json` glossary pins most terms `ml == en`; calibration batch with the +native reviewer decides proper-names and parenthetical-first-use policy before PR #71 leaves +draft. + +**Refs**: issue #70, PR #71. diff --git a/.dev/decisions/D-2026-07-05-adopt-dev-notes-convention.md b/.dev/decisions/D-2026-07-05-adopt-dev-notes-convention.md new file mode 100644 index 00000000..fb04f93c --- /dev/null +++ b/.dev/decisions/D-2026-07-05-adopt-dev-notes-convention.md @@ -0,0 +1,21 @@ +# Adopt the `.dev/` notes convention (pilot repo) + +**Context**: Working notes lived in `dev-notes/` (a 1,850-line completed plan, one design +record) with no lifecycle discipline; agent sessions had no durable state or decision record. +QuantEcon needed a cross-repo pattern (design: QuantEcon/QuantEcon.manual#103). An earlier +iteration with YAML frontmatter, CI lint, and metrics was simplified same-day after pilot +feedback — complexity must be earned by observed pain. #promote + +**Decision**: `action-translation` is the first pilot of the lightweight `.dev/` convention: +living docs at the root (`STATE.md` with a `verified:` first line, `PLAN.md`, optional +`ARCHITECTURE.md`/`FUTURE.md`), append-only `decisions/` (`D-YYYY-MM-DD-.md`, superseded +via a note at the top of the old file, never edited) and `log/` (dated session notes), inline +`#promote` tags for cross-repo findings, maintenance via occasional agent "tidy" sessions with +human-approved PRs. No in-repo archive: distill, supersede, or delete — git holds history. + +**Consequences**: Contract lives in `AGENTS.md`; scratch moves to `.dev/scratch/` +([D-2026-07-05-scratch-moves-to-dev-scratch](D-2026-07-05-scratch-moves-to-dev-scratch.md)); +CI ignores `.dev/**`. Keep/expand/kill judged after a few months of real use: is STATE.md the +thing agents and humans actually read first? + +**Refs**: QuantEcon/action-translation#73, QuantEcon/QuantEcon.manual#103, PR #72. diff --git a/.dev/decisions/D-2026-07-05-scratch-moves-to-dev-scratch.md b/.dev/decisions/D-2026-07-05-scratch-moves-to-dev-scratch.md new file mode 100644 index 00000000..87f6d5cf --- /dev/null +++ b/.dev/decisions/D-2026-07-05-scratch-moves-to-dev-scratch.md @@ -0,0 +1,14 @@ +# Scratch moves from `.tmp/` to `.dev/scratch/` + +**Context**: The repo's documented scratch location was a gitignored `.tmp/` folder (anchored +by `.gitkeep`, described in copilot-instructions). The `.dev/` convention brings its own +gitignored `scratch/`; two scratch locations would split agent behaviour. + +**Decision**: `.dev/scratch/` is the single scratch convention. `.tmp/` is retired: its +`.gitkeep` anchor removed, the path left gitignored so stale local copies stay invisible. +`copilot-instructions.md` and `AGENTS.md` updated. + +**Consequences**: Agents write all scratch/working files (PR bodies, command output, drafts) +to `.dev/scratch/`; nothing under `.dev/scratch/` is ever committed. + +**Refs**: QuantEcon/action-translation#73 (adoption step 2). diff --git a/.dev/log/2026-07-05-65a4.md b/.dev/log/2026-07-05-65a4.md new file mode 100644 index 00000000..4a63b0d7 --- /dev/null +++ b/.dev/log/2026-07-05-65a4.md @@ -0,0 +1,23 @@ +# 2026-07-05 — Deep review → `.dev/` pilot adoption (PR #72) + +Ran the full technical review (all `src/`, tests, CI, docs, 26 issues, 41 PRs) and seeded +`.dev/` from it; adopted the pilot convention (issue #73) in the same PR, through three +iterations in one day: + +1. First cut: `dev-notes/` → `.dev/` with PLAN/FUTURE/ARCHITECTURE + single `DECISIONS.md`, + archive folder (commit `0994913`); archive then dropped in favour of distill-and-delete. +2. Full-mechanism cut: YAML 4-key frontmatter, `decisions/` + `log/`, deterministic health + check + CI workflow. +3. Final (this state): **simplified spec** after design review — no YAML, no CI gates; + `verified:` first line, `D--` decision files with supersession notes, inline + `#promote` tags, tidy-sessions for maintenance. Machinery deleted; decisions pruned to the + 9 real ones; scratch migrated `.tmp/` → `.dev/scratch/`; rebase-mode security item redacted + in the plan and pulled forward to Phase 1. + +Spec feedback already absorbed upstream (manual#103 rev 3): living-docs-by-location, date+slug +ids, append-only-vs-pruning clarification, public-content rule. Still open for the spec: +`paths-ignore: ['.dev/**']` on a *required* status check would deadlock `.dev`-only PRs +(skipped required checks never report) — harmless here since `main` has no required checks, +but the convention doc should carry the caveat. #promote + +Refs: PR #72, issue #73, QuantEcon/QuantEcon.manual#103; review assessment in the #73 thread. diff --git a/.dev/log/2026-07-14-state-refresh.md b/.dev/log/2026-07-14-state-refresh.md new file mode 100644 index 00000000..6ad8ecd8 --- /dev/null +++ b/.dev/log/2026-07-14-state-refresh.md @@ -0,0 +1,14 @@ +# 2026-07-14 — STATE refresh + R2 wording (PR #72 review) + +Review of the `.dev/` pilot PR (#72) surfaced two small fixes, applied here: + +1. **STATE.md was stale on arrival** — it listed PR #68 (fr glossary) as open/awaiting review + with a missing `LANGUAGE_CONFIGS` entry, but #68 merged 2026-07-14 with its config wiring. + Moved fr to a new "Recently landed" section, dropped it from the in-flight list, updated + Health & context (fr now enabled), and bumped `verified:` to 2026-07-14. +2. **ARCHITECTURE.md R2** described the rebase PR-body channel as an "input surface whose + validation is being hardened in PLAN Phase 1.5" — more trust-boundary detail than the + public-content rule wants while the fix is unshipped. Dropped that clause; kept the + legitimate engineering rationale (size cap, editability, staleness). + +Refs: PR #72, issue #73. diff --git a/.tmp/.gitkeep b/.dev/scratch/.gitkeep similarity index 100% rename from .tmp/.gitkeep rename to .dev/scratch/.gitkeep diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index d66651c1..ee2f1bac 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -117,27 +117,27 @@ npm run build # Compile TypeScript + bundle dist-action/index.js - Always work on a branch, never commit directly to `main` - Use PRs for all changes, including docs - **Always use create/edit file tools** for file content — never heredoc or shell string escaping -- Multi-line commit messages: write to `.tmp/` first, then use `-F`: +- Multi-line commit messages: write to `.dev/scratch/` first, then use `-F`: ```bash - git commit -F .tmp/msg.txt + git commit -F .dev/scratch/msg.txt ``` ### Using the `gh` CLI -Always write output to the local **`.tmp/`** folder (not `/tmp/`) to keep work repo-scoped: +Always write output to the local **`.dev/scratch/`** folder (not `/tmp/`) to keep work repo-scoped: ```bash # Read PR details -gh pr view 123 > .tmp/pr.txt && cat .tmp/pr.txt +gh pr view 123 > .dev/scratch/pr.txt && cat .dev/scratch/pr.txt # Create PR (write body with file tool first, then:) -gh pr create --title "..." --body-file .tmp/pr-body.txt --base main > .tmp/pr-result.txt && cat .tmp/pr-result.txt +gh pr create --title "..." --body-file .dev/scratch/pr-body.txt --base main > .dev/scratch/pr-result.txt && cat .dev/scratch/pr-result.txt # Create release (write notes with file tool first, then:) -gh release create vX.Y.Z --title "..." --notes-file .tmp/release-notes.md > .tmp/release-result.txt && cat .tmp/release-result.txt +gh release create vX.Y.Z --title "..." --notes-file .dev/scratch/release-notes.md > .dev/scratch/release-result.txt && cat .dev/scratch/release-result.txt ``` -The `.tmp/` folder is committed (via `.gitkeep`) but its contents are git-ignored. +The `.dev/scratch/` folder is committed (via `.gitkeep`) but its contents are git-ignored. ### Addressing Copilot PR Review Comments @@ -147,13 +147,13 @@ After pushing a PR, Copilot may leave review comments. To address them: ```bash gh api repos/QuantEcon/action-translation/pulls/PR_NUM/comments \ --jq '.[] | {id, path, line, body: (.body | split("\n")[0])}' \ - > .tmp/pr-comments.txt && cat .tmp/pr-comments.txt + > .dev/scratch/pr-comments.txt && cat .dev/scratch/pr-comments.txt ``` 2. **Push fixes** to the PR branch addressing the feedback 3. **Reply to each comment** — write reply to a file, then post: ```bash gh api repos/QuantEcon/action-translation/pulls/PR_NUM/comments/COMMENT_ID/replies \ - -f body="$(cat .tmp/reply.txt)" 2>&1 | jq -r '.html_url' + -f body="$(cat .dev/scratch/reply.txt)" 2>&1 | jq -r '.html_url' ``` 4. **Resolve threads** on the GitHub web interface @@ -223,11 +223,11 @@ Docs live in `docs/` — see `docs/index.md` for the full structure. Before creating a release, verify the following: 1. **CHANGELOG is up to date** — all merged PRs and features are listed under `[Unreleased]`; promote `[Unreleased]` → `[X.Y.Z] - YYYY-MM-DD` -2. **Version bumped** — update `package.json`, this file (`copilot-instructions.md`), and `dev-notes/PLAN.md` +2. **Version bumped** — update `package.json`, this file (`copilot-instructions.md`), and `.dev/PLAN.md` 3. **Tests pass** — run `npm test` and confirm all tests pass 4. **Build succeeds** — run `npm run build` to compile TypeScript and update `dist-action/` 5. **Commit, tag, push** — commit all changes, create git tag `vX.Y.Z`, push with `--tags` -6. **Create GitHub release** — `gh release create vX.Y.Z --title "..." --notes-file .tmp/release-notes.md` +6. **Create GitHub release** — `gh release create vX.Y.Z --title "..." --notes-file .dev/scratch/release-notes.md` --- diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 742c291b..cf78a382 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -3,8 +3,12 @@ name: CI on: push: branches: [main] + paths-ignore: ['.dev/**'] pull_request: branches: [main] + # Note: if `test` ever becomes a *required* status check, drop this ignore — + # a skipped required check never reports and would deadlock .dev-only PRs. + paths-ignore: ['.dev/**'] workflow_dispatch: permissions: diff --git a/.gitignore b/.gitignore index 9933bcfe..e64fae7a 100644 --- a/.gitignore +++ b/.gitignore @@ -22,10 +22,12 @@ docs/_build/ # TypeScript build info *.tsbuildinfo -# Temporary files — .tmp/ dir is committed (for scoped gh CLI output) but contents are ignored -.tmp/* +# Scratch — .dev/scratch/ dir is committed (via .gitkeep) but contents are ignored +.dev/scratch/* +!.dev/scratch/.gitkeep +# Legacy scratch location (retired 2026-07-05; see .dev/decisions/) — keep ignoring stale local copies +.tmp/ **/*.tmp -!.tmp/.gitkeep .cache/ # Generated reports (output of CLI tools) diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..c47ddc58 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,39 @@ +# AGENTS.md + +Guidance for coding agents working in this repository (GitHub Action + `translate` CLI for +translating QuantEcon MyST lectures via the Anthropic API; TypeScript). + +## Project notes (`.dev/`) + +Working notes — state, decisions, design ideas — live in [`.dev/`](.dev/README.md) +(the QuantEcon `.dev/` convention; this is the pilot repo). + +- Read [`.dev/STATE.md`](.dev/STATE.md) before starting; it carries a `verified: ` + first line — trust it less as that ages. It points to [`PLAN.md`](.dev/PLAN.md), + [`FUTURE.md`](.dev/FUTURE.md), and [`ARCHITECTURE.md`](.dev/ARCHITECTURE.md). +- Finish each session by appending a short log entry to [`.dev/log/`](.dev/log/) + (`YYYY-MM-DD-.md`) and updating STATE.md if reality changed. +- Record settled decisions in [`.dev/decisions/`](.dev/decisions/) in the same PR that makes + them (`D-YYYY-MM-DD-.md`; never edited — supersede with a new file + a note at the + top of the old one). +- Tag cross-repo findings inline with `#promote`. +- Keep it curated: distill, supersede, or delete — git holds the history. +- `.dev/` is public: no credentials, no unpatched-vulnerability specifics (security + advisories until fixed). + +## Commands + +- `npm install` — setup +- `npm run build` — compile TypeScript (`dist/`) + bundle the action (`dist-action/`) +- `npm test` — Jest suite (build first: the CLI smoke tests execute `dist/cli/index.js`) +- `npm run lint` — ESLint + +## Rules + +- `dist-action/` is committed and must stay in sync with `src/` — always `npm run build` + after source changes; CI fails on drift. +- Use `.dev/scratch/` (gitignored) for scratch files; never create standalone summary/notes + markdown files for individual changes. +- Update `CHANGELOG.md` under `[Unreleased]` for user-visible changes. +- Detailed conventions, module map, and the release checklist: + [`.github/copilot-instructions.md`](.github/copilot-instructions.md). diff --git a/dev-notes/FIX-ISSUE-63.md b/dev-notes/FIX-ISSUE-63.md deleted file mode 100644 index 2c2c0224..00000000 --- a/dev-notes/FIX-ISSUE-63.md +++ /dev/null @@ -1,339 +0,0 @@ -# Fix: Translation sync PR conflicts (Issue #63) - -**Issue**: [#63 — Translation sync PRs conflict when multiple upstream PRs modify the same file — 62% failure rate](https://github.com/QuantEcon/action-translation/issues/63) - ---- - -## Root Cause - -Each translation PR produces a **full file replacement**. When `file-processor.ts` runs `processSectionBased()`, it: -1. Translates only **changed** sections (via Claude) -2. Copies **unchanged** sections verbatim from target repo's `main` -3. Reconstructs the **entire file** from these components - -When two upstream PRs (A, B) both touch the same file: -- Translation PR X (from A) and PR Y (from B) are each generated against the **same target `main`** -- Both PRs contain the full file, each with their own translated sections + shared unchanged sections -- When X merges, the file on `main` changes substantially -- Y's version of the unchanged sections is now stale → three-way merge fails - -This is fundamentally different from typical code conflicts where small localized changes can often auto-merge. Translation PRs replace large blocks of text, making three-way merge almost always impossible. - ---- - -## Recommended Solution: Rebase-on-Merge - -Inspired by Dependabot's default `rebase-strategy`, which automatically rebases open PRs when conflicts are detected after a push to the target branch. - -### Key Insight - -Re-generating a translation PR against an updated `main` is cheap: -- **Sections changed by PR Y**: Same source diff → same translation. Can be cached from the first run, or costs one Claude API call per section at worst. -- **Sections NOT changed by PR Y**: Simply copied from the new target `main` — free, and this is exactly what eliminates the conflict. -- **In the common case** where PRs X and Y modify *different sections* of the same file: **zero Claude API calls** needed for the rebase. - -### Will Rebase-on-Merge Ever Fail With Merge Conflicts? - -**No**, as long as each merge triggers a rebase and the rebase completes before the next PR is merged. Here's the proof by tracing through the same scenario from the issue: - -Given upstream PRs A, B, C all touching `file.md`, creating translation PRs X, Y, Z: - -``` -1. X merges → triggers rebase of Y, Z - - Y regenerated against post-X main → clean - - Z regenerated against post-X main → clean - -2. Y merges → triggers rebase of Z - - Z regenerated against post-X+Y main → clean - -3. Z merges → no remaining PRs → done -``` - -**Order doesn't matter either.** If Z merges first: -``` -1. Z merges → rebase X, Y against post-Z main → both clean -2. X merges → rebase Y against post-Z+X main → clean -3. Y merges → done -``` - -This works because each rebase **regenerates the full file** against current `main`. There is no accumulation of stale state. - -**The only theoretical failure mode** is a race condition: two PRs merged within seconds of each other before the rebase workflow completes. But even this self-heals — the second merge triggers another rebase of any remaining PRs. The system is **eventually consistent** and **idempotent**. Since merging is a human action (review → approve → merge), there is always ample time for the rebase workflow to complete (<1 minute typically). - -### Architecture - -``` -Target repo: translation-sync PR merged - │ - ▼ -Workflow fires (pull_request.closed + merged) - │ - ▼ -Find other open translation-sync PRs - │ - ▼ -For each PR touching the same files: - 1. Read source PR metadata from PR body (repo, PR#, commit SHA) - 2. Fetch source content at original commit SHA - 3. Fetch UPDATED target main (post-merge) - 4. Re-run SyncOrchestrator pipeline - 5. Force-push result to existing PR branch - 6. Comment: "♻️ Rebased after #N was merged. Translations unchanged." -``` - -### Metadata Already Available - -The PR body (built by `buildPrBody()` in `pr-creator.ts`) already contains: -- Source repo owner/name -- Source PR number (with link) -- Source/target language -- Claude model used -- List of files changed - -**Needed additionally** (machine-readable format in PR body): -- Source commit SHA (the `merge_commit_sha` used during translation) -- Per-file source content hashes or the SHA references -- A structured metadata block (e.g., HTML comment with JSON) so the rebase mode can parse it reliably - -### Branch Name Convention - -Current: `translation-sync-{timestamp}-pr-{N}` - -This prefix is already unique enough to identify translation sync PRs programmatically via the GitHub API. - ---- - -## Implementation Plan - -### Phase 1: Structured Metadata in PR Body - -Embed machine-readable metadata in the PR body so the rebase mode can reconstruct the pipeline inputs: - -```html - -``` - -This is a non-breaking change — existing PRs without metadata simply can't be rebased. - -**Status**: ✅ Complete — commit `edefffb` - -### Phase 2: Rebase Mode - -New action mode `rebase` that runs in the **target repo** when a translation-sync PR is merged: - -1. **List open translation-sync PRs** via GitHub API (filter by branch prefix `translation-sync-`) -2. **Check file overlap** — compare files touched by merged PR with each sibling PR's metadata -3. **Parse metadata** from each PR's body -4. **Re-run the sync pipeline** for conflicted PRs: - - Fetch source file contents at the recorded commit SHA - - Fetch updated target `main` content - - Run `SyncOrchestrator.processFiles()` with the same inputs - - Force-push the result to the existing PR branch (reset to main SHA, then commit) -5. **Post a comment** on the rebased PR explaining what happened - -**Status**: ✅ Complete — commit `0eed2e3` - -### Phase 3: Translation Cache (Optimization) - -To ensure rebases cost zero Claude API calls in the common case: - -- Store `targetBaseSha` (target repo's default branch SHA at PR creation) in the PR metadata -- Before resetting the branch during rebase, read previously translated files from the PR branch -- Also fetch old target content at `targetBaseSha` (the original baseline) -- Parse both into document components and compare section-by-section: - - If a section's target content is unchanged since PR creation → cache hit → skip Claude call - - If a section's target content changed (due to the merged PR) → cache miss → re-translate -- For added sections: use the heading map from the cached translation to match English→translated sections -- For title and intro: same comparison logic (old target vs current target) -- Graceful degradation: if cache parsing fails, falls through to normal re-translation - -This makes rebase effectively free for PRs that modify different sections of the same file (the common case from issue #63). - -**Status**: ✅ Complete — 7 new tests (999 total) - -### Phase 4: Target Repo Workflow Template - -Ready-to-use workflow template at `examples/rebase-translations.yml`: - -```yaml -name: Rebase Translation PRs -on: - pull_request: - types: [closed] - -jobs: - rebase: - if: > - github.event.pull_request.merged == true && - startsWith(github.event.pull_request.head.ref, 'translation-sync-') - runs-on: ubuntu-latest - concurrency: - group: rebase-translations - cancel-in-progress: false - steps: - - uses: quantecon/action-translation@v0.15 - with: - mode: rebase - anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }} - github-token: ${{ secrets.GITHUB_TOKEN }} -``` - -Includes a `concurrency` group to prevent overlapping rebases from racing. - -**Status**: ✅ Complete — `examples/rebase-translations.yml` - ---- - -## Feature: Auto-Merge (Configurable) - -Auto-merging high-quality translation PRs reduces editor overhead for the ~80-90% of translations that score well on review. This is built **after** rebase-on-merge, so it becomes a genuine workflow improvement rather than a conflict workaround. - -**Why build both**: Rebase-on-merge solves the root cause (conflicts). Auto-merge solves a different problem (editor bottleneck). Without rebase-on-merge, auto-merge is a race condition — you're hoping to merge before the next sync fires. With rebase-on-merge in place, auto-merge becomes safe and purely a productivity feature. - -### Configuration - -This **must be a configurable option** that can be enabled/disabled per target repo: - -```yaml -# In the target repo's workflow -- uses: QuantEcon/action-translation@v0.15 - with: - mode: sync - auto-merge: true # default: false - auto-merge-quality-threshold: 9 # minimum review score (1-10, default: 9) - auto-merge-labels: "auto-merged" # label applied to auto-merged PRs - auto-merge-digest: "weekly" # digest frequency: "weekly" | "monthly" | "none" (default: "weekly") - auto-merge-digest-assignees: "editor-username" # GitHub users assigned to digest issues -``` - -### Behavior When Enabled - -1. After sync creates the translation PR, the review mode runs automatically -2. If review score >= threshold AND no structural issues: - - PR is approved and merged automatically - - Label `auto-merged` is applied -3. If review score < threshold: - - PR remains open for human review (normal flow) - - Label `needs-review` is applied - -### Digest Report (Accountability Layer) - -A scheduled workflow generates periodic digest issues so editors can audit auto-merged translations without watching every PR: - -**Trigger**: Scheduled workflow (cron), configurable as weekly or monthly. - -**Digest issue contents**: -- Summary: "N translation PRs were auto-merged since the last report" -- Table of each auto-merged PR: - - PR number + link - - Source PR number + link - - Files changed - - Review score (overall + per-section breakdown) - - Any review comments or warnings flagged by the automated reviewer - - Link to the full review comment on the PR -- Assigned to the configured editor(s) -- Labelled `translation-digest` - -**Example workflow for digest**: -```yaml -name: Translation Digest Report -on: - schedule: - - cron: '0 9 * * 1' # Weekly, Monday 9am UTC - -jobs: - digest: - runs-on: ubuntu-latest - steps: - - uses: QuantEcon/action-translation@v0.15 - with: - mode: digest - digest-period: weekly # or "monthly" - digest-assignees: "editor-username" - target-repo-token: ${{ secrets.GITHUB_TOKEN }} -``` - -**Editor workflow**: -1. Editor receives the digest issue (weekly/monthly) -2. Reviews the summary table — most entries need no action -3. If something looks wrong, clicks through to the PR and its review comments -4. Comments on the digest issue to flag problems → can trigger a fix PR or `/translate-resync` -5. Closes the digest issue when satisfied - -### Safeguards - -- **Off by default** — repos must explicitly opt in -- **Score threshold is configurable** — conservative default of 9/10 -- **Digest reports** — editors can retroactively flag issues on a comfortable cadence -- **Label tracking** — easy to find and audit auto-merged PRs via GitHub search -- **Structural issue veto** — heading misalignment, missing sections, etc. always block auto-merge regardless of score -- **Digest assignees** — ensures someone is accountable for reviewing the reports - ---- - -## Comparison of Approaches - -| Criterion | Rebase-on-merge | Auto-merge | Sequential queue | Batching | -|---|---|---|---|---| -| Preserves human review | ✅ Yes | ❌ No (bypassed) | ✅ Yes | ✅ Yes | -| No added latency | ✅ Yes | ✅ Yes | ❌ Queue depth | ❌ Time window | -| Per-PR provenance | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Combined | -| Zero/low API cost | ✅ Section reuse | N/A | ❌ Re-translates | ✅ Yes | -| Handles rapid upstream | ✅ Yes | ✅ Yes | ❌ Slow | Partially | -| Complexity | Medium | Low | High | Medium | -| Solves root cause | ✅ Yes | ❌ Sidesteps it | ✅ Yes | Partially | - -**Recommendation**: Build both features sequentially — rebase-on-merge first (solves the root cause), then auto-merge with digest reports (reduces editor overhead). Rebase-on-merge is the foundation; auto-merge is safe to deploy only once rebasing is reliable. - ---- - -## Risks and Mitigations - -### Force-push invalidates existing reviews -- **Mitigation**: Post a clear comment explaining only "copied from main" sections changed, not the translated content -- **Mitigation**: If the PR had an approved review, re-request review automatically -- **Mitigation**: The diff between old and new branch should be minimal — reviewers can verify quickly - -### Race condition on rapid sequential merges -- **Mitigation**: The system is idempotent — each merge triggers a rebase, so the system self-heals -- **Mitigation**: Add a de-bounce mechanism (e.g., workflow concurrency group per file) to avoid redundant rebases - -### Source PR metadata missing (pre-existing PRs) -- **Mitigation**: PRs without structured metadata are skipped with a comment: "Unable to rebase — metadata not found. Use `/translate-resync` to recreate." -- **Mitigation**: All new PRs will include metadata going forward - -### Claude API cost during rebase -- **Mitigation**: Phase 3 translation cache eliminates API calls for the common case (different sections modified) -- **Mitigation**: Rebase only processes conflicted PRs, not all open PRs - ---- - -## Implementation Sequence - -### Stage 1: Fix the Root Cause (Rebase-on-Merge) — ✅ Complete - -1. **Phase 1**: ✅ Structured metadata in PR body — `sourceCommitSha` + `targetBaseSha` + file list -2. **Phase 2**: ✅ Rebase mode — core solution, eliminates all merge conflicts -3. **Phase 3**: ✅ Translation cache — optimization, reduces API cost of rebases to near-zero -4. **Phase 4**: ✅ Rebase workflow template — ready-to-use YAML for target repos - -### Stage 2: Workflow Automation (Auto-Merge + Digest) - -Built **after** rebase-on-merge is stable and deployed. These are independent features that complement the fix. - -5. **Phase 5**: Auto-merge mode — configurable, off by default, quality-gated -6. **Phase 6**: Digest report mode — scheduled workflow, generates periodic summary issues for editor review - -Stage 2 depends on Stage 1 being complete. Without rebase-on-merge, auto-merge is a race condition. With it, auto-merge is a safe productivity feature. diff --git a/dev-notes/PLAN.md b/dev-notes/PLAN.md deleted file mode 100644 index 2760f297..00000000 --- a/dev-notes/PLAN.md +++ /dev/null @@ -1,1850 +0,0 @@ -# PLAN: Development Roadmap - -**Created**: 2026-02-16 -**Last Updated**: 2026-03-26 (v0.12.5 release, heading-map MyST role fix) -**Sources**: docs/DESIGN-RESYNC.md -**Current Version**: v0.12.5 -**Test Status**: 935 tests passing (39 test suites, 5 snapshots) - ---- - -## Overview - -This plan combines three work streams into a single prioritized roadmap: - -1. **Code Health** — Address technical review findings to strengthen the foundation -2. **Resync CLI** — Build the CLI tool with three commands: `backward`, `backward-sync`, and `forward` -3. **Cleanup** — Remove deprecated tools and improve repo hygiene - -### Four-Command Architecture - -| Command | Direction | Purpose | Output | Modifies files? | -|---------|-----------|---------|--------|------------------| -| `status` | — | Fast structural diagnostic | Console table | No | -| `backward` | TARGET → SOURCE | Identify improvements worth considering | Suggestion report folder | No | -| `review` | — | Interactive human review of backward suggestions | GitHub Issues | No (creates Issues) | -| `forward` | SOURCE → TARGET | Resync TARGET after drift or failed propagation | Updated translated files | Yes | - -**Workflow**: `status` (quick check) → `backward` (discover suggestions) → `review` (human decides, creates Issues) → human edits SOURCE → Action translates forward (incremental) *or* `forward` CLI (drift recovery) - -### Two-Stage Backward Architecture - -The `backward` command uses a two-stage approach to minimize cost and maximize accuracy: - -``` -Stage 1: Document-Level Triage (every file, 1 LLM call each) - Full SOURCE + Full TARGET → "Any substantive changes beyond translation?" - NO → skip file (vast majority) - YES → brief notes on what looks different → proceed to Stage 2 - -Stage 2: Whole-File Evaluation (flagged files only, 1 LLM call per file) - All matched section pairs in one prompt → per-section suggestions - Per-section suggestions with category, confidence, reasoning -``` - -**Why two stages**: Most translation changes improve the *translation*, not the *source content*. Backport candidates are rare. Stage 1 filters at ~$0.01/file, avoiding unnecessary Stage 2 analysis. - -**Why whole-file Stage 2**: Originally Stage 2 made 1 LLM call per section (N calls per file). Refactored to 1 call per file with all sections in one prompt. Real-world comparison on 51-file repo: - -| Metric | Section-by-Section | Whole-File | -|--------|-------------------|------------| -| Stage 2 API calls | 182 | 32 | -| High confidence findings | 6 | 7 | -| Medium confidence (noise) | 25 | 17 | -| Total suggestions | 31 | 24 | - -The whole-file approach is strictly better: ~6x fewer API calls, better signal-to-noise, and cross-section context helps the LLM make more accurate assessments. - -The plan is sequenced so that foundational improvements (especially `index.ts` refactoring) unblock the resync tool work. - ---- - -## Phase 0: Foundation (Pre-Resync Prerequisites) ✅ COMPLETE - -**Goal**: Address critical review findings that directly unblock resync CLI development -**Estimated effort**: 3-4 days - -### 0.1 Extract Sync Orchestration from `index.ts` (HIGH PRIORITY) ✅ - -`index.ts` was 766 lines combining GitHub Action entry point with full sync orchestration logic. Now refactored into three modules. - -**Tasks**: -- [x] Create `src/sync-orchestrator.ts` — extracted sync logic (424 lines) - - File classification (`classifyChangedFiles`) - - Translation dispatch (section-based vs full) - - Document reconstruction coordination - - Error aggregation across files - - Glossary loading utility (`loadGlossary`) - - Logger interface for GitHub Action / CLI decoupling -- [x] Create `src/pr-creator.ts` — PR creation logic (323 lines) - - Branch creation - - File commits - - PR body generation (`buildPrBody`, `buildPrTitle`) - - Label and reviewer assignment (`buildLabelSet`) -- [x] Slim `index.ts` to ~447 lines of GitHub Action glue - - Mode routing - - Input fetching - - GitHub API content fetching (`fetchFileContent`, `fetchAllFileContents`) - - Delegating to orchestrator and PR creator -- [x] Add tests for `sync-orchestrator.ts` (26 tests) - - Multi-file processing - - Error recovery (one file fails, others continue) - - File filtering logic - - Glossary loading -- [x] Add tests for `pr-creator.ts` (12 tests) - - PR title generation - - PR body formatting - - Label set deduplication - -### 0.2 Add Retry Logic to Translator (MEDIUM PRIORITY) ✅ - -Translation API calls now have retry with exponential backoff. - -**Tasks**: -- [x] Add exponential backoff retry in `translator.ts` (3 attempts: 1s, 2s, 4s) -- [x] Retry on: `RateLimitError`, `APIConnectionError`, transient `APIError` (5xx) -- [x] Skip retry on: `AuthenticationError`, `BadRequestError`, document-too-large -- [x] Add tests for retry behavior (12 tests) - -### 0.3 Quick Cleanup ✅ - -- [x] Remove deprecated `findTargetSectionIndex()` from `file-processor.ts` -- [x] Remove deprecated `findMatchingSectionIndex()` from `file-processor.ts` -- [x] Remove dead `findSourceSectionIndex()` method (always returns -1) -- [x] `coverage/` already in `.gitignore` -- [x] Run tests to confirm no regressions - ---- - -## Phase 1: Resync CLI — Single-File Backward with Two-Stage Architecture ✅ COMPLETE - -**Goal**: Validate two-stage triage + section analysis on a single file -**Prerequisite**: Phase 0 (orchestrator extraction for module reuse) ✅ -**Status**: All modules implemented, 93 CLI tests passing, validated against real QuantEcon repos - -### 1.1 Directory Structure & CLI Scaffolding ✅ - -- [x] Create `src/cli/` directory structure: - ``` - src/cli/ - ├── index.ts # CLI entry point (commander.js) - ├── types.ts # CLI-specific types - ├── section-matcher.ts # Cross-language section matching - ├── git-metadata.ts # File-level git dates + interleaved timeline - ├── document-comparator.ts # Stage 1: whole-document triage - ├── backward-evaluator.ts # Stage 2: section-level detail - ├── report-generator.ts # Markdown/JSON output - ├── commands/ - │ └── backward.ts # resync backward command - └── __tests__/ - ├── document-comparator.test.ts - ├── section-matcher.test.ts - ├── backward-evaluator.test.ts - ├── report-generator.test.ts - ├── backward.test.ts - ├── git-metadata.test.ts - └── fixtures/ # 6 paired fixture sets - ``` -- [x] Install `commander` dependency -- [x] Configure `package.json` `bin` entry for `resync` command -- [x] Add CLI build script (`npm run build:cli`) - -### 1.2 CLI Types (`src/cli/types.ts`) ✅ - -- [x] `TriageResult` interface (file, verdict, notes, tokenCount) -- [x] `TriageVerdict` type (`CHANGES_DETECTED`, `IN_SYNC`, `SKIPPED_TOO_LARGE`) -- [x] `SectionPair` interface (sourceSection, targetSection, status) -- [x] `SectionSyncStatus` type (`SOURCE_ONLY`, `TARGET_ONLY`, `MATCHED`) -- [x] `BackportSuggestion` interface (category, confidence, summary, changes, reasoning) -- [x] `BackportCategory` type (`BUG_FIX`, `CLARIFICATION`, `EXAMPLE`, `CODE_IMPROVEMENT`, `I18N_ONLY`, `NO_CHANGE`) -- [x] `FileGitMetadata` interface (lastModified, lastCommit, lastAuthor) -- [x] `TimelineEntry` interface (date, repo, sha, message) -- [x] `FileTimeline` interface (entries, counts, estimatedSyncDate, sourceCommitsAfterSync) -- [x] `BackwardReport` interface (file, triageResult, suggestions, metadata, timeline) - -### 1.3 Document Comparator — Stage 1 (`src/cli/document-comparator.ts`) ✅ - -- [x] `triageDocument()` function — single LLM call per file -- [x] Recall-biased prompt (false positives are cheap, false negatives lose backports) -- [x] Pre-flight size check (skip Stage 1 for very large documents) -- [x] `--test` flag support (deterministic mock response) -- [x] Reuse retry logic from `translator.ts` -- [x] Commit timeline context in prompt (prevents directional reasoning errors) -- [x] Unit tests + prompt snapshot tests - -### 1.4 Section Matcher (`src/cli/section-matcher.ts`) ✅ - -- [x] `matchSections()` — position-based matching with heading-map validation -- [x] Handle `SOURCE_ONLY`, `TARGET_ONLY`, `MATCHED` pairs -- [x] Reuse `MystParser` from `parser.ts` for both languages -- [x] Reuse `extractHeadingMap()` from `heading-map.ts` -- [x] Unit tests for section matching - -### 1.5 Backward Evaluator — Stage 2 (`src/cli/backward-evaluator.ts`) ✅ - -- [x] `evaluateSection()` function — per-section cross-language comparison -- [x] Stage 1 notes passed as context to focus analysis -- [x] Commit timeline context in prompt -- [x] Structured JSON response with category, confidence, suggestion -- [x] Respectful suggestion tone (SOURCE is truth) -- [x] Robust JSON parsing (handles Claude's inconsistent formatting) -- [x] Reuse retry logic from `translator.ts` -- [x] Unit tests + prompt snapshot tests - -### 1.6 Git Metadata (`src/cli/git-metadata.ts`) ✅ - -- [x] `getFileGitMetadata()` — last commit date, SHA, author per file -- [x] `getFileTimeline()` — interleaved SOURCE/TARGET commit history -- [x] `getRepoCommits()` — full commit history for a file from one repo -- [x] `parseTimelineEntry()` — parse `git log` output lines -- [x] `formatTimelineForPrompt()` — compact format for LLM prompts -- [x] Estimated sync point detection (earliest TARGET commit) -- [x] Post-sync SOURCE commit counting -- [x] Unit tests against current repo - -### 1.7 Report Generator (`src/cli/report-generator.ts`) ✅ - -- [x] Clear top-level **Result** verdict (IN SYNC / NO ACTION NEEDED / N SUGGESTIONS / SKIPPED) -- [x] Commit Timeline section in Markdown reports -- [x] Stage 1 triage summary -- [x] Per-section suggestions with confidence, category, reasoning -- [x] Confidence labels (HIGH / MEDIUM / LOW) -- [x] JSON report output -- [x] Bulk report support -- [x] Unit tests - -### 1.8 Backward Command (`src/cli/commands/backward.ts`) ✅ - -- [x] Parse CLI arguments: `-f`, `-s`, `-t`, `-o`, `-l`, `--test`, `--json` -- [x] Load and parse SOURCE and TARGET files -- [x] Get git metadata + interleaved timeline -- [x] **Stage 1**: `triageDocument()` — full document comparison with timeline -- [x] **Stage 2**: Extract heading-map, match sections, `evaluateSection()` per pair with timeline -- [x] Generate and write report (Markdown and/or JSON) - -### 1.9 Test Fixtures & Validation ✅ - -- [x] 6 paired fixture sets: - - `aligned-pair/` — faithful translation, Stage 1 returns `IN_SYNC` - - `bug-fix-in-target/` — TARGET corrected a formula, flags `BUG_FIX` - - `clarification-in-target/` — TARGET added context - - `i18n-only-changes/` — font/punctuation changes only, filters out - - `section-count-mismatch/` — TARGET has extra sections - - `no-heading-map/` — position-only matching (graceful degradation) -- [x] Validated against real QuantEcon repos (`lecture-python-intro` ↔ `lecture-intro.zh-cn`) -- [x] Timeline context resolved real false positive (unicode variables on `solow.md`) - -### Key Learning from Real-World Testing - -Running against `solow.md` revealed a critical false positive: the LLM suggested backporting TARGET's ASCII variable names (`alpha`) to replace SOURCE's unicode names (`α`). But SOURCE had adopted unicode *after* the translation was created. Adding the interleaved commit timeline to prompts eliminated this error — Stage 2 now correctly produces zero backport suggestions for this case. - -**Phase 1 Deliverable**: Working `npx resync backward -f file.md` with two-stage triage ✅ - ---- - -## Phase 2: Resync CLI — Bulk Analysis & Status (2-3 days) ✅ COMPLETE - -**Goal**: Scale backward to full repository + quick diagnostic command -**Status**: Status command + bulk backward implemented, 45 new tests (456 total) -**Validated**: Full LLM bulk run on 51-file repo (49 analyzed, 20 suggestions found, 5 high-confidence BUG_FIX) - -### 2.1 Status Command (`src/cli/commands/status.ts`) ✅ - -No LLM calls — fast and free diagnostic. Output goes to the **CLI console** (like `git status`), not report files. - -- [x] Check heading-map presence in each TARGET file -- [x] Detect structural differences (section count mismatch) -- [x] Compare file modification dates (git metadata) -- [x] Report per-file sync status: - - `ALIGNED` — structure matches, heading-map present, no newer SOURCE commits - - `SOURCE_AHEAD` — SOURCE has more sections than TARGET (sections added upstream) - - `TARGET_AHEAD` — TARGET has more sections than SOURCE (unexpected divergence) - - `OUTDATED` — SOURCE has newer commits than TARGET (needs forward sync) - - `MISSING_HEADINGMAP` — no heading-map in TARGET - - `SOURCE_ONLY` — file missing in TARGET - - `TARGET_ONLY` — file missing in SOURCE -- [x] Print summary table to stdout (compact format with `↳` detail lines) -- [x] Support `--file` flag (single file diagnostic) -- [x] Support `--json` flag (prints JSON to stdout) -- [x] Unit tests (21 tests) - -### 2.2 Bulk Backward Processing ✅ - -Bulk mode writes reports into a **date-stamped folder** — the folder *is* the report: -``` -reports/backward-2026-03-04/ -├── _summary.md # Aggregate summary -├── _summary.json # (with --json) -├── .resync/ # Hidden subfolder for machine-readable data -│ ├── _progress.json # Checkpoint manifest -│ ├── _log.txt # Detailed per-file processing log -│ ├── cobweb.json # Per-file JSON sidecar -│ └── solow.json -├── cobweb.md # Per-file report -├── solow.md -└── ... -``` - -- [x] File discovery (find all `.md` files in docs folder) -- [x] File filtering: - - `--exclude ` option (e.g., `--exclude README.md`) - - Respect `_toc.yml` if present to discover the actual lecture list -- [x] Progress bar (`cli-progress` library, TTY-only): - - Single updating line: `█████░░░ 24/51 | ✓ 8 sync 📝 5 suggestions ❌ 0 errors | current_file` - - Clears on completion, replaced by final summary - - Detailed per-file output goes to `.resync/_log.txt` log file -- [x] Two-stage bulk flow: - - Stage 1 triage on all files (fast, 1 call each) - - Stage 2 section analysis only on flagged files -- [x] Parallel processing (5 concurrent files via `Promise.all` batching) -- [x] Buffered logger (`BufferedLogger` class) — collects per-file output, flushes to log file atomically to prevent interleaving -- [x] Fresh start on re-run — wipes output folder unless `--resume` flag is set -- [x] Incremental checkpointing: - - Write each per-file report to disk as it completes - - Maintain `.resync/_progress.json` tracking which files are done - - Support `--resume` to skip already-completed files in the output folder -- [x] Per-file reports (individual Markdown/JSON per analyzed file) -- [x] Aggregate summary report across all files -- [x] Cost estimation (`--estimate` flag) - - Count files for Stage 1 triage - - Estimate how many files will be flagged (~5-10% based on experience) - - Estimate Stage 2 section calls for flagged files - - Calculate estimated total API cost - - Calculate estimated time - - Prompt user to proceed (y/N) -- [x] Robust LLM response parsing (3-strategy approach: code fence → greedy regex → keyword fallback) -- [x] Default model upgrade to `claude-sonnet-4-6` - -### 2.3 Output Formats ✅ - -- [x] Wire `--json` into bulk backward (per-file + aggregate) -- [x] Wire `--json` into status command (stdout) - -> **Note**: Stable JSON schema definition and documentation deferred to Phase 4. - -**Phase 2 Deliverable**: `npx resync status` (console) + full-repo `npx resync backward` (report folder) ✅ - -### Key Learnings from Phase 2 Real-World Testing - -- **Parallel output interleaving**: Running 5 files concurrently with direct console output was unreadable. Solved with `BufferedLogger` that collects per-file output and flushes atomically to a log file. -- **Progress bar UX**: A single animated progress line (stderr, TTY-only) with counters (sync/suggestions/errors) is far better than scrolling file-by-file output. The log file preserves all detail for debugging. -- **LLM response variability**: The same file can produce 0-2 suggestions across runs. Borderline confidence scores (~0.72) fluctuate. This is expected LLM behavior, not a bug. -- **JSON parsing brittleness**: Claude wraps JSON in code fences, omits them, or returns partial JSON unpredictably. The 3-strategy parsing approach (code fence → greedy regex → keyword fallback) handles all observed formats. -- **Cost validation**: Full 51-file run cost ~$0.85, completed in ~4 minutes with 5-way parallelism. Stage 1 flagged ~67% of files (higher than estimated 5-10%), but Stage 2 filtered effectively — only 20 actionable suggestions from 49 files. -- **High-value findings**: 5 high-confidence BUG_FIX suggestions (confidence 0.85-0.97) represent genuine improvements made in the Chinese translation that should be backported to the English source. - ---- - -## Phase 3a: Resync CLI — Interactive Review (3-4 days) ✅ COMPLETE - -**Goal**: Interactive human review of backward suggestions with GitHub Issue creation -**Status**: All 5 steps implemented, 125 new tests (515 → 640 total, 24 → 29 suites), validated end-to-end on test repos -**Key insight from Phase 2**: Backward suggestions are rare (5 high-confidence out of 51 files) and need human judgment. Automating backward-sync via LLM is over-engineered — the value is in making the human review loop fast and pleasant. - -### Design Decisions - -**`backward-sync` deferred**: Originally planned as an LLM reverse-translation step. Deferred because: -- Backport candidates are rare — each gets careful human attention -- Edits are typically small (fix a formula, add a sentence) -- A human reading the backward report can edit SOURCE directly, often faster and higher quality than an LLM round-trip -- Risk of the LLM misunderstanding *why* the translation diverged -- If LLM assistance is needed for a specific edit, it can be done ad-hoc outside this tool - -**`review` command (new)**: Interactive CLI that walks through backward report suggestions, lets a human accept/skip/reject each one, and creates GitHub Issues for accepted suggestions. This bridges the gap between "discovery" (backward) and "action" (human edits SOURCE). - -**`forward` command — RESYNC mode**: Different from the GitHub Action's forward sync. The Action handles incremental updates triggered by PRs (UPDATE mode with old/new SOURCE diff). The forward CLI handles **drift recovery** — when repos are out of sync due to failed propagation, manual edits, or initial onboarding. Uses a new RESYNC prompt that preserves translation nuances. - -### CLI Framework: `ink` v4 (React for CLI) - -The `review` command needs rich terminal rendering: syntax-highlighted MyST markdown, panels, interactive prompts. **Decision: `ink` v4** — React-based CLI framework in Node.js, ESM module system. - -**Why `ink`**: -- **Unified codebase** — Everything stays TypeScript. The `forward` command imports `translator.ts`, `parser.ts`, `section-matcher.ts` directly. No IPC, no subprocess, no duplication. -- **Component model** — ``, ``, `` map naturally to the review UI -- `ink-syntax-highlight` for code blocks, custom components for MyST directives -- Testable via `ink-testing-library` (renders to string, asserts output) -- No new runtime dependency — users already need Node.js for the existing CLI -- Production precedent: Gatsby, Prisma, Shopify CLIs - -**Why v4 (ESM)**: ink v4 is ESM-only. The existing CLI compiles to CommonJS, but the Node.js ecosystem is firmly moving to ESM. Migrating now avoids a later migration. Requires `tsconfig.json` module changes and import path adjustments for the CLI build. - -**Rendering approach**: Custom `` component using `chalk` + `cli-highlight` for syntax highlighting, `` for directive panels, styled `` for headers/math. Gets ~80% of `rich`'s rendering quality. See "Future: Python Rewrite" section for the path to best-in-class rendering. - -**Rejected alternative**: Python with `rich` — superior rendering but requires a full rewrite of the entire CLI to avoid a mixed-language project. Documented in the Future section as a long-term option. - -### 3a.0 Prerequisites - -#### Formalize Backward Report JSON Schema ✅ - -The `review` command reads `.resync/*.json` sidecars produced by `backward`. Define the schema contract before building the consumer. - -- [x] Document the JSON schema for per-file sidecar files (`.resync/.json`) -- [x] Document the JSON schema for `_summary.json` -- [x] Document the JSON schema for `_progress.json` -- [x] Add TypeScript types or Zod schema for runtime validation — `src/cli/schema.ts` with Zod -- [x] Add schema version field for future compatibility — `SCHEMA_VERSION` constant (`1.0.0`) - -Implemented in `src/cli/schema.ts` (Zod schemas, parse/load/filter utilities) with 41 tests. PR #17. - -#### ESM Migration for CLI Build ✅ - -- [x] Update `tsconfig.json` for ESM output — `module: node16`, `moduleResolution: node16`, `target: ES2022` -- [x] Update import paths to include `.js` extensions where needed — all ~50 source + test files -- [x] Install `ink` v4, `react` 18 — installed as runtime deps -- [x] Verify existing CLI commands (`backward`, `status`) still work after migration — 515 tests pass -- [x] Update `build:cli` script — `tsc` for ESM, `esbuild` for CJS action bundle (`dist-action/`) - -Also replaced `@vercel/ncc` with `esbuild` for action bundling (CJS format). PR #17. - -### 3a.1 Review Command (`review`) - -Interactive CLI that reads a backward report folder and walks through each suggestion: - -``` -npx resync backward ... # Phase 2 (done) — generates report folder -npx resync review # Phase 3 (new) — interactive walk-through -``` - -**Per-suggestion display**: -- File name + section heading -- Category badge + confidence score (e.g., `BUG_FIX 0.92`) -- LLM reasoning (why this was flagged) -- Syntax-highlighted SOURCE and TARGET excerpts for the relevant section -- Suggested change description - -**Actions per suggestion**: **[A]ccept** → create Issue · **[S]kip** → move on · **[R]eject** → mark as false positive - -**End-of-session summary**: N accepted / N skipped / N rejected, with links to created Issues - -#### Build Plan (agreed 2026-03-04) - -**Key decisions**: -- Build `--dry-run` first to iterate on human factors before wiring up Issue creation -- Start with basic `chalk`-styled output, add rich rendering incrementally -- Issues target the SOURCE repo (e.g., `lecture-python-intro`), since suggestions are about improving the English source - -#### Step 1: Command scaffold + report loading ✅ - -- [x] Register `resync review ` in commander.js (`src/cli/commands/review.ts`) -- [x] Parse CLI arguments: ``, `--repo ` (SOURCE repo for Issues), `--dry-run` -- [x] Load report folder using `loadResyncDirectory()` from `schema.ts` -- [x] Filter to actionable suggestions using `filterActionableSuggestions()` from `schema.ts` -- [x] Flatten to a sorted list of suggestions across all files (highest confidence first) -- [x] Unit tests for loading + filtering pipeline — 20 tests (PR #18) - -#### Step 2: `--dry-run` formatter (non-interactive) ✅ - -Chalk-styled stdout output — no ink yet. Fast iteration on what information matters. - -- [x] Per-suggestion display: file name, section heading, category badge + confidence, LLM reasoning, suggested change -- [x] End-of-run summary: total suggestions, breakdown by category/confidence -- [x] Test with real report data from `reports/lecture-python-intro/backward-2026-03-04-section-by-section/` -- [x] Unit tests for formatter output (`review-formatter.test.ts`, 33 tests) - -#### Step 3: Issue body generator ✅ - -- [x] Format GitHub Issue body for a suggestion -- [x] Issue title format: `[filename] brief description of suggestion` -- [x] Issue body includes: - - Category + confidence - - Section heading and location in file - - Full LLM reasoning - - SOURCE and TARGET excerpts - - "Generated by `resync backward` on YYYY-MM-DD" footer -- [x] Labels: `translate`, `translate:{category}` (e.g., `translate:bug-fix`), `translate:{language}` (e.g., `translate:zh-cn`) -- [x] `--dry-run` shows Issue preview (title + body + labels) without creating anything -- [x] Unit tests for Issue body generation (`issue-generator.test.ts`, 33 tests) - -#### Step 4: Ink interactive mode ✅ - -Layer ink on top of the dry-run formatter. - -- [x] `` ink component renders each suggestion with card + Issue preview -- [x] [A]ccept / [S]kip / [R]eject keypresses per suggestion -- [x] Accept queues suggestion for Issue creation -- [x] Track session state (accepted/skipped/rejected counts) in `review-session.ts` -- [x] End-of-session summary with counts + list of accepted suggestions -- [x] Pure state machine `review-session.ts` tested independently of ink rendering -- [x] State machine tests (`review-session.test.ts`, 20 tests) -- [x] Dynamic imports for `ink`/`react` to keep ESM modules out of Jest CJS environment - -#### Step 5: `gh` Issue creation ✅ - -- [x] Wire accepted suggestions to `gh issue create` on SOURCE repo (`--repo` flag) -- [x] Labels: `translate`, `translate:{category}`, `translate:{language}` -- [x] Print Issue URLs in end-of-session summary -- [x] Injectable `GhRunner` type for testability — no subprocess in tests -- [x] `--dry-run` end-to-end shows Issue preview (title + body + labels) without creating -- [x] Unit tests for arg building, single Issue creation, batch creation (`issue-creator.test.ts`, 17 tests) - -#### MyST-Aware Terminal Rendering (incremental, across steps 2-4) - -Start with basic chalk styling, add richer rendering as needed: - -- [ ] Code blocks → syntax-highlighted (language-aware) -- [ ] Math blocks → styled LaTeX source (with optional Unicode symbol substitution: α, β, ∑) -- [ ] Directives (`{note}`, `{code-cell}`) → colored/boxed panels with directive name as header -- [ ] Headers/lists/links → standard markdown styling -- [ ] Frontmatter → YAML syntax highlighting -- [ ] Side-by-side or sequential SOURCE/TARGET display - ---- - -## Phase 3b: Resync CLI — Forward Resync (2-3 days) - -**Goal**: Drift recovery via forward resync with RESYNC translation mode + optional GitHub PR creation - -**Status**: ✅ Complete. Whole-file RESYNC implemented (§3b.5). Triage validated on real data. - -### 3b.0 Triage Experiment Results (5 March 2026) - -Ran forward triage on all 49 file pairs between `lecture-python-intro` and `lecture-intro.zh-cn`: - -| Verdict | Count | % | -|---------|-------|---| -| 🔄 CONTENT_CHANGES | 9 | 18% | -| 🌐 I18N_ONLY | 36 | 74% | -| ✅ IDENTICAL | 4 | 8% | - -**Cost**: ~$2.26 (754K tokens) for 49 files. Higher than the $0.01/file estimate because full documents are sent to the LLM as context. - -**The 9 content-change files with LLM-identified reasons**: - -| File | Issue | -|------|-------| -| about.md | Missing contributors in Credits section | -| business_cycle.md | Missing developed economies comparison section | -| cagan_adaptive.md | Missing section + different function names + code restructuring | -| equalizing_difference.md | Major code rewrite (namedtuple → class approach) | -| heavy_tails.md | Different API usage (pandas_datareader → wbgapi) | -| intro.md | Added author info in Chinese version | -| pv.md | Incomplete formulas + incorrect vector definition | -| supply_demand_heterogeneity.md | Formula error (^T rendered as ^2) | -| troubleshooting.md | Extra content in Chinese + different issue tracker URL | - -**Key findings**: -1. **Triage accuracy is high** — reasons are specific (formula errors, missing sections, API changes), no obvious false positives -2. **The triage reasons alone are valuable** — pv.md formula error and supply_demand_heterogeneity.md ^T→^2 are essentially bug reports -3. **82% of files can be skipped** — massive cost savings by filtering before translation -4. **Cost was 4× estimate** — real documents are large; actual triage cost is ~$0.05/file, not $0.01 - -### 3b.5 Design Decision: Whole-File vs Section-by-Section RESYNC ✅ DECIDED - -**Decision (5 March 2026)**: Use **whole-file RESYNC with glossary** for the `forward` command. Section-by-section remains for SYNC mode (PR-driven) where it's the right design. - -**Full experiment report**: `experiments/forward/whole-file-vs-section-by-section/REPORT.md` - -#### Experiment Results (pv.md — 7 sections, 458 lines, zh-cn) - -| Metric | Whole-file (glossary) | Section-by-section | Fresh translate | -|--------|----------------------|-------------------|----------------| -| Changed lines vs original | **29** | 52 | 188 | -| Total tokens | 23,905 | 72,681 | 10,600 | -| API calls | 1 | 7 | 1 | -| Estimated cost | **$0.137** | $0.281 | $0.098 | - -Both approaches made the same 5 correct fixes (formula error, missing content, exercise updates, Wikipedia link). The critical differences: - -1. **Localization preservation**: Whole-file preserved Chinese plot labels (`label='股息'`, `ax.set_xlabel('时间')`). Section-by-section reverted **all 4 plotting blocks** back to English — each section was translated in isolation without seeing the document's consistent Chinese localization pattern. - -2. **Cost**: Section-by-section used 3× more tokens because the 357-term glossary is sent with every section call (7× overhead). - -3. **Unnecessary churn**: Whole-file changed 29 lines (all intentional fixes). Section-by-section changed 52 lines (29 fixes + 23 regressions). - -#### Why section-by-section works for SYNC but not forward - -| Factor | SYNC (PR-driven) | Forward RESYNC | -|--------|------------------|----------------| -| Change signal | Git diff — exact sections | None — whole document question | -| Sections changed | 1-3 per PR | Unknown | -| Heading-map | Fresh (just updated) | May be stale or missing | -| Cross-section context | Not needed (surgical) | Critical (localization patterns) | -| Reconstruction | Simple (few sections) | Fragile (~300 lines of code) | - -#### Risks and mitigations - -- **Unwanted edits**: Prompt says "only modify where SOURCE content changed" — experiment showed 29/458 lines changed (94% preserved) -- **All-or-nothing failure**: Retry logic handles transient errors; single call is actually more reliable than N calls + reconstruction -- **Output verification**: `git diff` review before committing; natural workflow with `git restore .` to undo - -``` -npx resync status # identify drifted files (OUTDATED, SOURCE_AHEAD) -npx resync forward -f cobweb.md # resync specific file (local) -npx resync forward # resync all OUTDATED files (local) -npx resync forward --github # resync all, create one PR per file in TARGET repo -``` - -**Two execution modes**: -- `-f ` — single file -- *(none)* — bulk: all OUTDATED files detected by `status` - -**Two output modes**: -- Default — write updated TARGET files to local disk -- `--github` — create one PR per file in TARGET repo's default branch via `gh` - -**Per-file pipeline** (whole-file approach — decided in §3b.5): -1. **Forward triage** (LLM, ~$0.05/file) — "Content changes or i18n only?" - - Content changes → proceed to RESYNC - - i18n only → skip with brief reason (e.g., "punctuation and terminology style") -2. **Whole-file RESYNC translation** — send current SOURCE + current TARGET + glossary → get back updated TARGET -3. Output: write to disk or create PR - -- [x] Parse CLI arguments: `-f `, `-s `, `-t `, `-l `, `--github` -- [x] Support single-file mode (`-f`) and full directory mode -- [x] Integrate with `status` to auto-detect OUTDATED files (when no `-f` given) -- [x] Cost estimation via `--estimate` -- [x] `--github` mode: one PR per file, branch `resync/{filename}`, labels `action-translation-sync`, `resync` -- [x] **Refactor to whole-file RESYNC** — added `translateDocumentResync()`, simplified forward pipeline (572→371 lines) - -### 3b.2 Forward Triage (`src/cli/forward-triage.ts`) - -LLM-based content-vs-i18n filter. Runs on every file before RESYNC to avoid noise. - -``` -cobweb.md: RESYNCED (3 sections updated, 1 new, 8 unchanged) -solow.md: SKIPPED (i18n only — terminology style, full-width punctuation) -``` - -- [x] `triageForward()` — single LLM call per file, returns verdict + brief reason -- [x] Verdicts: `CONTENT_CHANGES` (proceed), `I18N_ONLY` (skip), `IDENTICAL` (skip) -- [x] Prompt: "Compare SOURCE and TARGET. Are there substantive content differences (structure, formulas, examples, code logic), or only internationalisation differences (punctuation, word choice, terminology style)?" -- [x] Report skip reason in summary (e.g., "punctuation and terminology style differences") -- [x] Unit tests + prompt snapshot tests -- [x] **Validated on real data** — 49 file pairs, results in §3b.0 above - -### 3b.3 RESYNC Translation Mode - -New translation mode distinct from NEW and UPDATE: - -| Mode | Inputs | Use case | -|------|--------|----------| -| **NEW** | SOURCE only | Fresh translation (no prior work) | -| **UPDATE** | Old SOURCE + New SOURCE + Current TARGET | Incremental change (PR-driven, Action) | -| **RESYNC** | Current SOURCE + Current TARGET | Drift recovery (no baseline available) | - -RESYNC preserves translation nuances because the LLM sees the existing translation: -- Maintains translator's style, terminology choices, localization decisions -- Only changes what the SOURCE actually changed -- Far less churn than re-translating from scratch - -- [x] Add RESYNC mode to `translator.ts` (`translateSectionResync()` method) -- [x] RESYNC prompt: "Update this translation to accurately reflect the current source. Preserve existing translation style, terminology, and localization wherever the meaning hasn't changed." -- [x] Section-by-section implementation (working, tested) -- [x] Handle `SOURCE_ONLY` sections (new — translate with NEW mode) -- [x] Handle `TARGET_ONLY` sections (deleted in SOURCE — flag for removal) -- [x] Preserve heading-map (via `heading-map.ts`) -- [x] Preserve frontmatter -- [x] Unit tests for RESYNC mode (4 tests) -- [x] **Evaluate whole-file RESYNC** — experiment completed, whole-file wins (see §3b.5) -- [x] Add `translateDocumentResync()` method to `translator.ts` -- [x] Refactor `forward.ts` to use whole-file RESYNC (eliminate parse/match/reconstruct) - -### 3b.4 Forward Output - -**Local mode** (default): -- [x] Write updated TARGET files to disk -- [x] Sync summary: sections resynced / unchanged / new / removed / errors / skipped (i18n) -- [x] No `--dry-run` — use git workflow: run forward, `git diff`, `git restore .` - -**GitHub mode** (`--github`): -- [x] Create branch `resync/{filename}` in TARGET repo -- [x] Commit updated file -- [x] Create PR: title `🔄 [resync] filename.md`, body with section change summary -- [x] Labels: `action-translation-sync`, `resync` -- [x] Print PR URL per file -- [x] Injectable `GhRunner` for testability (same pattern as `issue-creator.ts`) - -**Phase 3b Deliverable**: Working `npx resync forward` with whole-file RESYNC mode, local + `--github` output - -**Remaining**: None — Phase 3b complete. - ---- - -## Phase 4: Refinement & Documentation ✅ - -**Goal**: Production-ready CLI -**Status**: Core refinement complete (PR #24). Remaining items moved to Future Work. - -### 4.1 Testing (completed) - -- [x] Add CLI smoke tests (invoke commands as external processes) — 11 tests in `cli-smoke.test.ts` -- [x] Add LLM prompt snapshot tests (catch unintended prompt drift) — 5 snapshots across 3 suites - -### 4.3 Error Handling (completed) - -- [x] Malformed frontmatter — `parseTocLectures()` catches YAML parse errors and empty files -- [x] `gh` CLI not available — `checkGhAvailable()` pre-flight in review + forward --github, differentiates ENOENT/ETIMEDOUT/other - -### 4.6 Review Actions (completed) - -- [x] Update `@anthropic-ai/sdk` to latest version — 0.27.0 → 0.78.0 -- [x] Add Unicode heading ID test case — `\p{L}\p{N}` in parser.ts and reviewer.ts - ---- - -## Testing Strategy - -The CLI decouples translation logic from GitHub Actions infrastructure, enabling a layered testing approach from fast/cheap to realistic. - -### Testing Pyramid - -| Layer | Speed | Cost | What It Tests | When to Run | -|-------|-------|------|---------------|-------------| -| **Unit tests** | ~5ms each | Free | Individual functions (comparator, matcher, evaluator, generator) | Every commit | -| **Fixture integration** | ~50ms each | Free | Cross-language pipeline with paired repo fixtures | Every commit | -| **Snapshot tests** | ~10ms each | Free | Report format stability + prompt text stability | Every commit | -| **Git integration** | ~200ms each | Free | Git metadata extraction with temp repos | Every commit | -| **CLI smoke tests** | ~1s each | Free | Full command execution with `--test` flag | Every commit | -| **Real repo tests** | ~30s each | Free (test mode) | Full pipeline against real lecture repos | Pre-release | -| **LLM prompt regression** | ~5s each | ~$0.05 each | Prompt quality (golden responses) | Weekly / pre-release | -| **GitHub Action tests** | ~2min each | Free (test mode) | Full PR workflow via tool-test-action-on-github | Pre-release | - -### Layer 1: Unit Tests (existing pattern) - -Each CLI module gets a corresponding test file. CLI modules are pure functions (no GitHub API, no Actions context) making them easy to test thoroughly. - -**Two-stage specific tests**: -- `document-comparator.test.ts` — Stage 1 triage with mocked LLM, prompt construction -- `backward-evaluator.test.ts` — Stage 2 section analysis with mocked LLM -- Verify Stage 1 `IN_SYNC` result skips Stage 2 entirely -- Verify Stage 1 notes are passed through to Stage 2 prompts - -### Layer 2: Paired Fixture Repos (new — biggest opportunity) - -Current fixtures are same-language triplets (old English → new English → current Chinese). The CLI needs **cross-language pairs** where both sides are controlled: - -``` -src/cli/__tests__/fixtures/ -├── aligned-pair/ # Stage 1 should return IN_SYNC -│ ├── source/lectures/intro.md -│ └── target/lectures/intro.md # faithful translation + heading-map -├── bug-fix-in-target/ # Stage 1 flags, Stage 2 detects BUG_FIX -│ ├── source/lectures/cobweb.md -│ └── target/lectures/cobweb.md # fixed formula error -├── i18n-only-changes/ # Stage 1 might flag, Stage 2 filters out (I18N_ONLY) -│ ├── source/lectures/growth.md -│ └── target/lectures/growth.md # only font/punctuation changes -├── missing-heading-map/ # Tests graceful degradation -├── section-count-mismatch/ # TARGET has extra section -└── structural-drift/ # Sections reordered -``` - -The `status` command can be tested entirely with fixtures. The `backward` command uses fixtures + a mock LLM evaluator. - -### Layer 3: Snapshot Tests (new — natural fit for reports + prompts) - -Two snapshot targets: - -**a) Report output** — `report-generator.ts` produces markdown and JSON reports: - -```typescript -it('generates correct markdown report for bug-fix suggestion', () => { - const report = generateMarkdownReport('cobweb.md', suggestions, metadata); - expect(report).toMatchSnapshot(); -}); -``` - -**b) Prompt text** — catch unintended prompt changes: - -```typescript -it('constructs correct Stage 1 triage prompt', () => { - const prompt = buildTriagePrompt(sourceContent, targetContent, metadata); - expect(prompt).toMatchSnapshot(); -}); - -it('constructs correct Stage 2 evaluation prompt', () => { - const prompt = buildEvaluationPrompt(sourceSection, targetSection, metadata, triageNotes); - expect(prompt).toMatchSnapshot(); -}); -``` - -### Layer 4: Git Integration Tests (new — tests real git plumbing) - -`git-metadata.ts` calls `git log`. Test with **temporary git repos** created in test setup: - -```typescript -beforeEach(async () => { - tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'resync-test-')); - await exec('git init', { cwd: tmpDir }); - await fs.writeFile(path.join(tmpDir, 'test.md'), 'content'); - await exec('git add . && git commit -m "initial"', { cwd: tmpDir }); -}); -``` - -Tests actual git plumbing without needing a GitHub remote. Fast and deterministic. - -### Layer 5: CLI Smoke Tests (new — validates full commands) - -Test the CLI binary as an external process: - -```typescript -it('backward command produces report for single file', async () => { - const result = await exec( - `npx resync backward -f intro.md -s ${sourceDir} -t ${targetDir} -o ${tmpDir} --test` - ); - expect(result.exitCode).toBe(0); - expect(fs.existsSync(path.join(tmpDir, 'intro-backward.md'))).toBe(true); -}); -``` - -A `--test` flag (like existing TEST mode) skips real LLM calls and uses deterministic responses. Validates argument parsing, file I/O, and full two-stage orchestration. - -### Layer 6: Real Repo Tests (extends tool-test-action-on-github) - -The CLI makes testing against real repos much easier — no GitHub Actions pipeline needed: - -```bash -# Test backward analysis against real lecture repos (local clones) -npx resync backward \ - -s ~/repos/lecture-python-intro \ - -t ~/repos/lecture-python-intro.zh-cn \ - -f cagan_adaptive.md \ - --test \ - -o /tmp/test-reports - -# Compare output against known-good baseline -diff /tmp/test-reports/cagan_adaptive-backward.md baseline/ -``` - -Can be scripted into a test suite that runs against real repos without GitHub infrastructure. - -### Layer 7: LLM Prompt Regression Tests (new — catches prompt drift) - -Two complementary approaches: - -**a) Prompt snapshot tests** (free, every commit) — snapshot the *constructed prompt* to catch unintended changes: - -```typescript -it('constructs correct backward evaluation prompt', () => { - const prompt = buildEvaluationPrompt(sourceSection, targetSection, metadata, notes); - expect(prompt).toMatchSnapshot(); -}); -``` - -**b) Golden response tests** (costs ~$0.05/test, weekly) — verify real LLM responses for known fixtures produce expected verdicts: - -```typescript -it('Stage 1 correctly flags file with bug fix', async () => { - const result = await triageDocument(sourceContent, targetContent, metadata); - expect(result.verdict).toBe('CHANGES_DETECTED'); -}); - -it('Stage 2 correctly identifies bug fix category', async () => { - const result = await evaluateSection(sourceSection, targetSection, metadata, notes); - expect(result.recommendation).toBe('BACKPORT'); - expect(result.category).toBe('BUG_FIX'); - expect(result.confidence).toBeGreaterThan(0.8); -}); -``` - -Run as a separate test suite, not on every commit. - -### Test Organization - -``` -src/cli/__tests__/ -├── document-comparator.test.ts # Layer 1: unit + Layer 7a: prompt snapshots -├── section-matcher.test.ts # Layer 1: unit -├── backward-evaluator.test.ts # Layer 1: unit + Layer 7a: prompt snapshots -├── git-metadata.test.ts # Layer 4: temp git repos -├── report-generator.test.ts # Layer 3: snapshot tests -├── status.test.ts # Layer 2: fixture integration -├── backward.test.ts # Layer 2: fixture integration + Layer 5: CLI smoke -├── review.test.ts # Layer 2: interactive flow tests + Issue generation -├── forward.test.ts # Layer 2: fixture integration + Layer 5: CLI smoke -├── prompt-regression.test.ts # Layer 7b: golden responses (separate suite, --test-llm flag) -└── fixtures/ - ├── aligned-pair/ - ├── bug-fix-in-target/ - ├── i18n-only-changes/ - ├── missing-heading-map/ - ├── section-count-mismatch/ - └── structural-drift/ -``` - -### Running Test Suites - -```bash -# All fast tests (Layers 1-5, every commit) -npm test - -# LLM prompt regression tests (Layer 7b, weekly/pre-release) -npm run test:llm - -# Real repo tests (Layer 6, pre-release) -npm run test:real-repos - -# GitHub Action tests (Layer 8, pre-release) -./tool-test-action-on-github/test-action-on-github.sh -``` - ---- - -## Phase 5: CLI Rename & Init Command - -**Goal**: Rename CLI from `resync` to `translate`, integrate bulk translation as `init` command, simplify flags -**Status**: ✅ Complete (PR #23) - -### Design Decisions (6 March 2026) - -**CLI rename `resync` → `translate`**: The tool's domain is translation management. `translate` is immediately understandable and works naturally as a prefix for all subcommands: `translate init`, `translate forward`, `translate backward`, `translate status`, `translate review`. - -**`init` command**: Incorporates `tool-bulk-translator` functionality directly into the CLI. One-time bulk translation of an entire lecture series from a local source repo. Uses `translateFullDocument()` from `translator.ts` (same as bulk-translator), reads `_toc.yml` for lecture discovery, generates heading-maps, produces a translation report. - -**Local paths only**: Unlike `tool-bulk-translator` which fetched from GitHub via Octokit, `init` uses local paths (`-s`/`-t`) consistent with all other CLI commands. No `@octokit/rest` dependency needed. User clones repos first. - -**`--dry-run` over `--estimate`**: These tools run infrequently. `--dry-run` (lists what would be done, no API calls, no file writes) is more useful for understanding and debugging than a cost estimate. Remove `--estimate` from `backward` and `forward` commands; add `--dry-run` to `init`. - -**`setup` command (Phase 5c)**: Scaffold a new target repo by appending language code to source repo name and using `gh` CLI. Separate from `init` (which does translation). Kept as a follow-up PR to keep Phase 5 scope focused. - -### 5.1 CLI Rename (`resync` → `translate`) ✅ - -- [x] Update `package.json` `bin` entry: `resync` → `translate` -- [x] Update `src/cli/index.ts` `.name('resync')` → `.name('translate')` -- [x] Update `src/cli/index.ts` description -- [x] Update all `npx resync` references in docs (`cli-reference.md`, `README.md`, etc.) -- [x] Update `copilot-instructions.md` CLI references -- [ ] Update PLAN.md `npx resync` references (historical — left as-is in earlier phases) - -### 5.2 Init Command (`src/cli/commands/init.ts`) ✅ - -Adapted from `tool-bulk-translator/src/bulk-translate.ts` with local-path approach. - -``` -translate init -s /path/to/source -t /path/to/target \ - --target-language zh-cn \ - [--docs-folder lectures] \ - [--model claude-sonnet-4-6] \ - [--batch-delay 1000] \ - [--resume-from cobweb.md] \ - [--dry-run] -``` - -**Pipeline** (7 phases from bulk-translator, adapted): -1. Load glossary (built-in `glossary/.json`) -2. Setup target folder -3. Copy non-markdown files from local source (replaces GitHub API fetch) -4. Parse `_toc.yml` from local source for lecture list -5. Translate lectures sequentially (with retry, batch delay) -6. Generate heading-maps per file -7. Generate `TRANSLATION-REPORT.md` - -- [x] Create `src/cli/commands/init.ts` -- [x] Register `init` command in `src/cli/index.ts` -- [x] Add `InitOptions` in `src/cli/commands/init.ts` -- [x] Implement local file copy (no Octokit) -- [x] Implement local `_toc.yml` parsing -- [x] Implement sequential translation with progress bar -- [x] Implement heading-map generation (reuse from bulk-translator) -- [x] Implement report generation -- [x] `--dry-run` mode (list lectures, no API calls, no file writes) -- [x] `--resume-from` support -- [x] `--skip-existing` — skip lectures already in `.translate/state/`, enabling idempotent re-runs (PR #34) -- [x] `-j, --parallel` — concurrent translation with configurable worker count (PR #33) -- [x] Add tests (16 tests: `parseTocLectures`, `copyNonMarkdownFiles`) - -### 5.3 Remove `--estimate` Flag ✅ - -- [x] Remove `--estimate` from `backward` command in `index.ts` -- [x] Remove `--estimate` from `forward` command in `index.ts` -- [x] Remove `estimate` from `BackwardOptions` and `ForwardOptions` in `types.ts` -- [x] Remove `estimateBulkCost()` from `backward.ts` -- [x] Remove `estimateCost()` from `forward.ts` -- [x] Update tests that reference `--estimate` -- [x] Update `cli-reference.md` - -### 5.4 Documentation ✅ - -- [x] `cli-reference.md`: full `init` command section with options, pipeline, examples -- [x] `cli-reference.md`: rename all `resync` → `translate`, remove `--estimate` -- [x] `README.md`: update CLI examples -- [x] `quickstart.md`: update CLI references -- [x] `architecture.md`: add `init.ts` to module tree, update CLI references -- [x] `CHANGELOG.md`: Phase 5 entries - ---- - -## Phase 5b: Cleanup & Repo Hygiene ✅ - -**Goal**: Clean up deprecated tools and repo structure - -- [x] Document `tool-onboarding` and `tool-alignment` in `docs/developer/legacy-tools.md` -- [x] Deprecate `tool-onboarding/` (add deprecation notice to README) -- [x] Deprecate `tool-alignment/` (add deprecation notice to README) -- [x] Remove `tool-onboarding/` and `tool-alignment/` from tree (preserved in git history) -- [x] Remove `tool-bulk-translator/` (functionality moved to `translate init`) — git rm -r, preserved in history -- [x] Clean up `.gitignore` — removed stale `tool-bulk-translator/dist/` entry, removed `*.test.ts.snap` (snapshots tracked for CI) -- [x] Remove `coverage/` from tracked files — already untracked (in `.gitignore`, 0 files in git index) -- [x] ~~Clean up `dist/` build output~~ — N/A: `dist-action/` must be committed (GitHub Action entry point) -- [x] Update `copilot-instructions.md` to reflect new CLI structure — test counts updated in PR #24 - ---- - -## Phase 6: `.translate/` Metadata Folder - -**Goal**: Add minimal persistent metadata to the target repo so the CLI can make exact staleness decisions, skip redundant work, and record translation provenance — without adding complexity. -**Status**: In progress (PR pending) -**Prerequisite**: Phase 5 (PR #23) - -### Motivation - -Today all sync state is either ephemeral (recomputed every run) or approximated from git history. This has three pain points: - -1. **Staleness is a guess** — `status` and `backward` estimate sync state from commit timestamps, which breaks if someone edits a target file for non-translation reasons (formatting fix, typo). -2. **No skip optimisation** — `backward` evaluates every file every run, even when source hasn't changed since last evaluation. For a 51-file repo at ~$0.01/file, that's wasted cost on repeated runs. -3. **No provenance** — after translation you can't tell what model produced it, or what source state it was translated from. - -### Design Principles - -- **Minimum viable metadata** — track only what enables concrete decisions. No speculative fields. -- **Lives in TARGET repo only** — source repo stays untouched. -- **Committed to git** — metadata is part of the translation record (not ephemeral cache). -- **Heading-map stays in frontmatter** — it travels with the document and serves as a quick structural verifier. `.translate/` complements it, doesn't replace it. -- **Graceful absence** — if `.translate/` doesn't exist, all commands work exactly as today (git-heuristic fallback). Existing projects are unaffected. - -### Structure - -``` -.translate/ -├── config.yml # Project-level settings -└── state/ - ├── intro.md.yml # Per-file sync metadata - ├── cobweb.md.yml - └── solow.md.yml -``` - -**Project config** (`.translate/config.yml`): -```yaml -source-language: en -target-language: zh-cn -docs-folder: lectures -``` - -This replaces the need to pass `--source-language`, `--target-language`, `--docs-folder` on every CLI invocation. Commands read config as defaults, flags override. - -**Per-file state** (`.translate/state/.yml`): -```yaml -source-sha: abc1234f # Source file's commit SHA at time of sync -synced-at: 2026-03-06 # Explicit sync timestamp (ISO date) -model: claude-sonnet-4-6 # Model used for translation -mode: RESYNC # Translation mode: NEW / UPDATE / RESYNC -section-count: 5 # Source section count at sync time -``` - -### What this enables - -| Capability | Today | With `.translate/` | -|---|---|---| -| "Is this file stale?" | Git timestamp heuristic | Exact: `git log --since` on source-sha | -| "Skip unchanged files" | Can't — re-evaluates all | Compare source-sha to HEAD, skip if unchanged | -| "What model translated this?" | Unknown | Recorded per-file | -| Default CLI flags | Must pass every time | Read from `config.yml` | -| "Translation coverage" | Computed live each run | Instant: count files with state entries | - -### How each command uses `.translate/` - -- **`translate init`** — Creates `.translate/config.yml` + per-file state entries after each lecture is translated. Natural first producer. -- **`translate status`** — If state exists, uses source-sha for exact comparison instead of git heuristic. Falls back to current behaviour if absent. -- **`translate backward`** — Skips files where source-sha matches current HEAD (source unchanged since last sync). Saves LLM cost on repeated runs. -- **`translate forward`** — Updates state entry after successful resync. -- **GitHub Action (sync mode)** — Reads config for defaults. Updates state after successful translation PR. -- **`translate setup`** — Creates `.translate/config.yml` as part of repo scaffolding. - -### Bootstrap / Migration - -For existing paired projects that predate `.translate/`, bootstrap state via the `status` command: - -```bash -translate status \ - -s /path/to/source \ - -t /path/to/target \ - --target-language zh-cn \ - --write-state -``` - -`--write-state` adds a one-time side effect to the normal `status` run: -1. Create `.translate/config.yml` from the provided flags -2. For each translated file, find the most recent target commit and use it as `synced-at` -3. Record the source SHA at that point as a best-effort `source-sha` -4. Mark `model: unknown` (not recoverable from history) - -After bootstrap, normal commands (`init`, `forward`, Action) maintain state automatically. No new command needed — `status` already walks both repos and compares structure, so it has all the information required. - -### Tasks - -- [x] Define `TranslateConfig` and `FileState` types in `src/cli/types.ts` -- [x] Create `src/cli/translate-state.ts` — read/write `.translate/` config and state - - `readConfig(targetPath)` → `TranslateConfig | undefined` - - `readFileState(targetPath, filename)` → `FileState | undefined` - - `writeFileState(targetPath, filename, state)` → void - - `writeConfig(targetPath, config)` → void -- [x] Update `translate init` to write config + per-file state after each lecture -- [x] Update `translate status` to use source-sha when available -- [x] Add `--write-state` flag to `translate status` for bootstrap / migration -- [x] Update `translate backward` to skip unchanged files (source-sha check) -- [x] Update `translate forward` to write state after resync -- [x] Update GitHub Action sync mode to write/update state after successful translation -- [x] Add tests for state read/write, skip logic, bootstrap via `--write-state` -- [x] Add `.translate/` section to `cli-reference.md` - ---- - -## Phase 6b: Setup Command (Future PR) - -**Goal**: Scaffold a new target repo so `translate init` has somewhere to translate into -**Status**: In progress (PR pending) -**Prerequisite**: Phase 5 (PR #23) - -**Concept**: `translate setup` creates and initialises a target translation repository. It pairs with `init` to provide the complete onboarding workflow: `setup` → `init` → push → configure Action. - -```bash -# Create target repo and local clone -translate setup \ - --source QuantEcon/lecture-python-intro \ - --target-language zh-cn - -# Then translate into it -translate init \ - -s ~/repos/lecture-python-intro \ - -t ~/repos/lecture-python-intro.zh-cn \ - --target-language zh-cn -``` - -### What `setup` would do - -1. **Derive target repo name**: `{source-repo}.{lang}` (e.g., `lecture-python-intro.zh-cn`) -2. **Create GitHub repo**: `gh repo create {owner}/{target-name} --public --clone` -3. **Copy repo scaffolding**: `.github/workflows/`, `LICENSE`, `.gitignore` -4. **Create initial `_config.yml`**: Set title, language metadata -5. **Create translation workflow file**: Pre-configured `action-translation` sync workflow -6. **Initial commit and push** - -### Tasks - -- [x] Create `src/cli/commands/setup.ts` -- [x] Register `setup` command in `src/cli/index.ts` -- [x] Implement repo name derivation -- [x] Implement `gh repo create` integration (injectable `GhRunner` for testing) -- [x] Implement scaffolding file generation -- [x] Implement workflow template generation -- [x] `--dry-run` mode (show what would be created) -- [x] Add tests -- [x] Add docs to `cli-reference.md` - ---- - -## Phase 7: Real-World Readiness - -**Goal**: Close gaps identified during full lifecycle review, fix documentation drift, create tutorials, fix `setup` workflow generation, add missing CLI commands, and validate the full lifecycle with end-to-end testing against real repositories. -**Status**: Complete (PR #30 + PR #31 + PR #32) — All sub-phases done including 7.7 E2E testing and GitHub Action integration testing. -**E2E Testing**: Test Plan 2 (lecture-python-programming, 2026-03-18) + Test Plan 3 (test-translation-sync, 2026-03-19) validated. 5 bugs fixed (PR #31), 879 tests passing. -**GitHub Action Integration**: Both zh-cn and Farsi workflows pass end-to-end on test repos (2026-03-19). `tool-test-action-on-github` extended with Farsi support (commit 85afafa) + QA fixes (PR #32). -**Comprehensive Evaluation (2026-03-19)**: 24-scenario test plan across 3 repos (source + zh-cn + fa) — 48 target PRs, 100% trigger reliability, 100% diff correspondence, zh-cn quality ~9.6/10, fa quality ~9.3/10. Full report: `reports/test-translation-sync/evaluation-2026-03-19.md`. -**Prerequisite**: Phase 6 + Phase 6b - -### Context — Lifecycle Review (2026-03-16) - -A full review of the SOURCE ↔ TARGET lifecycle identified seven gaps in tooling and documentation. The complete tool lifecycle is: - -``` -SETUP INIT ONGOING SYNC MAINTENANCE -(scaffold target) (bulk translate) (automated pipeline) (CLI analysis) - -translate setup translate init GitHub Action translate status - (sync mode on merge) translate backward - (review mode on PR) translate review - translate forward -``` - -The `.translate/state/` directory links all phases — `setup` plants the config, `init` writes per-file state, `status` reads/bootstraps it, `backward` uses it to skip unchanged files, and `forward` updates it after resyncing. - -### Gap Analysis - -| # | Gap | Severity | Resolution | -|---|---|---|---| -| 1 | **`setup` only scaffolds TARGET workflow** — no SOURCE workflow generated | Medium | Add `--source-workflow` flag to `setup` (7.1) | -| 2 | **`setup` workflow uses `repository_dispatch`** but quickstart shows `pull_request: closed` — inconsistent trigger architectures | Medium | Standardise on `pull_request: closed` pattern (7.1) | -| 3 | **No standalone heading-map generation** — connecting existing repos requires either `forward` (re-translates, expensive) or manual heading-map creation | Medium | Add `translate headingmap` command (7.2) | -| 4 | **FAQ references removed `--estimate` flag** | Low | Fixed (2026-03-16) ✅ | -| 5 | **`docs/index.md` referenced old CLI name `resync`** | Low | Fixed (2026-03-16) ✅ | -| 6 | **Test counts stale in `docs/index.md`** (724 → 824) | Low | Fixed (2026-03-16) ✅ | -| 7 | **No validation/health-check command** to verify target repo is fully configured | Low | Add `translate doctor` command (7.3) | - -### 7.1 Improve `setup` Workflow Generation - -The current `setup` command generates a `repository_dispatch` workflow in the TARGET repo. But the actual sync trigger runs from the SOURCE repo (on `pull_request: closed`). This means: - -- The user must still manually create the SOURCE workflow — the most confusing step for new users -- The TARGET workflow template uses a different trigger architecture than the documented quickstart - -**Tasks:** - -- [x] Change `setup` to generate the SOURCE workflow file (`sync-translations.yml`) as a local file that the user can copy to their source repo, or print it to console with copy instructions -- [x] Change the TARGET workflow to use the standard `pull_request: closed` trigger pattern (matching quickstart docs) -- [x] Add `--source-workflow ` flag: write the source workflow YAML to a file (e.g., `--source-workflow ~/repos/source/.github/workflows/sync-translations.yml`) -- [x] Print clear post-setup instructions for both repos (secrets, workflow placement) -- [x] Update `setup` tests -- [x] Update `cli-reference.md` for `setup` - -### 7.2 Standalone Heading-Map Generation (`translate headingmap`) - -Currently, the only ways to get heading-maps are: -- `init` (generates them during bulk translation — new projects only) -- `forward` (generates them during RESYNC — re-translates the whole file, ~$0.12) -- Manual creation (tedious and error-prone) - -For **connecting existing targets** (Scenario 2), we need a free, local-only tool that generates heading-maps by comparing source and target section headings by position — no LLM calls needed. - -**Tasks:** - -- [x] Create `src/cli/commands/headingmap.ts` -- [x] `translate headingmap -s -t ` — bulk generate heading-maps for all files -- [x] `translate headingmap -s -t -f ` — single file mode -- [x] Pipeline: parse both files → match sections by position → build heading-map → inject into target frontmatter -- [x] `--dry-run` flag: show what heading-maps would be generated (print to console without modifying files) -- [x] Handle mismatched section counts: warn and generate partial map for matched sections -- [x] Register in `src/cli/index.ts` -- [x] Add tests (section matching, mismatch handling, frontmatter injection, existing heading-map update) -- [x] Add to `cli-reference.md` -- [x] Update connect-existing tutorial to reference this command - -### 7.3 Health Check (`translate doctor`) - -A diagnostic command that verifies a target repo is fully configured for action-translation. Like `brew doctor` or `flutter doctor`. - -**Tasks:** - -- [x] Create `src/cli/commands/doctor.ts` -- [x] `translate doctor -t ` — check target repo health -- [x] Checks: - - [x] `.translate/config.yml` exists and is valid - - [x] `.translate/state/` has entries for all target `.md` files - - [x] All target files have `heading-map` in frontmatter - - [x] Source repo is accessible (if `-s ` provided) - - [x] Section counts match between source and target - - [x] GitHub workflow file exists (`.github/workflows/`) - - [x] `gh` CLI is available and authenticated (if `--github` mode) -- [x] Traffic-light output: ✅ pass, ⚠️ warning, ❌ fail for each check -- [x] `--json` flag for CI/scripting -- [x] Register in `src/cli/index.ts` -- [x] Add tests -- [x] Add to `cli-reference.md` - -### 7.4 Fix `setup` Workflow Trigger Architecture - -The `generateWorkflowYaml()` function in `setup.ts` generates a `repository_dispatch` trigger. This should be updated to generate the `pull_request: closed` trigger (consistent with quickstart and tutorials). The `repository_dispatch` pattern is an internal implementation detail that adds confusion. - -**Tasks:** - -- [x] Update `generateWorkflowYaml()` to produce `pull_request: closed` trigger with `paths` filter -- [x] Requires `source-repo` to know the docs-folder path pattern -- [x] Update test snapshots -- [x] Verify the generated workflow against the quickstart template - -### 7.5 Tutorials - -Three tutorials created (2026-03-16): `docs/user/tutorials/` - -- [x] **Fresh Setup** (`fresh-setup.md`): `setup` → `init` → configure workflows → test pipeline -- [x] **Connect Existing Target** (`connect-existing.md`): assess → bootstrap `.translate/` → fix heading-maps → configure workflows -- [x] **Resync Drifted Target** (`resync-drifted.md`): diagnose → understand changes → resync → review → verify - -Additional tutorials: - -- [x] **Backward Analysis & Review** (`backward-review.md`): Full workflow — run backward → review suggestions → create Issues → fix source → verify sync -- [x] **Adding a New Language** (`add-language.md`): Create glossary + language config + target repo + workflows for a new language (e.g., Japanese) -- [x] **Automated Maintenance** (`automated-maintenance.md`): Set up scheduled `status` and `backward` runs via GitHub Actions (ties into Phase 8) - -### 7.6 Documentation Consistency Pass - -- [x] Fix FAQ `--estimate` references (removed in Phase 5) -- [x] Fix `docs/index.md` CLI name (`resync` → `translate`) -- [x] Fix `docs/index.md` test counts (724 → 824, 32 → 37) -- [x] Add tutorials section to `docs/index.md` -- [x] Update `copilot-instructions.md` test counts to 824 / 37 suites -- [x] Audit all docs for `resync` → `translate` rename consistency -- [x] Update `docs/index.md` CLI description to include `init` and `setup` -- [x] Verify all cross-references between tutorials and CLI reference - -### 7.7 End-to-End Smoke Test - -Validate the complete lifecycle against real GitHub repositories. This is the definitive test that all phases work together. - -**Test Plan: Full Lifecycle (Fresh Setup)** - -Using `lecture-python-intro` as the source repo and a new `lecture-python-intro.test-zh-cn` as a throwaway target: - -1. **Setup** — `translate setup --source QuantEcon/lecture-python-intro --target-language zh-cn` - - [ ] Verify target repo created on GitHub - - [ ] Verify `.translate/config.yml` generated correctly - - [ ] Verify workflow files generated (both SOURCE and TARGET) - - [ ] Verify `_config.yml` has correct language metadata - -2. **Init** — `translate init -s ~/repos/lecture-python-intro -t ~/repos/lecture-python-intro.test-zh-cn --target-language zh-cn` - - [ ] Verify all lectures translated (check for non-empty `.md` files) - - [ ] Verify heading-maps generated in every file's frontmatter - - [ ] Verify `.translate/state/` entries created for each file - - [ ] Verify `TRANSLATION-REPORT.md` generated - - [ ] Spot-check 2-3 translations for quality - -3. **Push + Action trigger** — Push target repo, make a small edit to source, open and merge a PR - - [ ] Verify the sync Action triggers on PR merge - - [ ] Verify a translation PR appears in the target repo - - [ ] Verify the PR contains only the changed sections (UPDATE mode) - - [ ] Verify heading-map updated if headings changed - - [ ] Verify `.translate/state/` updated in the PR - -4. **Status** — `translate status -s ~/repos/lecture-python-intro -t ~/repos/lecture-python-intro.test-zh-cn` - - [ ] Verify status correctly reports ALIGNED for files that haven't changed - - [ ] Verify status correctly reports OUTDATED for files edited in source since last sync - - [ ] Verify `--write-state` bootstrap works (delete `.translate/state/`, re-bootstrap) - -5. **Backward + Review** — `translate backward -s ... -t ... -o ./reports` then `translate review ./reports/lecture-python-intro.test-zh-cn/backward-YYYY-MM-DD --repo QuantEcon/lecture-python-intro` - - [ ] Verify backward report generated under `reports//backward-/` - - [ ] Verify review interactive UI launches and displays suggestions - - [ ] Verify Issue creation works (if suggestions accepted) - -6. **Forward** — Make a manual edit to target, then `translate forward -s ... -t ... -f ` - - [ ] Verify RESYNC re-translates the file - - [ ] Verify heading-map regenerated - - [ ] Verify `.translate/state/` updated - -7. **Doctor** — `translate doctor -t ~/repos/lecture-python-intro.test-zh-cn` - - [ ] Verify all checks pass on the fully-configured target repo - - [ ] Intentionally break something (delete a state file) and verify doctor catches it - -**Cleanup:** -- [ ] Delete test target repo after validation (`gh repo delete`) -- [ ] Document any bugs found → open Issues -- [ ] Record any prompt improvements needed - -**Test Plan: Connect Existing Target** - -Primary test pair: `lecture-python-programming.myst` ↔ `lecture-python-programming.fa` (Farsi, not yet launched) - -**0. Repo Assessment (2026-03-16)** ✅ - -| Dimension | State | -|-----------|-------| -| File coverage | 25 files — perfect 1:1 match, no gaps | -| Translation state | All 25 files contain Farsi content (previously translated) | -| Heading-maps | 19 of 25 files have heading-maps | -| Missing heading-maps | `about_py.md`, `functions.md`, `getting_started.md`, `intro.md`, `python_by_example.md`, `status.md` | -| `.translate/` state | Does not exist — no sync metadata | -| Sync workflows | None — only build/publish workflows (`cache.yml`, `ci.yml`, `publish.yml`) | -| docs-folder | `lectures` | - -1. **Status** — `translate status -s ... -t ... -l fa -d lectures` ✅ (2026-03-18) - - [x] Run initial status diagnostic — 18 ALIGNED, 7 MISSING_HEADINGMAP - - [x] Found YAML parse error on `python_essentials.md` (unquoted colon in hand-written heading-map) - - [x] Confirmed `about_py.md` had ~20% drift from source - -2. **Bootstrap `.translate/` state** — `translate status ... --write-state` ✅ (2026-03-18) - - [x] `config.yml` created (source-language: en, target-language: fa, docs-folder: lectures) - - [x] 25 state files created with best-effort `source-sha` - - [x] **Bug found**: `tool-version: unknown` — npx bin shim issue (Fixed: Strategy 3 in `getToolVersion()`) - -3. **Fix missing heading-maps** — `translate headingmap` ✅ (2026-03-18) - - [x] 23 of 25 files now ALIGNED (2 section-less files: `intro.md`, `status.md`) - - [x] Existing heading-maps preserved, new ones generated by position matching - - [x] **Bug found**: Malformed YAML fallback left orphaned entries (Fixed: line-by-line parser) - - [x] **Bug found**: `headingmap` didn't update `section-count` in state (Fixed: state sync added) - -4. **Forward selective resync** ✅ (2026-03-18) - - [x] `scipy.md` correctly SKIPPED (IDENTICAL — aligned) - - [x] `about_py.md` correctly detected as CONTENT_CHANGES after triage fix → resynced - - [x] Controlled test: added section to `troubleshooting.md`, forward correctly detected and resynced - - [x] **Bug found**: Triage prompt too permissive — `about_py.md` was I18N_ONLY despite 20% drift (Fixed: tightened prompt) - -5. **Backward** ✅ (2026-03-18) - - [x] `about_py.md`: CHANGES_DETECTED, 1 BACKPORT suggestion (CLARIFICATION, 0.65 confidence) - - [x] `scipy.md`: IN_SYNC, no suggestions (correct — no false positives) - - [x] Reports written as both markdown and `.resync/` JSON sidecars - -6. **Review** ✅ (2026-03-18) - - [x] `translate review .tmp/reports --dry-run` — loaded 2 reports, 1 actionable suggestion - - [x] Interactive ink TUI launched with Accept/Skip/Reject controls - - [x] Dry-run summary correct ("Would have created 1 GitHub Issue") - -7. **Doctor** ✅ (2026-03-18) - - [x] 5 pass, 1 warning (expected: heading maps missing for section-less files) - - [x] Workflow detected after adding `review-translations.yml` - -**Bugs found (5, all fixed in PR #31):** -- `getToolVersion()`: npx bin shim → Strategy 3 (process.cwd()) -- `injectHeadingMap()`: Orphaned YAML entries → line-by-line parser (reviewed: removed allowlist per Copilot) -- `headingmap` state sync: section-count not updated → added `readFileState`/`writeFileState` -- Forward triage prompt: Too permissive I18N_ONLY → explicit examples, path detection, safety rule -- Source language: Hardcoded `'English'`/`'en'` → `--source-language` option with config fallback - -**Test Plan 3: Fresh Setup with Test Repos** (2026-03-19) - -Using dedicated test repos: `test-translation-sync` (source, 2 lectures) ↔ `test-translation-sync.zh-cn` (existing Chinese) + new `test-translation-sync.fa` (Farsi) - -1. **Status** on source ↔ zh-cn ✅ — 2 ALIGNED -2. **Setup** — `translate setup --source QuantEcon/test-translation-sync --target-language fa` ✅ - - [x] Repo `QuantEcon/test-translation-sync.fa` created on GitHub - - [x] `.translate/config.yml` generated (source-language: en, target-language: fa, docs-folder: .) - - [x] `review-translations.yml` workflow scaffolded in target - - [x] Source sync workflow generated via `--source-workflow` -3. **Init** — `translate init -s ... -t ... --target-language fa` ✅ - - [x] 2 of 3 lectures translated (index.md missing from source — expected) - - [x] Heading-maps generated in both files - - [x] `.translate/state/` entries created with correct section-count - - [x] `TRANSLATION-REPORT.md` generated (25K tokens, ~1 min) - - [x] Quality: Clean Farsi translations, math/code preserved, heading-maps correct -4. **Status** on source ↔ fa ✅ — 2 ALIGNED, 2 TARGET_ONLY (README, report) -5. **Doctor** on fa ✅ — 4 pass, 2 warnings (expected: README/report not tracked) -6. **Push** ✅ — Translations pushed to main, source workflow as [PR #592](https://github.com/QuantEcon/test-translation-sync/pull/592) -7. **GitHub Action testing** ✅ (2026-03-19) - - [x] PR #592 (Farsi sync workflow) merged — updated to `@v0.9.0`, `QUANTECON_SERVICES_PAT`, `types: [closed, labeled]` + `test-translation` label trigger - - [x] `tool-test-action-on-github` extended with Farsi support: `workflow-template-fa.yml`, `base-minimal-fa.md`, `base-lecture-fa.md`, `base-toc-fa.yml`, step 2b reset, step 3 PR close, README table updated - - [x] Created test PR #594 (`lecture-minimal.md` edit, `test-translation` label) - - [x] **Translation Sync (Farsi)** — ✅ SUCCESS (22s, run 23281110629) → created PR #1 on `test-translation-sync.fa` - - [x] **Translation Sync (zh-cn)** — ✅ SUCCESS (1m12s, run 23281110636) → created PR #559 on `test-translation-sync.zh-cn` - - [x] Both PRs verified: correct translation content, `action-translation`+`automated` labels, `.translate/state/` updated with `tool-version: 0.9.0` - - [x] QA fixes committed (PR #32): missing heading-map entries in Farsi base files, zh-cn reset now clears `.translate/`, `workflow-template.yml` test-mode pattern aligned with fa template - -### Comprehensive Evaluation — 24-Scenario Test (2026-03-19) - -Full structured evaluation of v0.9.0 across 24 source PRs (#595–#618), each triggering translation PRs in both zh-cn and fa target repos — **48 target PRs total**. - -**Test repos**: `QuantEcon/test-translation-sync` (source) → `QuantEcon/test-translation-sync.zh-cn` + `QuantEcon/test-translation-sync.fa` - -**Scenario coverage** (24 tests): - -| Category | Tests | Scenarios | -|----------|-------|-----------| -| Minimal document | 01–08, 16, 21, 24 | Intro/title/section edits, reorder, add/remove section, subsection, multiple elements, preamble, empty sections | -| Lecture document | 09–15, 22, 23 | Real-world update, sub-subsection add/edit/delete, code comments, math equations, deep nesting (##### ######), special characters | -| Structural | 17–20 | New document + TOC, document deleted + TOC, multi-file, rename + TOC | - -**Results scorecard**: - -| Category | Score | -|----------|-------| -| Trigger reliability | 24/24 (100%) — every source PR generated both target PRs | -| PR mapping accuracy | 48/48 (100%) — all target PRs correctly reference source | -| Diff intent match | 48/48 (100%) — all target diffs match source operation type | -| Label consistency | 47/48 (97.9%) — fa PR #16 missing labels (reorder-only edge case) | -| Metadata completeness | 48/48 (100%) — SHA, model, version, date, mode all present | -| zh-cn translation quality | ~9.6/10 — one terminology concern at deep nesting (test 22: 8.4/10) | -| fa translation quality | ~9.3/10 — natural Farsi phrasing, correct economic terminology | -| State file tracking | 48/48 (100%) — `.translate/state/*.yml` in every PR | - -**Key observations**: -- Math (LaTeX, display equations) preserved verbatim in all 48 PRs -- Code blocks untouched; only comments and markdown titles translated -- TOC operations (add, delete, rename) propagate correctly to both targets -- Frontmatter-only changes handled without unnecessary content retranslation -- Multi-file changes (test 19) handled in a single target PR with separate state files -- Line count variations between languages are expected (translated text length differs) - -**Minor issue**: fa PR #16 (pure section reorder) missing `action-translation` and `automated` labels — possible race condition with reorder-only changes. Does not affect translation correctness. - -**Overall assessment: PASS** — action functioning as designed across all tested scenarios. - -### Test Plan 4: Init on lecture-python-intro (2026-03-20) - -Full-scale `translate init` test on the production `lecture-python-intro` source repo (50 lectures) → fresh `test-lecture-intro.zh-cn` target, compared against the real human-curated `lecture-intro.zh-cn`. - -**Source**: `QuantEcon/lecture-python-intro` (50 lectures in `lectures/`) -**Target**: Fresh clone, `--target-language zh-cn`, model `claude-sonnet-4-6` - -**Run 1 — Sequential init** (2026-03-19): -- 47/50 lectures translated successfully -- 3 failures: `solow.md`, `input_output.md`, `lake_model.md` — network timeouts (undici fetch layer), NOT token limits (mid-sized 16-17KB files; 38KB files succeeded) -- 1,104,392 tokens, 94.9 minutes sequential -- All `.translate/state/` entries and heading-maps generated for successful files - -**Run 2 — Recovery with `--skip-existing`** (2026-03-20): -- `translate init --skip-existing` → skipped 47 already-translated lectures -- 3 remaining lectures translated successfully -- **Final result: 50/50 (100%)** - -**Structural integrity (50 files)**: - -| Check | Result | -|-------|--------| -| Code-cell count match (source vs target) | 50/50 (0 mismatches) | -| Heading structure match | 50/50 | -| Heading-maps present | 47/50 (3 section-less files: `index.md`, `troubleshooting.md`, `status.md`) | -| Font config (`mpl.rcParams`) | 40/42 applicable files | -| `.translate/state/` entries | 50/50 | - -**Quality comparison vs real `lecture-intro.zh-cn`**: -- Terminology and phrasing comparable to human-curated translations -- Math (LaTeX) preserved verbatim in all files -- Code blocks untouched; only comments and markdown translated -- Font configuration (`SimHei`/`STFangsong`) correctly injected in matplotlib cells -- Minor differences: AI translations slightly more literal; human repo has some editorial embellishments - -**New features validated**: -- `--skip-existing` (PR #34): Reads `.translate/state/` to skip already-translated files, enabling idempotent re-runs after partial failures -- `-j, --parallel` (PR #33): Concurrent translation with configurable worker count (not used in this test — validated separately) - -**Report**: `reports/lecture-python-intro/init-2026-03-19/README.md` - -### 7.8 Track Tool Version in `.translate/` Metadata - -The `.translate/` schema currently has no record of which version of action-translation created or last managed the project. Adding a `tool-version` field enables schema migration, version mismatch warnings, and audit trails. - -**Tasks:** - -- [x] Add `tool-version` to `TranslateConfig` interface (`config.yml`) — records the version that last wrote the config -- [x] Add `tool-version` to `FileState` interface (per-file state) — records the version that performed each translation -- [x] Read version from `package.json` at runtime (single source of truth) -- [x] Update `writeConfig()` and `writeFileState()` to include `tool-version` automatically -- [x] Update `readConfig()` / `readFileState()` validation — `tool-version` should be optional for backward compatibility with existing `.translate/` directories -- [ ] Future: `doctor` command can warn if config `tool-version` is older than installed version -- [x] Update `translate-state.ts` tests -- [x] Update CLI reference docs - ---- - -## Production Deployments - -### v0.11.1 Release (2026-03-20) - -Bugfix release addressing `--write-state` model preservation: - -- **Fix**: `translate status --write-state` now reads existing state files via `readFileState()` and preserves the `model` field if previously set by `forward` or `init`, instead of always overwriting with `unknown` -- **Fix**: Stale mock model names in `integration.test.ts` and `e2e-fixtures.test.ts` updated from `claude-sonnet-4.5-20241022` to `claude-sonnet-4-6` -- **Tests**: 898 → 900 (2 new tests for model preservation) -- **PR**: #38 (merged) -- **Release**: https://github.com/QuantEcon/action-translation/releases/tag/v0.11.1 - -### Farsi — `lecture-python-programming.fa` (2026-03-20) - -Connected existing Farsi translation of `QuantEcon/lecture-python-programming` using the connect-existing workflow: - -1. **Status diagnostic**: `translate status` → 18 ALIGNED, 7 MISSING_HEADINGMAP -2. **Check-sync triage**: `translate status --check-sync` → identified 2 files needing forward resync -3. **Forward resync**: `translate forward -f status.md -f python_by_example.md` → content resynced -4. **Heading-maps**: `translate headingmap` → all heading-maps generated/updated -5. **Doctor**: `translate doctor` → all checks pass -6. **Write-state bootstrap**: `translate status --write-state` → 25 state files created -7. **Push**: PR #67 merged on target repo -8. **Source workflow**: PR #486 merged on source repo — `sync-translations-fa.yml` (v0.11.0) - -**Result**: Repos fully linked. Merged PRs on source auto-create translation PRs on fa target. - -### Simplified Chinese — `lecture-python-programming.zh-cn` (2026-03-20) - -Fresh setup of Simplified Chinese translation using the fresh-setup tutorial workflow: - -1. **Scaffold**: `translate setup --source QuantEcon/lecture-python-programming --target-language zh-cn --docs-folder lectures` - - Created repo on GitHub with `.translate/config.yml`, review workflow, README -2. **Bulk translate**: `translate init -j 5` (5 parallel workers) - - 25/25 lectures translated, 504,954 tokens, 41.7 minutes - - Glossary: 357 terms (zh-cn) - - Font needed: `SourceHanSerifSC-SemiBold.otf` for CJK matplotlib labels -3. **Verify**: `translate status` → 23 ALIGNED, 2 MISSING_HEADINGMAP (section-less: `intro.md`, `status.md`) - - `translate doctor` → all 4 checks pass -4. **Push**: 132 files committed and pushed (25 lectures, 25 state files, 81 non-markdown assets, TRANSLATION-REPORT.md) -5. **Source workflow**: PR #487 merged on source repo — `sync-translations-zh-cn.yml` (v0.11.1) - - Also bumped fa workflow from v0.11.0 → v0.11.1 - -**Result**: Repos fully linked. Both fa and zh-cn sync workflows trigger on merged PRs touching `lectures/**/*.md`. - -### Deployment Summary - -| Repo | Role | Workflows | Status | -|------|------|-----------|--------| -| `QuantEcon/lecture-python-programming` | SOURCE | `sync-translations-fa.yml` (v0.11.1), `sync-translations-zh-cn.yml` (v0.11.1) | ✅ Active | -| `QuantEcon/lecture-python-programming.fa` | TARGET (Farsi) | `review-translations.yml` | ✅ Linked | -| `QuantEcon/lecture-python-programming.zh-cn` | TARGET (zh-cn) | `review-translations.yml` | ✅ Linked | - ---- - -## Phase 8: GitHub Action Automation (Future — 1-2 days) - -**Goal**: Scheduled backward analysis via GitHub Actions -**Prerequisite**: Phase 7 (real-world validated CLI) - -**Scope reduced**: Originally planned auto-PR creation from backward-sync. With the revised approach (human review via `resync review`), automation is limited to running the analysis and notifying maintainers. - -- [ ] Create workflow template: monthly `backward` analysis (two-stage) -- [ ] Create workflow template: monthly `status` check -- [ ] Store backward report as workflow artifact -- [ ] Notification: comment on a tracking Issue or Slack webhook with summary -- [ ] Maintainer runs `translate review` locally on the downloaded report -- [ ] Documentation: "Setting up automated backward analysis" - ---- - -## Phase 9: Whole-File Translation Architecture (Future — Investigation) - -**Goal**: Evaluate whether the whole-file LLM evaluation pattern from backward analysis should be applied to the core forward translation pipeline - -**Background**: The backward command's Stage 2 was originally designed with 1 LLM call per section (matching the forward sync architecture in `translator.ts`). Refactoring to 1 LLM call per file with all sections in a single prompt produced strictly better results: - -| Metric | Per-Section | Per-File | -|--------|------------|----------| -| API calls (51-file repo) | 182 | 32 | -| High-confidence findings | 6 | 7 | -| Noise (medium-confidence) | 25 | 17 | - -This raises the question: should `translator.ts` (forward sync) also move to whole-file translation instead of section-by-section? - -### Considerations - -**Arguments for whole-file forward translation**: -- Cross-section context (terminology consistency, narrative flow) -- Fewer API calls (cost and latency reduction) -- The LLM can see how terminology is used across the document - -**Arguments for keeping section-by-section forward translation**: -- Section-level caching — only re-translate changed sections (UPDATE mode). Whole-file would re-translate everything on any change. -- Granular error recovery — if one section fails, others succeed. Whole-file is all-or-nothing. -- Token limits — large documents (30K+ tokens) may not fit source + target + instructions in one call. -- Current architecture is battle-tested in production (GitHub Action sync mode). - -### Investigation Tasks - -- [ ] Measure current forward sync API call count for typical repos -- [ ] Estimate cost/latency savings from whole-file approach -- [ ] Design hybrid approach: whole-file for initial translation, section-level for UPDATE mode -- [ ] Test whole-file translation quality vs section-by-section on real lectures -- [ ] Determine if context window supports full document + translation + instructions -- [ ] Prototype and compare translation quality - ---- - -## Success Metrics - -### Quality - -| Metric | Target | -|--------|--------| -| Stage 1 triage recall | ≥95% (never miss a real backport candidate) | -| Stage 1 triage precision | ≥50% (some false positives are acceptable) | -| Stage 2 suggestion precision | ≥80% (suggestions accepted by reviewers) | -| Stage 2 suggestion recall | ≥70% (real improvements detected) | -| Forward sync accuracy | ≥95% (correct translations) | -| False positive rate (end-to-end) | ≤10% | - -### Performance - -| Metric | Target | -|--------|--------| -| Stage 1 triage per file | <5 seconds | -| Stage 2 analysis per file | <15 seconds | -| Full backward (51 files, 5 parallel) | ~4 minutes | -| API cost — backward (51 files) | ~$0.85 total (real measurement) | -| API cost — backward-sync per file | ~$0.10 | -| API cost — forward per file | ~$0.10 | -| Status check | <5 seconds (no LLM) | - -### Code Health - -| Metric | Current | Target | -|--------|---------|--------| -| Test count | 900 | 400+ | -| Test suites | 39 | — | -| Snapshots | 5 | — | -| `index.ts` lines | ~447 | ~447 (stable) | -| Deprecated methods | 0 | 0 | -| Dead tool directories | 0 | 0 | - ---- - -## Timeline Summary - -| Phase | Duration | Dependencies | Key Deliverable | -|-------|----------|--------------|-----------------| -| **Phase 0**: Foundation | 3-4 days | None | `index.ts` refactored, retry logic | -| **Phase 1**: Single-file backward | 3-4 days | Phase 0 ✅ | `npx resync backward -f file.md` (two-stage) | -| **Phase 2**: Bulk + status | 2-3 days | Phase 1 ✅ | `npx resync status` + bulk backward | -| **Phase 3a**: Interactive review | 3-4 days | Phase 2 ✅ | `npx resync review` with Issue creation | -| **Phase 3b**: Forward resync | 2-3 days | Phase 3a ✅ | `npx resync forward` with RESYNC mode | -| **Phase 4**: Refinement | 2-3 days | Phase 3b ✅ | Production-ready CLI ✅ | -| **Phase 5**: CLI rename + init | 2-3 days | Phase 3b ✅ | `translate init`, rename resync→translate ✅ | -| **Phase 5b**: Cleanup | 1 day | Phase 5 ✅ | Legacy tool deprecation, repo hygiene ✅ | -| **Phase 6**: `.translate/` metadata | 2-3 days | Phase 5 ✅ | Exact staleness, skip optimisation, provenance | -| **Phase 6b**: Setup command | 1-2 days | Phase 6 | `translate setup` — scaffold target repo | -| **Phase 7**: Real-World Readiness | 3-5 days | Phase 6b ✅ | Fix setup workflows, headingmap + doctor commands, e2e testing ✅ | -| **Phase 7.7**: E2E Testing | 2 days | Phase 7 ✅ | Real-world testing, 5 bugs fixed, v0.9.0 release ✅ | -| **Phase 7 GitHub Action Test** | 1 day | Phase 7.7 ✅ | Both zh-cn + fa workflows pass end-to-end on test repos ✅ | -| **Phase 7 Init Testing** | 1 day | Phase 7.7 ✅ | Init 50/50 on lecture-python-intro, --skip-existing + --parallel (PRs #33, #34) ✅ | -| **Phase 8**: Automation | 1-2 days | Phase 7 ✅ | Scheduled backward analysis | -| **Phase 9**: Whole-file translation | TBD | Phase 7 ✅ | Evaluate whole-file approach for forward sync | - -**Total**: 15-23 days (Phase 0-4), +3-4 days (Phase 5-5b), +2-3 days (Phase 6), +3-5 days (Phase 7 + 7.7), +2 days (Phase 8), +TBD (Phase 9) - ---- - -## Open Questions - -1. ~~**Stage 1 token limits**: Very large documents (30K+ tokens per side) may exceed context window for single-call triage.~~ **Resolved**: `SKIPPED_TOO_LARGE` verdict handles this. Only 2 files hit the limit in real testing (README.md, tax_smooth.md). -2. ~~**Backport confidence threshold**: Default 0.6~~ **Validated**: 0.6 works well. Real BUG_FIX findings came in at 0.85-0.97. Lower-confidence suggestions (0.6-0.7) are borderline but worth flagging. -3. ~~**Multi-section changes**: Group in one suggestion or separate?~~ **Resolved**: Separate per-section suggestions. Each gets its own category, confidence, and reasoning. -4. ~~**TARGET-only files**: Flag for addition to SOURCE, or just report?~~ **Resolved**: `status` reports only (diagnostic tool). Action on `TARGET_ONLY` / `SOURCE_ONLY` belongs to Phase 3 commands. -5. **Run frequency**: Monthly default, option for more frequent? — Decide in Phase 4 -6. ~~**backward-sync PR format**: Should `backward-sync` create PRs directly, or write files for manual PR creation?~~ **Resolved**: `backward-sync` deferred. The `review` command creates GitHub Issues instead. Human edits SOURCE directly. -7. ~~**Report-driven backward-sync**: The `--from-report` flag reads a backward JSON report and syncs only marked suggestions.~~ **Resolved**: Replaced by interactive `review` command that reads the report folder and walks through suggestions with accept/skip/reject. -8. **Stage 1 precision**: Flagging rate was ~67% vs estimated 5-10%. High recall is good, but Stage 1 could be tuned to reduce false positives and save Stage 2 costs. — Address in Phase 4 prompt tuning -9. **Whole-file vs section-by-section translation**: Backward Stage 2 showed ~6x fewer API calls and better quality with whole-file evaluation. Should forward sync (`translator.ts`) adopt the same pattern? Trade-off: better context vs loss of section-level caching in UPDATE mode. — Investigate in Phase 9 -10. ~~**CLI framework**: `ink` (Node.js) vs `rich` (Python) for the `review` command's terminal rendering.~~ **Resolved**: `ink` (Node.js). Keeps unified codebase, direct module imports for `forward` command. Python `rich` rewrite documented as a future option (see Future section). - ---- - -## Lessons Learned - -### From Phase 1 Real-World Testing - -- **Temporal context is critical**: Without the interleaved commit timeline, the LLM makes directional errors (flagging SOURCE's newer code as a TARGET improvement). Adding timeline to prompts eliminated this class of false positive. -- **Two-stage design validated**: Stage 1 correctly flags differences; Stage 2 correctly filters non-actionable ones. The cost savings are real (~$0.01 triage vs ~$0.10-0.50 per-section analysis). -- **Real repo names**: The zh-cn repo for `lecture-python-intro` is `lecture-intro.zh-cn` (not `lecture-python-intro.zh-cn`). - -### From Phase 2 Bulk Testing - -- **Whole-file evaluation wins**: Refactoring Stage 2 from per-section (182 calls) to per-file (32 calls) produced better results — more high-confidence findings, less noise, and ~6x fewer API calls. Cross-section context helps the LLM avoid false positives. -- **5-way parallelism** is the sweet spot — fast enough to complete 51 files in ~4 minutes, without overwhelming the API. -- **Buffered logging** is essential for parallel work — interleaved output from concurrent files is unreadable. -- **Progress bar** provides much better UX than scrolling output — users see status at a glance, details go to log file. -- **Fresh start by default** (wipe output folder) is more intuitive than accumulating stale results. `--resume` is the opt-in for incremental runs. -- **Date-only folder naming** (`backward-2026-03-04`) is cleaner than timestamped (`backward-2026-03-04_01-38-48`). Same-day re-runs overwrite, which matches the fresh-start default. -- **Stage 1 flagging rate** was ~67% (33/49 files flagged), much higher than the estimated 5-10%. This suggests the Stage 1 prompt has high recall (good — false negatives are worse than false positives) but precision could be improved. Stage 2 effectively filters: only 20 suggestions survived from 33 flagged files. - -### From Init Testing (lecture-python-intro, 50 lectures) - -- **Network failures are transient, not size-correlated**: 3 failures were mid-sized files (16-17KB) while 38KB files succeeded. Caused by undici fetch layer timeouts, not Claude token limits. -- **Idempotent re-runs are essential**: `--skip-existing` (reads `.translate/state/`) enables safe recovery from partial failures without re-translating completed work. Linear `--resume-from` is insufficient when failures are scattered. -- **Structural integrity is near-perfect**: Zero code-cell mismatches across 50 files. Heading-maps generated correctly. Font configuration injected in matplotlib cells. The init pipeline is production-ready. -- **AI vs human translation quality**: AI translations are slightly more literal than human-curated versions, but terminology and technical accuracy are comparable. No hallucinated content or structural corruption observed. - ---- - -## Future Work - -Items moved from completed phases. These are candidates for future development, not committed work. - -### Integration Testing - -- [x] Test `backward` + `review` workflow with `lecture-python-programming.myst` ↔ `lecture-python-programming.fa` (2026-03-18) -- [x] Test `forward` resync with `lecture-python-programming.myst` ↔ `lecture-python-programming.fa` (2026-03-18) -- [x] Test `setup` + `init` with `test-translation-sync` → `test-translation-sync.fa` (2026-03-19) -- [x] Test `init` on production repo: `lecture-python-intro` → 50/50 lectures translated (2026-03-20) -- [x] Validate RESYNC translation quality — about_py.md resynced correctly -- [x] Validate forward triage accuracy — fixed: about_py.md CONTENT_CHANGES, scipy.md IDENTICAL -- [x] Test review → dry-run with real reports (2026-03-18) -- [x] Document 5 bugs found → all fixed in PR #31 -- [ ] Test `backward` + `review` workflow with `lecture-python-intro` ↔ `lecture-intro.zh-cn` -- [ ] Test review → Issue creation end-to-end (non-dry-run) -- [ ] Validate two-stage triage accuracy (Stage 1 recall ≥95%) - -### Prompt Tuning - -- [ ] Review Stage 1 triage accuracy (false negatives are critical failures) -- [ ] Review Stage 2 suggestions from Phase 1-2 runs -- [ ] Identify false positives and false negatives -- [ ] Tune Stage 1 prompt for recall (bias toward flagging) -- [ ] Tune Stage 2 prompt for precision (reduce noise in suggestions) -- [ ] Tune RESYNC prompt for translation preservation quality -- [ ] Re-run validation tests - -### Error Handling - -- [ ] Missing source/target files -- [ ] API timeout/rate limit -- [ ] Invalid heading-map -- [ ] Oversized documents (Stage 1 token limit exceeded) -- [ ] Graceful degradation with warnings - -### Review Command UX Polish - -- [ ] Scroll viewport — fixed-height card area with up/down arrow scrolling -- [ ] Truncate long Before/After blocks — show first N lines, `[E]xpand` to see full -- [ ] Syntax highlighting in Before/After code blocks (chalk + cli-highlight) -- [ ] MyST-aware rendering — styled directives, math, headers in card output -- [ ] Colour-coded inline diff (word-level Before→After highlighting) - -### Documentation — Restructure into User & Developer Guides - -Restructure `docs/` into two clear audiences and deploy via GitHub Pages. - -#### User Documentation - -- [ ] **Quick Start** — streamlined Action setup for new users -- [ ] **Action Reference** — inputs, outputs, modes (sync/review), examples -- [ ] **CLI Reference** — `status`, `backward`, `review`, `forward` with usage examples and options -- [ ] **Glossary Guide** — how to use and extend the translation glossary -- [ ] **Heading Maps** — user-friendly explanation (what they are, when to edit manually) -- [ ] **FAQ** — common issues, troubleshooting, "how do I..." answers - -#### Developer Documentation - -- [ ] **Architecture** — module map, data flow diagrams, key design constraints -- [ ] **Sync Workflow** — internal lifecycle, UPDATE/NEW/RESYNC modes -- [ ] **Implementation** — technical reference (parser, diff-detector, translator internals) -- [ ] **Testing Guide** — test pyramid, fixtures, how to add tests -- [ ] **Design: Resync CLI** — two-stage architecture, review workflow, CLI framework decision -- [ ] **Claude Models** — model selection, token limits, retry logic - -#### GitHub Pages Deployment - -- [ ] Use **mystmd** as the static site generator -- [ ] Configure `docs/` with `mst.yml` -- [ ] Add GitHub Actions workflow: auto-deploy docs on push to `main` -- [ ] Landing page with navigation to User / Developer sections -- [ ] Ensure existing doc links remain functional (redirects or path mapping) -- [ ] Add docs site URL to repo About section and README - -#### General - -- [ ] Update main README with link to docs site -- [ ] Migrate content from existing `docs/*.md` files into new structure - -### Additional Review Actions - -- [ ] Use atomic Git commits (Tree API) for multi-file PRs in sync mode -- [ ] Add pre-flight check for section-level translation token limits -- [ ] Refactor `reviewer.ts` to reuse `MystParser` instead of local parsing -- [ ] Simplify `parseTranslatedSubsections()` wrapper approach - ---- - -## Future: Python Rewrite with `rich` - -If the `ink`-based CLI proves limiting for MyST rendering quality, the long-term path is a **full rewrite of the CLI in Python** — not a mixed-language project. - -### Motivation - -`rich` (Python) is the gold standard for terminal rendering: native Markdown, Pygments-powered syntax highlighting, panels, tables, columns, and tree views — all built-in. A `MystRenderable` subclass could provide directive-aware rendering (`{note}`, `{code-cell}` as styled panels). `textual` (built on `rich`) offers a full TUI framework for scrolling, mouse support, and complex interaction. - -Python is also a natural fit for the QuantEcon ecosystem — users are Python developers, and a `pip install`-able CLI published to PyPI would feel native. - -### What a rewrite involves - -To avoid a mixed-language project (two runtimes, two package managers, fragmented testing), a Python rewrite would port **the entire CLI**: - -| Module | Lines | Complexity | -|--------|-------|------------| -| `parser.ts` | ~280 | Stack-based MyST parser, battle-tested edge cases | -| `section-matcher.ts` | ~150 | Position-based matching with heading-map | -| `heading-map.ts` | ~250 | Extract/update/inject heading maps | -| `translator.ts` | ~460 | Claude API calls, NEW/UPDATE/RESYNC modes, retry logic | -| `file-processor.ts` | ~670 | Document reconstruction, subsection handling | -| `language-config.ts` | ~100 | Language-specific translation rules | -| `diff-detector.ts` | ~195 | Change detection, recursive subsection comparison | -| CLI commands + types | ~1,500 | backward, status, review, forward, report generator | -| **Total** | **~3,600** | Plus duplicate test suites | - -### When to consider this - -- If `ink` rendering proves insufficient for reviewing MyST content (math, directives, side-by-side) -- If the QuantEcon team wants to maintain the CLI in Python long-term -- If `textual` TUI capabilities become important (scrollable diff views, etc.) - -The GitHub Action itself would remain Node.js (Actions require JavaScript). Only the CLI tool would move to Python. - -### Prerequisites - -- Phase 4 complete (stable CLI interfaces and JSON schemas) -- Clear rendering gaps identified in `ink` that justify the rewrite cost -- Decision on whether to publish to PyPI - ---- - -*Last updated: 2026-03-20 (v0.11.1 release, production deployment: lecture-python-programming → fa + zh-cn)* diff --git a/dev-notes/README.md b/dev-notes/README.md deleted file mode 100644 index 531499c9..00000000 --- a/dev-notes/README.md +++ /dev/null @@ -1,12 +0,0 @@ -# Developer notes - -Internal working documents — design plans, roadmaps, and historical fix -write-ups. These are **not** part of the published documentation site -(`docs/`) and are kept here for maintainers' reference. - -| File | Description | -|------|-------------| -| [PLAN.md](PLAN.md) | Detailed development roadmap and phase tracking. Maintained by the team — see the document header for the version it was last updated against. | -| [FIX-ISSUE-63.md](FIX-ISSUE-63.md) | Historical write-up of the rebase-on-merge fix (issue #63), shipped in v0.15.0. | - -For user-facing and architectural documentation, see [`docs/`](../docs/). diff --git a/docs/projects/README.md b/docs/projects/README.md index 36fe6d01..814a2807 100644 --- a/docs/projects/README.md +++ b/docs/projects/README.md @@ -124,7 +124,7 @@ Summer students would likely focus on **Project A** (the most coding-intensive), ## Related Documents -- [PLAN.md](../../dev-notes/PLAN.md) -- Development roadmap for `action-translation`; Phase 9 discusses whole-file vs section-by-section translation architecture, and `experiments/forward/` contains initial experiment results that Project A should build on +- [ARCHITECTURE.md](../../.dev/ARCHITECTURE.md) -- Open design questions for `action-translation`; Q3 discusses whole-file vs section-by-section translation architecture, and `experiments/forward/` contains initial experiment results that Project A should build on - [_archive/PROJECT-BENCHMARK.md](_archive/PROJECT-BENCHMARK.md) -- Original comprehensive benchmark plan with detailed infrastructure specs (CI/CD, issue templates, provider interfaces, dashboard mockup) -- useful reference when implementing - [architecture.md](../developer/architecture.md) -- `action-translation` module structure - [testing.md](../developer/testing.md) -- How the action's test suite works