feat(tools): add Copilot backend, per-commit cache, and async refactor to release notes generator#7264
Conversation
… cache, async Add a GitHub Copilot backend (--backend copilot) that authenticates via the Copilot CLI instead of ANTHROPIC_API_KEY, making the tool usable without an Anthropic account. Add a per-commit analysis disk cache (.release_notes_cache/ by default). The cache key hashes backend name, model, token budget, system prompt, and commit SHA, so any change to prompts or model automatically invalidates stale entries. Re-running over a range that was already analyzed skips all API calls. Migrate the analysis pipeline from concurrent.futures.ThreadPoolExecutor to native asyncio, enabling a single asyncio.run() entry point and a live heartbeat loop during the long compose call. Tighten commit categorisation prompts: add a CRITICAL RULE block and annotate every domain category as "(features only)", so crash fixes / data-corruption fixes / race conditions always land in bugfix rather than leaking into domain sections. Refactor for separation of concerns: - LLMBackend interface gains analyze_commit(), compose_progress_status(), post_process_notes(), notes_filename() — backend-specific logic stays in the backend, not in callers - AnthropicBackend._sync_stream() replaces a 75-line nested closure; _ComposeStats dataclass replaces five loose instance variables - _model renamed to public model; _make_cache_key(backend, sha) replaces the old 5-argument function - main() decomposed into _build_arg_parser, _check_prerequisites, _build_backend, _resolve_cache_dir, _run_async - FAIL_FAST_THRESHOLD, _is_fatal_error(), and all numeric constants (ANALYZE_MAX_TOKENS_*, COMPOSE_MAX_TOKENS, COMPOSE_HEARTBEAT_S, …) promoted to module level Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
romange
force-pushed
the
feat/release-notes-copilot-backend
branch
from
2241a32 to
8d2e3e5
Compare
🤖 Augment PR SummarySummary: This PR significantly expands Changes:
Technical Notes: Backends are now unified behind an 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
| text = text.strip() | ||
| text = re.sub(r"^```(?:json)?\s*", "", text) | ||
| text = re.sub(r"\s*```$", "", text) | ||
| match = re.search(r"\{.*\}", text, re.DOTALL) |
There was a problem hiding this comment.
re.search(r"\{.*\}", ...) is greedy and can capture more than the intended JSON object if the backend returns extra braces (or multiple JSON objects), leading to intermittent json.loads failures. This makes the Copilot JSON path more brittle than necessary.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
There was a problem hiding this comment.
Pull request overview
This PR upgrades the tools/release_notes_generator.py release-notes tool by adding a GitHub Copilot-backed LLM option, introducing a per-commit disk cache for analysis results, and refactoring the execution pipeline to use asyncio (including streaming/heartbeat progress during composition).
Changes:
- Add a new
--backend copilotoption alongside the existing Anthropic backend, with backend-specific behavior encapsulated behind anLLMBackendinterface. - Add a per-commit analysis disk cache (default
.release_notes_cache/) keyed by backend/model/prompt/token budget + commit SHA. - Migrate commit analysis + composition from
ThreadPoolExecutortoasyncio, adding compose heartbeats and progress counters.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tools/requirements.txt | Notes optional Copilot SDK install for the new backend. |
| tools/release_notes_generator.py | Implements Copilot backend, async pipeline refactor, streaming compose progress, and per-commit analysis caching. |
| .gitignore | Ignores the new default on-disk cache directory. |
GoldBefore the pr. Copilot and Anthropic - after this changeMost notable change is more issues are put under bug fixes category. release_notes_copilot_v1.37.0_to_v1.38.0.md |
- Separate anthropic and pydantic imports so pydantic (needed by
CommitAnalysis) is still available for --backend copilot when the
anthropic package is not installed
- Fix _make_cache_key to length-prefix each field before hashing,
preventing hash collisions between distinct (name, model, ...) tuples
that share the same raw byte concatenation
- Fix _parse_commit_analysis_json to use json.JSONDecoder.raw_decode
instead of a greedy r"\{.*\}" regex, which correctly handles nested
braces and stops at the first complete JSON object
- Fix _analyze_round to add commits that were silently skipped due to
abort_event back into the failed list after asyncio.gather completes,
so they are visible to the caller and included in the next retry round
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
augment review |
| return parser | ||
|
|
||
|
|
||
| def _check_prerequisites(backend_name: str) -> Optional[str]: |
There was a problem hiding this comment.
_check_prerequisites doesn’t validate that Pydantic is installed/compatible, but later code unconditionally uses Pydantic v2 APIs like model_dump()/model_dump_json() (including for the Copilot backend + cache). This can crash at runtime if users follow the Copilot install path (pip install github-copilot-sdk) or have Pydantic v1 (Anthropic allows <3).
Severity: medium
Other Locations
tools/release_notes_generator.py:799tools/release_notes_generator.py:943tools/release_notes_generator.py:959
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| python tools/release_notes_generator.py HEAD~50..HEAD --max-parallel 4 | ||
|
|
||
| # GitHub Copilot backend (uses Copilot CLI auth, no API key needed): | ||
| pip install github-copilot-sdk |
There was a problem hiding this comment.
pip install -r tools/requirements.txt
Summary
--backend copilot) as an alternative to the Anthropic backend; routes requests through the Copilot CLI — noANTHROPIC_API_KEYrequired<repo>/.release_notes_cache/); cache key covers backend name, model, token budget, system prompt, and commit SHA — any prompt or model change auto-invalidatesconcurrent.futures.ThreadPoolExecutorto nativeasyncio;compose_release_notesstreams output with a 15 s heartbeat and live token countersCRITICAL RULEblock now strongly enforce that any fix (crash, data corruption, race, wrong result) lands inbugfixregardless of subsystem; domain categories are reserved for new featuresrelease_notes_copilot_<range>.md) to distinguish from Anthropic runsRefactoring (no behaviour change to existing Anthropic path):
LLMBackendinterface extended withanalyze_commit,compose_progress_status,post_process_notes,notes_filename— all backend-specific decisions live in the backend, not in callersAnthropicBackend._sync_streamreplaces the 75-line nested_runclosure;_ComposeStatsdataclass replaces five loose instance variables_modelrenamed to publicmodel;_make_cache_key(backend, sha)replaces the old 5-argument functionmain()decomposed:_build_arg_parser,_check_prerequisites,_build_backend,_resolve_cache_dir,_run_async(singleasyncio.runcall)_is_fatal_error,FAIL_FAST_THRESHOLD, and all token/heartbeat/threshold constants promoted to module levelRelease notes comparison (v1.37.0..v1.38.0)
Three outputs were generated from the same commit range to compare quality:
release_notes_v1.37.0_to_v1.38.0.gold.mdrelease_notes_v1.37.0_to_v1.38.0.mdrelease_notes_copilot_v1.37.0_to_v1.38.0.mdAnthropic new vs gold — structurally identical; new output groups HNSW improvements more tightly in Highlights and surfaces the 30+ bug-fix count explicitly in the opening.
Copilot vs gold — equivalent quality and coverage; the Copilot model occasionally violates the "no preamble" system-prompt constraint (the spurious leading line above), which
post_process_notesdoes not yet strip.release_notes_v1.37.0_to_v1.38.0.gold.md (pre-PR baseline)
release_notes_v1.37.0_to_v1.38.0.md (new Anthropic backend)
release_notes_copilot_v1.37.0_to_v1.38.0.md (new Copilot backend)
🤖 Generated with Claude Code