Repository navigation
Cut review latency with a bounded output budget and cache-stable prompt prefix - #34
Merged
Merged
Conversation
A review's wall-clock is dominated by output generation: the recorded 12k-completion run spends 3 to 7 minutes generating at typical provider throughput. Cap review, JSON-repair, and semantic-correction calls at 4096 output tokens and instruct terse findings (40-word summary, 12-word titles, 60-word bodies with evidence, impact, and action). Interactive answers keep their provider default. Reorder the system prompt so the global rules, response contract, and built-in content-policy baseline form a byte-stable prefix across repositories, with repo focus, guardrails, policy additions, tone, and the diff after it, matching implicit prefix-cache semantics. The effective policy text is unchanged. Fetch PR metadata/check ownership and the full diff in parallel; incremental diffs still wait for the selected head SHA.
![postil-dev[bot]](https://avatars.githubusercontent.com/in/3455612?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review latency is dominated by output generation, not startup: a 12k-token completion alone takes minutes at typical provider throughput. Review, JSON-repair, and semantic-correction calls now carry a 4096-token output cap and the prompt asks for terse findings (40-word summary, 12-word titles, 60-word bodies with evidence, impact, and action); interactive answers keep their provider default. The system prompt is reordered so global rules, the response contract, and the built-in content-policy baseline form a byte-stable prefix across repositories for implicit prompt caching, with repo-specific sections and the diff after it; the effective policy text is unchanged. PR metadata and full-diff fetches now run in parallel. Verified with fmt, clippy -D warnings, 82 library tests, 34 wiremock e2e tests, and the 40-fixture hermetic benchmark (33 TP, 0 FP, 0 FN).