Salvage: GitHub App review precision, pinned source context and evidence-based findings - #6240
Conversation
…ndings Give the shared CLI/tool PR reviewer numbered surrounding source from raw regular blobs at the exact PR head. Preserve the complete diff, inline anchors, existing input/pass budgets, and provider-call count. Collect source off the async runtime and identify omitted context. Require concrete triggers, evidence, impact and fixes; reject speculative compiler claims and generic test nits in the review rubric. Refs #6135 (SHA-6514). Validation: npm test && npm run check:web passed: 66 package tests, 9 SDK tests, 446 web tests, 0 failures; web lint/types/facts/production build passed. Focused cargo test -p codewhale-tui --lib --locked tools::review: 60 passed, 0 failed, 0 ignored. Standalone hunk-parser tests: 15 passed, 0 failed. The Rust repeated-match negative control compiled with -D warnings and passed all four variants. No provider call, GitHub review publication or deployment. (cherry picked from commit 14eb885)
The salvaged commit compiles but fails CI's clippy line on
review_pr.rs:625 — `serde_json::from_str(&prompt)` where `prompt` is already
a `&str`, so `-D clippy::needless-borrow` rejects it.
Nobody had ever built this commit; it sat on a branch with no PR since
2026-09-13. That is exactly the class of thing a cherry-pick's clean apply
does not tell you.
cargo clippy --workspace --all-targets --all-features --locked
(CI's allow list) clean
tools::review 60 passed; 0 failed
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJENKJ2smviQW4FVGzUTk9
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Claude finished @Hmbown's task in 3m 56s —— View job Review complete
Note: SummaryThis is a well-scoped, well-tested rescue of a stale commit, and the second commit's clippy fix ( Two source-level findings, both low severity — posted inline:
Nothing else stood out: |
| .expect("source files") | ||
| .push(file); | ||
| } | ||
| (report.to_string().chars().count() <= budget).then_some(report) |
There was a problem hiding this comment.
Confirms the gap the PR description itself calls out under "Why this does not close #6135": when the assembled report exceeds budget, this returns None and silently drops the already-computed unavailable_files/omitted_files counters along with it — the caller (build_pr_pass_prompt) then just sees repository_context: null, with no signal that context was dropped for being oversized rather than simply unneeded (small pass, no room to spare). docs/GITHUB_APP.md's new "Review evidence and precision" section says "The request explicitly records unavailable files and omitted context," which isn't true for this branch — the largest PRs (the ones the feature is meant to help with) get no context and no receipt saying why. Not a blocker for this PR since it's explicitly scoped out and #6135 stays open for it, but worth a one-line doc caveat, or truncating to fit instead of dropping wholesale, and worth a repository_context.omitted != None-vs-dropped-because-empty distinction. Fix this →
| @@ -8610,12 +8610,11 @@ async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> { | |||
| crate::tools::review::plan_pr_review(&diff, view, args.max_chars, args.max_passes) | |||
There was a problem hiding this comment.
Minor: std::env::current_dir()? is already computed a few lines above (line 8598) and passed to collect_diff. This adds a second, redundant syscall for the same value rather than reusing it — low risk here since nothing awaits between the two calls, but worth collapsing into one let workspace = std::env::current_dir()?; used for both.
There was a problem hiding this comment.
Codewhale review
Complete review coverage: 1/1 passes, 5 file patches, sha256:9abd3bc0fabcd0708437cac2ab5d759a7cd6fc5d7d2f999a103eec6c84dce763.
Pass 1: Adds source_context/context_blob (pinned-head Git blob excerpts around each pass's hunks), threads them into per-pass PR prompts via a new async build_pr_review_prompts used by both the CLI and the review tool, adds DiffHunks::paths/ranges, and documents the behaviour. The Git side is defensively written (literal pathspecs, --full-tree, -- separator, mode/type/size validation, symlink and NUL rejection, no replace objects / lazy fetch, per-file error containment), so no correctness or security failure was found in the collection path itself. The defects that remain are in the report/None handling: the receipt that the docs promise is discarded exactly on the passes with the tightest budget.
Findings
- [WARNING] Context is dropped together with its availability receipt when the budget is tight (
crates/tui/src/tools/review_pr.rs:409)
Trigger: a pass whose remaining budget is small, i.e.plan.manifest.max_chars_per_pass - pass.manifest.diff_chars < 512(review.rs:484-486), or any pass where the assembled report would not fit. At review_pr.rs:341-343source_contextreturnsNonebefore reading anything, and at review_pr.rs:409 it discards an already-built report that holds theunavailable_files(line 356) andomitted_files(lines 351/399) counters.build_pr_pass_promptthen emits"repository_context": null(review.rs:495) and the only signal left is the staticcontext_limittext (review.rs:496); nothing in the request says whether the head was missing, every blob was non-regular/oversized/binary, or the budget was simply too small. User-visible impact: on large passes exactly the model-side evidence this change exists to add is silently absent, and a maintainer reading a posted review cannot tell 'context was attempted and could not be collected' from 'context never ran', so the precision claim is unverifiable where it matters most. It also contradicts the new documentation, which states the request records unavailable files and omitted context (docs/GITHUB_APP.md, the new 'Review evidence and precision' section). Smallest useful fix: never returnNoneonce the head is valid — return a minimal receipt (head_sha, emptyfiles, the two counters, plus areasonsuch asbudget/no-commit) so the counters survive; keep within budget by emitting nolines. Thebudget < 512gate can additionally be documented as a threshold rather than being indistinguishable from a collection failure. - [INFO]
omitted_filesconflates two different causes of omitted context (crates/tui/src/tools/review_pr.rs:351)
review_pr.rs:351 initialisesomitted_filestopaths.len() - selected, i.e. files beyond theMAX_CONTEXT_FILES = 32cap, and review_pr.rs:399-400 then increments the same counter for files whose candidates all failed the per-file allowance. Trigger: a pass over 300 files where 32 are capped and 5 of the selected ones get no lines — the report saysomitted_files: 37and the reader cannot tell whether the pass was truncated by the file cap or by character starvation, which are different remediation (raise the cap vs. raiseCODEWHALE_REVIEW_MAX_CHARS). Smallest useful fix: keep the two counts separate (e.g.omitted_filesfor the cap andstarved_filesfor the budget) or add a shortreasonper entry.
Suggestions
crates/tui/src/tools/review_pr.rs:409— Replace the(... ).then_some(report)drop with a return that always keeps the receipt: when the report does not fit, return the same report withfilesemptied (or a trimmed per-file list) plus areasonfield instead ofNone, and move thebudget < 512case (line 341) to the same receipt-shaped result. This keepsunavailable_files/omitted_filesobservable in the request, which is what the new docs section promises, at no extra model input cost.
Assessment
Pass 1: The added collection path is deliberately conservative: --literal-pathspecs plus -- prevents pathspec/option injection from attacker-controlled diff paths, --full-tree makes paths independent of cwd, ls-tree output is checked for exact path, regular-file mode, blob type, a 40-hex object id and a 128 KiB bound, symlinks/submodules are rejected, cat-file blob bypasses checkout filters, NUL bytes and non-UTF-8 fail closed, and every per-file failure is contained as an unavailable_files increment (so the tool and CLI cannot fail a review because the head is not fetched locally). The two findings above are the concrete residue; the first is the same silent-failure shape the PR description acknowledges as open #6135 item 2, and I agree it is a real defect rather than just a size limit, because the counters are discarded rather than merely reduced. No build, clippy, test or runtime verification was performed by this review; the PR body's cargo check/clippy/60 passed claims were not reproduced here. Open questions I could not settle from the supplied context: (1) whether review_pr::model_diff (used for the prompt's diff field and for diff_chars) can alter hunk headers or the file set, since source_context parses its hunks and its contains_line exclusions from the raw pass.diff — if it can, lines shown as 'supplementary, not in the diff' could disagree with the patch the model receives; the added tests only exercise a plain single-file diff where the two are equal. (2) whether any caller of the signature-changed pub(crate) fn build_pr_pass_prompt exists outside the diffed files (it is crate-internal, and the two in-repo call sites plus the test are updated in this diff). (3) lib.rs:8613 now resolves std::env::current_dir()? unconditionally for codewhale review --pr, so the CLI's context source is the process cwd; any failure degrades silently, which is intended but means context quality depends on how the CLI was launched.
Advisory review by Codewhale (codewhale review --pr 6240 --post, head 672a54a228522ad2d1ab63542c1b358425d8b91f). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.
| .expect("source files") | ||
| .push(file); | ||
| } | ||
| (report.to_string().chars().count() <= budget).then_some(report) |
There was a problem hiding this comment.
[WARNING] Context is dropped together with its availability receipt when the budget is tight
Trigger: a pass whose remaining budget is small, i.e. plan.manifest.max_chars_per_pass - pass.manifest.diff_chars < 512 (review.rs:484-486), or any pass where the assembled report would not fit. At review_pr.rs:341-343 source_context returns None before reading anything, and at review_pr.rs:409 it discards an already-built report that holds the unavailable_files (line 356) and omitted_files (lines 351/399) counters. build_pr_pass_prompt then emits "repository_context": null (review.rs:495) and the only signal left is the static context_limit text (review.rs:496); nothing in the request says whether the head was missing, every blob was non-regular/oversized/binary, or the budget was simply too small. User-visible impact: on large passes exactly the model-side evidence this change exists to add is silently absent, and a maintainer reading a posted review cannot tell 'context was attempted and could not be collected' from 'context never ran', so the precision claim is unverifiable where it matters most. It also contradicts the new documentation, which states the request records unavailable files and omitted context (docs/GITHUB_APP.md, the new 'Review evidence and precision' section). Smallest useful fix: never return None once the head is valid — return a minimal receipt (head_sha, empty files, the two counters, plus a reason such as budget/no-commit) so the counters survive; keep within budget by emitting no lines. The budget < 512 gate can additionally be documented as a threshold rather than being indistinguishable from a collection failure.
| "head_sha": head_sha, | ||
| "files": [], | ||
| "unavailable_files": 0, | ||
| "omitted_files": paths.len() - selected, |
There was a problem hiding this comment.
[INFO] omitted_files conflates two different causes of omitted context
review_pr.rs:351 initialises omitted_files to paths.len() - selected, i.e. files beyond the MAX_CONTEXT_FILES = 32 cap, and review_pr.rs:399-400 then increments the same counter for files whose candidates all failed the per-file allowance. Trigger: a pass over 300 files where 32 are capped and 5 of the selected ones get no lines — the report says omitted_files: 37 and the reader cannot tell whether the pass was truncated by the file cap or by character starvation, which are different remediation (raise the cap vs. raise CODEWHALE_REVIEW_MAX_CHARS). Smallest useful fix: keep the two counts separate (e.g. omitted_files for the cap and starved_files for the budget) or add a short reason per entry.
| .expect("source files") | ||
| .push(file); | ||
| } | ||
| (report.to_string().chars().count() <= budget).then_some(report) |
There was a problem hiding this comment.
Replace the (... ).then_some(report) drop with a return that always keeps the receipt: when the report does not fit, return the same report with files emptied (or a trimmed per-file list) plus a reason field instead of None, and move the budget < 512 case (line 341) to the same receipt-shaped result. This keeps unavailable_files/omitted_files observable in the request, which is what the new docs section promises, at no extra model input cost.
Windows failure: cause, and a fix that keeps the coverage
Cause. The test creates a file literally named I see a Alternative that keeps all three platforms. Change one line: let path = "[l]iteral-other.rs";
So Happy either way — flagging it because the cfg-gate is the kind of fix that is invisible later, and this guard is worth keeping on the platform where paths behave differently. For the record, this is the second thing this salvaged commit needed that a clean cherry-pick could not reveal: it also failed CI's clippy on a needless borrow (fixed in |
The test writes a file literally named `[literal]*.rs` to prove the
glob-looking path is read literally. `*` cannot appear in a Windows
filename, so std::fs::write fails with InvalidFilename (OS error 123)
before any assertion runs — Test (windows-latest) failed twice on this
exact line (review_pr.rs:692).
Follows the adjacent source_context_never_follows_a_pinned_symlink
test, which is cfg(unix) for the same class of reason.
source_context_is_bounded_line_exact_and_uses_literal_paths
1 passed; 0 failed; finished in 0.21s
Signed-off-by: CodeWhale Bot <bot@codewhale.net>
|
Claude finished @Hmbown's task in 1m 27s —— View job Review complete
Note: SummaryThis sync adds exactly two commits on top of the previously reviewed
One thing worth flagging, though not a blocker: this took the No correctness bugs, provider/model-safety issues, or security concerns in either commit — both are test-only changes with no production code touched. The two substantive gaps from the earlier review round ( |
There was a problem hiding this comment.
Codewhale review
Complete review coverage: 1/1 passes, 5 file patches, sha256:18075e1c3b0067d58685d1b833da8d03d2e9359eb98cb024043439f2376a2831.
Pass 1: Cherry-pick adds pinned-head source context (review_pr::source_context/context_blob), a JSON pass-prompt builder (review::build_pr_review_prompts/build_pr_pass_prompt), DiffHunks::paths/ranges, the run_review wiring, and docs. I checked the new control flow, guards and budget arithmetic against the supplied diff and source context; I found no defect I can demonstrate, and one evidence-contract problem in the branch that discards an already-built context report.
Findings
- [INFO] Over-budget repository_context drops the unavailable/omitted counters it already built (
crates/tui/src/tools/review_pr.rs:409)
crates/tui/src/tools/review.rs::build_pr_pass_prompt passes the residual budget (plan.manifest.max_chars_per_pass - pass.manifest.diff_chars) to review_pr::source_context and puts its result in the request as "repository_context". source_context walks the post-image paths, increments "unavailable_files" for each context_blob failure and "omitted_files" for files that contributed no line, and then ends with(report.to_string().chars().count() <= budget).then_some(report)(line 409); when the serialized report is over budget the function returns None and every counter and excerpt already collected is thrown away, so the pass prompt carries"repository_context": null, which the prompt text describes only as "no source context could fit". A reader of the request therefore cannot distinguish "the pinned head was not available locally" (the case the tests cover at crates/tui/src/tools/review_pr.rs) from "context was collected and then dropped", and the new sentence in docs/GITHUB_APP.md ("The request explicitly records unavailable files and omitted context") is not true of that branch. Impact is on the change's own evidence contract rather than on diff coverage. I did not construct an input that overflows — each file's allowance is derived from the remaining budget, so overflow requires the per-entry+1slack to accumulate across files — so treat reachability as unproven; the discarding branch itself is plain in the source, and the PR description already lists this as open gap #2. The budget < 512 early return at line 342 has the same all-or-nothing shape but no pre-built counters.
Suggestions
crates/tui/src/tools/review_pr.rs:409— Do not turn an over-budget report into a bare None. Keep the counters that were already computed — return the report with its files array emptied (or add an explicit "dropped"/"truncated" marker) so the pass prompt distinguishes 'no pinned head locally' from 'excerpts collected but not sent'; otherwise qualify the docs/GITHUB_APP.md sentence that promises the request records unavailable files and omitted context. This needs a judgement call on the receipt shape, so no literal replacement is proposed.
Assessment
Pass 1: No defect in the five changed files could be demonstrated by inspection. The new source_context/context_blob path validates the pinned commit id, reads objects only through git ls-tree/git cat-file (no checkout, no filters, no fetch, symlink/submodule modes rejected, size and text-ness bounded), never clips a line, excludes lines already present in the diff, and its output is embedded only as supplementary evidence; the hunk parser is reused rather than reimplemented, and both the tool and CLI paths now do this work off the async runtime via spawn_blocking with the tool mapping failure to ToolError::execution_failed. No build, clippy or test run was performed by this review, so the author's compile/test claims are unverified here. Open questions, not asserted defects: (1) crates/tui/src/lib.rs is the one file missing from the supplied source context, so I could not check whether std::env::current_dir() in run_review is the same workspace the PR source/diff was resolved against; if it is not, every context_blob call fails and the change silently yields only unavailable_files counts for all passes; (2) the pass prompt is now JSON, so the diff's newlines/quotes are escaped (roughly one extra character per line) and the whole-plan manifest plus the untrimmed PR description also ride in the same string, none of which is subtracted when the context budget is computed as max_chars_per_pass - pass.diff_chars — the plan's input accounting may under-describe the real request size, but no in-repo consumer enforcing prompt length was visible to me; (3) the tests added here were not executed, so the glob/literal-path, nested-workspace, symlink, binary and oversized-blob expectations are unverified.
Advisory review by Codewhale (codewhale review --pr 6240 --post, head 55fffb3dba729a0d522cf197012a2863f28e8509). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.
| .expect("source files") | ||
| .push(file); | ||
| } | ||
| (report.to_string().chars().count() <= budget).then_some(report) |
There was a problem hiding this comment.
[INFO] Over-budget repository_context drops the unavailable/omitted counters it already built
crates/tui/src/tools/review.rs::build_pr_pass_prompt passes the residual budget (plan.manifest.max_chars_per_pass - pass.manifest.diff_chars) to review_pr::source_context and puts its result in the request as "repository_context". source_context walks the post-image paths, increments "unavailable_files" for each context_blob failure and "omitted_files" for files that contributed no line, and then ends with (report.to_string().chars().count() <= budget).then_some(report) (line 409); when the serialized report is over budget the function returns None and every counter and excerpt already collected is thrown away, so the pass prompt carries "repository_context": null, which the prompt text describes only as "no source context could fit". A reader of the request therefore cannot distinguish "the pinned head was not available locally" (the case the tests cover at crates/tui/src/tools/review_pr.rs) from "context was collected and then dropped", and the new sentence in docs/GITHUB_APP.md ("The request explicitly records unavailable files and omitted context") is not true of that branch. Impact is on the change's own evidence contract rather than on diff coverage. I did not construct an input that overflows — each file's allowance is derived from the remaining budget, so overflow requires the per-entry +1 slack to accumulate across files — so treat reachability as unproven; the discarding branch itself is plain in the source, and the PR description already lists this as open gap #2. The budget < 512 early return at line 342 has the same all-or-nothing shape but no pre-built counters.
| .expect("source files") | ||
| .push(file); | ||
| } | ||
| (report.to_string().chars().count() <= budget).then_some(report) |
There was a problem hiding this comment.
Do not turn an over-budget report into a bare None. Keep the counters that were already computed — return the report with its files array emptied (or add an explicit "dropped"/"truncated" marker) so the pass prompt distinguishes 'no pinned head locally' from 'excerpts collected but not sent'; otherwise qualify the docs/GITHUB_APP.md sentence that promises the request records unavailable files and omitted context. This needs a judgement call on the receipt shape, so no literal replacement is proposed.
…it off 55fffb3 correctly diagnosed why `source_context_is_bounded_line_exact_and_uses_literal_paths` failed on Windows: `*` is a reserved NTFS filename character, so `std::fs::write(dir.path().join("[literal]*.rs"))` returned InvalidFilename (os 123) before a single assertion ran. That test has never executed on Windows since it was written. Gating it to unix accepts that permanently, and it gives up coverage on the one platform whose filename rules caused the failure — which is also the platform where a path bug is most likely to differ from the others. The fixture can just be spelled portably. `[l]iteral-other.rs` uses only NTFS-legal characters (the reserved set is `< > : " / \ | ? *`; brackets are fine) and proves exactly the same property: - read literally, the pathspec names that file; - read as a glob, `[l]` matches the single character `l` and the pattern resolves to `literal-other.rs` — the "wrong glob match" decoy the test already creates two lines down. So an implementation that globs still reads the decoy and still trips `assert!(!context.to_string().contains("wrong glob match"))`. The `nested/` relative-path decoy is unaffected. Verified empirically rather than from the spec: with both files present, the shell glob `[l]iteral-other.rs` selects `literal-other.rs`. The diagnosis in 55fffb3 is what made this fix obvious; only the remedy changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AJENKJ2smviQW4FVGzUTk9
Windows: restored rather than gated —
|
|
Claude finished @Hmbown's task in 1m 10s —— View job Review complete
Note: Summary
No new correctness bugs, provider/model-routing concerns (EPIC #2608 is untouched by this diff), security issues, or missing test coverage introduced by |
|
Merging with one documented red: |
Rescues
14eb885d4eofffix/github-review-quality-20260913, a branch that has had no PR since 2026-09-13 and that nobody had ever built.Partially addresses #6135. See "Why this does not close it" below.
What it does
Gives the GitHub App reviewer pinned source context around each hunk and pushes it toward concrete, evidence-bearing findings instead of generic commentary. 5 files, +406/−25: a new
crates/tui/src/tools/review_pr.rs, changes toreview.rsandreview_hunks.rs, onelib.rswiring line, anddocs/GITHUB_APP.md.Why a cherry-pick and not a branch merge
Do not merge that branch. It carries a second, unrelated commit —
4517d294c0, a Computer Use 0.2.2 sync — which conflicts across ~8crates/tui/plugins/computer-use/paths. An earlier audit reported the branch as "zero conflicts"; that measured the whole-branch merge and is misleading as written.The single sha applies clean, verified before picking:
Nothing rides along.
The part that mattered
The commit compiles but does not pass CI's clippy line:
Fixed in the second commit here. A clean cherry-pick tells you the text applied; it tells you nothing about whether the result builds under the gates. This one had sat unbuilt for two days.
Why this does not close #6135
Two gaps, both worth someone's attention rather than being buried under a merge:
RegistryMutationcontrol; what exists is an assertion in a commit message. Until that control exists as something CI runs, the precision claim is unverified.source_contextfails quiet on the cases that need it most. It returnsNonewhenbudget < 512, and on overflow it discards an already-built report including itsunavailable_filescounters. So the large PRs with the most surrounding context to lose get none, and emit no receipt saying so. That is a silent-failure shape, not a size limit.#6135 stays open for both.
Evidence
🤖 Generated with Claude Code