Improve bot finding extraction

[gauravp-cubic]

Summary

This PR tightens the bot-comment extraction step so it extracts concrete code-review findings, not only comments phrased as explicit imperative suggestions.

The change is intentionally limited to extraction input/prompting:

• Reframe bot extraction around concrete review findings / flagged issues.
• Present bot comments before diffs, since comments are the primary source and diffs are context.
• Number and label comment blocks as REVIEW_BODY, INLINE_REVIEW_COMMENT, or ISSUE_COMMENT.
• Strip hidden/generated Markdown before extraction (<!-- ... -->, collapsed <details>, <sub> metadata, and generated footers).
• Align structured-output descriptions with the findings framing, without changing stored field names.

Why

The current extractor is framed around "actionable suggestions." In real review comments, a reviewer may clearly flag a defect or risk without writing it as "change X to Y." Those comments should still count as findings: if the later PR changes address the flagged issue, the benchmark should be able to match that review finding to the fix.

This is a measurement-correctness issue rather than a reviewer-specific rule. It affects comment styles differently: reviewers whose comments already look like explicit suggestions are easier for the current prompt to extract, while issue-first or risk-first comments can be under-extracted.

Each part of this extraction shape supports that same measurement goal:

• Findings framing: Review comments often describe the problem, risk, or defect directly. Asking for concrete findings lets the extractor capture those comments without requiring imperative fix wording.
• Comments before diffs: The bot comments are the evidence being measured. Putting them first makes the extraction task explicit, while the diff remains available as supporting context for file paths, line numbers, and code meaning.
• Numbered comment blocks: Stable COMMENT C1, COMMENT C2, ... boundaries make it easier for the model to scan every review round and reduce accidental skipping when a PR has many comments.
• Comment type labels: INLINE_REVIEW_COMMENT, REVIEW_BODY, and ISSUE_COMMENT give neutral structure that reflects GitHub source data. Inline review comments are especially likely to contain code-review findings, while review bodies and issue comments can contain summaries or additional findings.
• Generated Markdown cleanup: Hidden comments, collapsible prompt blocks, metadata footers, and bot-generated boilerplate are not the review finding itself. Removing them focuses the extractor on the human-visible review text and reduces chances that operational scaffolding is extracted as a finding.
• Specific footer stripping: Generated footer detection now uses specific footer phrases such as configuration used, rather than broad common words. That keeps known generated footer cleanup while preserving real findings that mention something being configured incorrectly.
• Structured-output wording: The schema descriptions now match the prompt's findings terminology, so the model receives one consistent task definition from prompt to response format while keeping the stored field names stable.

Local Evidence

The table below lists concrete examples from a local paired run for 2026-06-29 <= bot_reviewed_at < 2026-07-06. In each case, the revised extraction shape included a substantive review finding that the production extraction left out.

Reviewer	Recovered review finding	Evidence from paired run
Cubic	aidenybai/react-grab#515	Recovered an inline P1 finding about preview mode intercepting the normal Cmd/Ctrl+C copy activation path. Production extraction only picked up a later option-parsing issue from the same PR.
Cubic	LuoxuanLove/godot-dotnet-mcp#478	Recovered an inline finding about logging an unsanitized client-controlled request id in cancellation handling. This was a substantive security/logging issue in a follow-up review round.
Greptile	MervinPraison/PraisonAI#2408	Recovered an inline finding about concurrent idempotent task creation racing between select and insert. Production extraction captured only one issue from this PR.
Greptile	MervinPraison/PraisonAI#2408	Recovered an inline finding about invalid retry limits causing tasks to block immediately. The comment uses badge/title formatting and states the defect directly.
Codex	pedropaulovc/agent-plugins#28	Recovered an inline finding about paginated formatter responses producing multiple partial markdown documents unless slurped. The finding came from a later review round.
Codex	pedropaulovc/agent-plugins#28	Recovered an inline finding about body-only reviews being suppressed as feedback. The comment describes a concrete review concern even though it is phrased as a problem statement.
Gemini	aeroxy/chrome-devtools-cli#14	Recovered an inline finding about interpolating an unvalidated function name into JavaScript, creating code-execution risk. Production extraction captured earlier summary-style issues from the PR.
Gemini	aeroxy/chrome-devtools-cli#14	Recovered an inline finding about a fill helper mishandling checkbox/radio inputs. The revised extraction handled it as a distinct concrete finding.

Together, these examples show the intended measurement improvement: the extractor more reliably reads substantive issue-first comments across inline comments and later review rounds.

What This Does Not Change

• No matching/scoring logic changes.
• No human-action extraction changes.
• No model/provider/routing changes.
• No discovery, enrichment, GH Archive, or local-runner changes.
• No bot-specific rules or reviewer-specific markers.
• No generated benchmark artifacts or local analysis outputs.

Validation

• uv run pytest tests/test_analyze_formatting.py
• uv run python -m compileall llm/prompts.py llm/schemas.py pipeline/analyze.py tests/test_analyze_formatting.py

[ashleyzhang01]

i get what this is trying to do, but putting a specific bot name may be too targeted. and if they change format in the future, it would be affected

[ashleyzhang01]

i'd prefer not to strip <details> and <sub> blocks. some bots may put actual review content inside <details>, and we can't guarantee every bot only uses these tags for boilerplate. the extraction prompt already instructs the LLM to skip summaries and configuration, so the redundancy is handled there too. the <!-- --> stripping is fine.

[gauravp-cubic]

Thanks, agreed. I pushed a follow-up that removes the bot-specific footer heuristic and no longer strips visible Markdown like <details> or <sub>. The formatter now only strips hidden HTML comments, while keeping the neutral comment numbering/type labels and extraction prompt framing.