Skip to content

ci: add an advisory AI merge recommendation - #177

Open
littlegrasscao wants to merge 15 commits into
mainfrom
add-ai-merge-recommendation
Open

littlegrasscao wants to merge 15 commits into
mainfrom
add-ai-merge-recommendation

Conversation

@littlegrasscao

@littlegrasscao littlegrasscao commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

What did you change, and why?

Why: AI review findings are advisory, but they don't say whether the review as a whole recommends merging. This adds a deterministic, commit-specific merge recommendation from the AI review, so maintainers can see at a glance whether a PR is low-risk, without the bot approving anything.

Change:

  • Merge recommendation status. After the AI review's findings are published, the workflow posts an AI Review / Merge recommendation commit status on the reviewed head. It succeeds only when:

    • the review completed for the PR's current base and head, covered every selected file with none failed or waived, and had no tool failures;
    • there are no medium-or-higher or unknown-severity findings (low findings stay advisory);
    • no changed path, including a renamed file's old path, is sensitive: .github/, spec/, SECURITY.md, CODEOWNERS, anything with auth, credential, security, token or signer in its path, protocol request and response types, JPA entities, Flyway migrations, pagination code, or build, dependency and container files (pom.xml, Maven and Gradle wrappers, lockfiles, Dockerfiles and the like).

    The rules live in .github/scripts/ai_review_merge_recommendation.py, checked out from the base commit so a PR cannot change the rules it is judged by.

  • Summary comment. The workflow also puts the recommendation and its reason at the top of the AI review's summary comment, for example "Merge recommendation for abc1234: ❌ not recommended. … require human review", so maintainers see it in the conversation and not only in the checks list. Each run replaces the previous run's line.

  • Advisory only. The workflow never approves or requests changes on a PR, so the recommendation cannot satisfy the required review. Every PR still needs a maintainer's approval, and the maintainer decides whether to follow the recommendation. An earlier revision of this PR auto-approved PRs whose recommendation succeeded; that step was removed after review.

  • A new AI review policy workflow runs the classifier's unit tests when .github/scripts/ or that workflow changes.

  • CONTRIBUTING.md describes the status, its conditions, the summary line, and that it is advisory.

How do you know it works?

Testing:

  • 23 classifier unit tests pass locally under Python 3.13, and run in the new AI review policy workflow.
  • actionlint passes on ai-review.yml and ai-review-policy.yml.
  • The changed-paths step lists both names of a renamed file.
  • Ran the summary-comment step against a mocked GitHub API: it puts the line after the summary's hidden markers, replaces an earlier line instead of adding a second, keeps the review helper's checkpoint marker exactly once, ignores comments not written by the bot, and does nothing when there is no summary.
  • Replayed two real OCR artifacts: a complete no-findings review produced success, while a complete review with five medium findings produced failure.

End-to-end publication can only be exercised after merge, because pull_request_target loads workflow code from the default branch.

Sun Cao added 4 commits November 11, 2026 22:05
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
The merge recommendation now also needs human review for the sharing
protocol's request and response types and for database schema changes:
JPA entities and Flyway migrations.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
When the recommendation succeeds, the publish job approves the pull
request at the reviewed commit, so a low-risk change on a safe path needs
no human approval. It approves only while the base branch's rules dismiss
approvals on new pushes or require the last push to be approved, and only
if the pull request still has the reviewed base and head.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
- The path check also sees a renamed file's old path, so moving a file
  out of a sensitive path still needs human review.
- The workflow approves only pull requests whose author has write access;
  others still need a maintainer's approval.
- The publish job and the status step run unless the run was cancelled,
  instead of always.
- CI runs the merge recommendation's unit tests.
- CONTRIBUTING.md lists every condition for the approval.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao
littlegrasscao force-pushed the add-ai-merge-recommendation branch from 3d43510 to 22b48c5 Compare November 12, 2026 06:06
@littlegrasscao littlegrasscao changed the title ci: add deterministic AI merge recommendation ci: add an AI merge recommendation and auto-approve safe PRs Nov 12, 2026
@littlegrasscao
littlegrasscao marked this pull request as ready for review November 12, 2026 06:06
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

Comment thread .github/workflows/ai-review.yml Resolved
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Comment thread .github/workflows/ai-review.yml Resolved
Comment thread .github/workflows/ai-review.yml Resolved
pulls.listFiles stops at 3000 files, so a larger pull request could hide
a sensitive path from the merge policy. When the listed files fall short
of the pull request's changed_files, or reach the cap, the step writes an
empty list, which the classifier reports as an error; the findings are
still published. The policy test job also gets a five-minute timeout.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao

Copy link
Copy Markdown
Collaborator Author

On the summary finding about timeout-minutes: the policy test job now lives in its own ai-review-policy.yml workflow rather than ci.yml, and has timeout-minutes: 5 as of fa8e643. The server job's missing timeout predates this pull request, so it is left as is here.

The merge policy now compares the pull request's changed paths with the
paths the review actually covered and stays at failure when a
non-sensitive path is missing, including a rename's old path; coverage
items that record no path are an error. The approval guard paginates the
branch rules request and skips approving when a qualifying ruleset has
an always-bypass actor, whose pushes would not reset the approval. The
workflow header and CONTRIBUTING.md now state that the bot approval can
satisfy the branch's review requirement on its own, so a PR the policy
judges safe may merge without human review.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
The failure reason can quote changed file names, which the PR author controls
and which may contain newlines. Written to GITHUB_OUTPUT unchanged, a name like
"x\nstate=success" would add a second state line, and the last one wins.
Collapse control characters in the description and write the state last.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
A success here becomes a real approval, so changes that decide what ships
and runs, or how far a caller can list, should not rest on the LLM review
alone. Signed-URL code is already covered by the signer pattern.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Comment thread .github/workflows/ai-review.yml Resolved
Comment thread .github/workflows/ai-review.yml Resolved
…actors

A policy script failure no longer skips publishing the findings; the
status and approval treat the failed evaluation as an error. The approval
step now refuses a ruleset whose bypass actors the token cannot see,
instead of reading them as none. Evidence that isn't UTF-8 is an error
decision rather than a traceback.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@zhu-tom

zhu-tom commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Are we sure this is fine? Can the review AI be prompt-engineered to allow changes that are actually complex to merge with only AI approval?

@littlegrasscao
littlegrasscao marked this pull request as draft November 12, 2026 21:29
The workflow no longer submits an approval when the merge recommendation
succeeds. The AI Review / Merge recommendation status still says whether
the review recommends merging, but like the findings it is advisory: the
bot never approves or requests changes, so it cannot satisfy the required
review, and every PR still needs a maintainer's approval.

This removes the approval step with its branch-rule, bypass-actor and
author-permission guards, which only existed to protect that approval,
and the repository setting it needed.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao littlegrasscao changed the title ci: add an AI merge recommendation and auto-approve safe PRs ci: add an advisory AI merge recommendation Nov 12, 2026
@littlegrasscao
littlegrasscao marked this pull request as ready for review November 12, 2026 22:35
The recommendation was only a commit status, which a maintainer sees only
in the checks list. After publishing the status, the workflow now puts the
same verdict and reason at the top of the review's summary comment,
replacing the previous run's line. The status's fallback text no longer
says the review was not published when only the policy failed.

CONTRIBUTING.md now lists the pagination, build, dependency and container
paths the policy already treats as sensitive.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Comment thread .github/workflows/ai-review.yml Resolved
The changed paths came from the live pull request, while the review and
the status belong to the reviewed head. The step now lists the files
first and reads the pull request after, and keeps the paths only if its
head and base are still the reviewed ones; otherwise it writes an empty
list, which the classifier reports as an error.

Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao

Copy link
Copy Markdown
Collaborator Author

Are we sure this is fine? Can the review AI be prompt-engineered to allow changes that are actually complex to merge with only AI approval?

Good question. Yes, it could be steered, and that is why the AI no longer approves anything. Since b956046 the recommendation is advisory only:

  • The workflow never approves or requests changes, so it can't count toward the required review. main still requires a maintainer's approval on every PR, and the org setting that would let GitHub Actions approve PRs is off.
  • Only the AI's findings come from the model, and a diff can contain text that talks the model into reporting nothing. So a ✅ only means "the AI found nothing", never "safe to merge".
  • The checks a prompt can't influence still send a PR to a human: the changed paths come from the GitHub API, and any workflow, spec, protocol request/response, entity, Flyway, auth, credential, token, signing, pagination, build, dependency, or container path is never recommended. The policy also fails closed if the review didn't cover every changed file or ran against a different commit.

So the worst case of a manipulated review is a misleading ✅ next to a PR that still needs a maintainer's approval. The note in the summary comment and in CONTRIBUTING.md says so.

Comment thread .github/workflows/ai-review.yml Outdated
python3 .review-policy/.github/scripts/ai_review_merge_recommendation.py \
--result /tmp/ai-review/ocr-result.json \
--changed-paths /tmp/ai-review/changed-paths.json \
--expected-base "$EXPECTED_BASE_SHA" \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expected base here is the target branch tip, while manifest.input.resolved_base is the merge base of the target and PR head. So IIUC, these differ whenever main has advanced since the PR branched, so this check rejects a valid clean review?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in 69ac230. The authorize job now gets the merge base of the PR from the compare API (merge_base_commit.sha for base.sha...head.sha) and passes it as --expected-base, matching what the review runs from. The checks that notice a PR moving during the review still compare base.sha and head.sha. A new test covers a merge base that differs from the branch tip.

The review runs from the merge base of the target branch and the head, so the merge
recommendation now expects that commit rather than the target branch tip, which moves
on whenever the target branch advances.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Comment thread .github/workflows/ai-review.yml Resolved
…ry comment

The reason can quote changed file names, which the pull request author controls, so
it goes in a code span without backticks or angle brackets: mentions, links and HTML
stay inert, and the text cannot forge the block's markers.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants