Repository navigation
ci: add an advisory AI merge recommendation - #177
littlegrasscao wants to merge 15 commits into
Conversation
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
The merge recommendation now also needs human review for the sharing protocol's request and response types and for database schema changes: JPA entities and Flyway migrations. Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
When the recommendation succeeds, the publish job approves the pull request at the reviewed commit, so a low-risk change on a safe path needs no human approval. It approves only while the base branch's rules dismiss approvals on new pushes or require the last push to be approved, and only if the pull request still has the reviewed base and head. Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
- The path check also sees a renamed file's old path, so moving a file out of a sensitive path still needs human review. - The workflow approves only pull requests whose author has write access; others still need a maintainer's approval. - The publish job and the status step run unless the run was cancelled, instead of always. - CI runs the merge recommendation's unit tests. - CONTRIBUTING.md lists every condition for the approval. Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
3d43510 to
22b48c5
Compare
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s). |
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
pulls.listFiles stops at 3000 files, so a larger pull request could hide a sensitive path from the merge policy. When the listed files fall short of the pull request's changed_files, or reach the cap, the step writes an empty list, which the classifier reports as an error; the findings are still published. The policy test job also gets a five-minute timeout. Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
|
On the summary finding about |
The merge policy now compares the pull request's changed paths with the paths the review actually covered and stays at failure when a non-sensitive path is missing, including a rename's old path; coverage items that record no path are an error. The approval guard paginates the branch rules request and skips approving when a qualifying ruleset has an always-bypass actor, whose pushes would not reset the approval. The workflow header and CONTRIBUTING.md now state that the bot approval can satisfy the branch's review requirement on its own, so a PR the policy judges safe may merge without human review. Signed-off-by: Sun Cao <sun.cao+data@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
The failure reason can quote changed file names, which the PR author controls and which may contain newlines. Written to GITHUB_OUTPUT unchanged, a name like "x\nstate=success" would add a second state line, and the last one wins. Collapse control characters in the description and write the state last. Signed-off-by: Sun Cao <sun.cao+data@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
A success here becomes a real approval, so changes that decide what ships and runs, or how far a caller can list, should not rest on the LLM review alone. Signed-URL code is already covered by the signer pattern. Signed-off-by: Sun Cao <sun.cao+data@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
…actors A policy script failure no longer skips publishing the findings; the status and approval treat the failed evaluation as an error. The approval step now refuses a ruleset whose bypass actors the token cannot see, instead of reading them as none. Evidence that isn't UTF-8 is an error decision rather than a traceback. Signed-off-by: Sun Cao <sun.cao+data@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
Are we sure this is fine? Can the review AI be prompt-engineered to allow changes that are actually complex to merge with only AI approval? |
The workflow no longer submits an approval when the merge recommendation succeeds. The AI Review / Merge recommendation status still says whether the review recommends merging, but like the findings it is advisory: the bot never approves or requests changes, so it cannot satisfy the required review, and every PR still needs a maintainer's approval. This removes the approval step with its branch-rule, bypass-actor and author-permission guards, which only existed to protect that approval, and the repository setting it needed. Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
The recommendation was only a commit status, which a maintainer sees only in the checks list. After publishing the status, the workflow now puts the same verdict and reason at the top of the review's summary comment, replacing the previous run's line. The status's fallback text no longer says the review was not published when only the policy failed. CONTRIBUTING.md now lists the pagination, build, dependency and container paths the policy already treats as sensitive. Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
The changed paths came from the live pull request, while the review and the status belong to the reviewed head. The step now lists the files first and reads the pull request after, and keeps the paths only if its head and base are still the reviewed ones; otherwise it writes an empty list, which the classifier reports as an error. Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Good question. Yes, it could be steered, and that is why the AI no longer approves anything. Since b956046 the recommendation is advisory only:
So the worst case of a manipulated review is a misleading ✅ next to a PR that still needs a maintainer's approval. The note in the summary comment and in |
| python3 .review-policy/.github/scripts/ai_review_merge_recommendation.py \ | ||
| --result /tmp/ai-review/ocr-result.json \ | ||
| --changed-paths /tmp/ai-review/changed-paths.json \ | ||
| --expected-base "$EXPECTED_BASE_SHA" \ |
There was a problem hiding this comment.
Expected base here is the target branch tip, while manifest.input.resolved_base is the merge base of the target and PR head. So IIUC, these differ whenever main has advanced since the PR branched, so this check rejects a valid clean review?
There was a problem hiding this comment.
Good catch, fixed in 69ac230. The authorize job now gets the merge base of the PR from the compare API (merge_base_commit.sha for base.sha...head.sha) and passes it as --expected-base, matching what the review runs from. The checks that notice a PR moving during the review still compare base.sha and head.sha. A new test covers a merge base that differs from the branch tip.
The review runs from the merge base of the target branch and the head, so the merge recommendation now expects that commit rather than the target branch tip, which moves on whenever the target branch advances. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
…ry comment The reason can quote changed file names, which the pull request author controls, so it goes in a code span without backticks or angle brackets: mentions, links and HTML stay inert, and the text cannot forge the block's markers. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
What did you change, and why?
Why: AI review findings are advisory, but they don't say whether the review as a whole recommends merging. This adds a deterministic, commit-specific merge recommendation from the AI review, so maintainers can see at a glance whether a PR is low-risk, without the bot approving anything.
Change:
Merge recommendation status. After the AI review's findings are published, the workflow posts an
AI Review / Merge recommendationcommit status on the reviewed head. It succeeds only when:.github/,spec/,SECURITY.md,CODEOWNERS, anything withauth,credential,security,tokenorsignerin its path, protocol request and response types, JPA entities, Flyway migrations, pagination code, or build, dependency and container files (pom.xml, Maven and Gradle wrappers, lockfiles, Dockerfiles and the like).The rules live in
.github/scripts/ai_review_merge_recommendation.py, checked out from the base commit so a PR cannot change the rules it is judged by.Summary comment. The workflow also puts the recommendation and its reason at the top of the AI review's summary comment, for example "Merge recommendation for
abc1234: ❌ not recommended. … require human review", so maintainers see it in the conversation and not only in the checks list. Each run replaces the previous run's line.Advisory only. The workflow never approves or requests changes on a PR, so the recommendation cannot satisfy the required review. Every PR still needs a maintainer's approval, and the maintainer decides whether to follow the recommendation. An earlier revision of this PR auto-approved PRs whose recommendation succeeded; that step was removed after review.
A new
AI review policyworkflow runs the classifier's unit tests when.github/scripts/or that workflow changes.CONTRIBUTING.md describes the status, its conditions, the summary line, and that it is advisory.
How do you know it works?
Testing:
AI review policyworkflow.ai-review.ymlandai-review-policy.yml.End-to-end publication can only be exercised after merge, because
pull_request_targetloads workflow code from the default branch.