feat: enforce Copilot code review ruleset via sync script#9
Conversation
- Add sync_rulesets() to create and verify the Copilot code review ruleset on all repositories' default branches - Add rulesets config to baseline.json with deletion protection, non-fast-forward protection, and Copilot review rules - Add rulesets section to README with enforcement details - Add CodeRabbit manual setup note (no API for automation) - Add rulesets to baseline schema validation in quality-checks.yml
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR introduces GitHub branch protection rulesets for enforcing Copilot code reviews on default branches. Changes include new baseline configuration, a Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds organization-wide enforcement of a Copilot code review ruleset by extending the repo settings sync tooling, plus baseline configuration, documentation, and CI validation updates.
Changes:
- Add
sync_rulesets()to create/verify a Copilot review ruleset during repo sync runs. - Add a
rulesetssection toconfig/baseline.jsondefining the Copilot ruleset (and related protections). - Document ruleset behavior + CodeRabbit manual setup, and extend baseline schema checks to require
rulesets.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/sync-repo-settings.sh | Adds ruleset sync logic and includes it in the per-repo drift report. |
| config/baseline.json | Introduces baseline configuration for the Copilot code review ruleset. |
| README.md | Documents rulesets and CodeRabbit manual enablement. |
| CLAUDE.md | Updates project instructions to include rulesets in enforced settings. |
| .github/workflows/quality-checks.yml | Ensures baseline schema validation requires a rulesets section. |
scripts/sync-repo-settings.sh
Outdated
|
|
||
| # Check if the ruleset already exists | ||
| local existing | ||
| existing=$(gh api "repos/$OWNER/$repo/rulesets" --jq ".[] | select(.name == \"$ruleset_name\") | .id" 2>/dev/null || echo "") |
There was a problem hiding this comment.
The ruleset lookup builds a jq filter by interpolating ruleset_name into the --jq string. If the name contains quotes/backslashes (or other jq-significant chars), the filter can break and return the wrong ID. Prefer fetching the list JSON and piping to jq -r --arg name "$ruleset_name" 'map(select(.name == $name)) | .[0].id // empty' (or similar) to avoid injection/escaping issues and to guarantee a single ID.
| existing=$(gh api "repos/$OWNER/$repo/rulesets" --jq ".[] | select(.name == \"$ruleset_name\") | .id" 2>/dev/null || echo "") | |
| existing=$(gh api "repos/$OWNER/$repo/rulesets" 2>/dev/null | jq -r --arg name "$ruleset_name" 'map(select(.name == $name)) | .[0].id // empty' 2>/dev/null || echo "") |
| if [ -n "$existing" ]; then | ||
| # Verify enforcement is active | ||
| local current_enforcement | ||
| current_enforcement=$(gh api "repos/$OWNER/$repo/rulesets/$existing" --jq '.enforcement' 2>/dev/null || echo "") | ||
| if [ "$current_enforcement" != "active" ]; then | ||
| changes="- Copilot review ruleset: enforcement \`$current_enforcement\` -> \`active\`\n" | ||
| if [ "$MODE" = "--apply" ]; then | ||
| gh api -X PUT "repos/$OWNER/$repo/rulesets/$existing" \ | ||
| --input <(echo "$effective" | jq '.rulesets.copilot_code_review') \ | ||
| > /dev/null 2>&1 || log "WARN: Could not update ruleset for $repo" | ||
| log "APPLIED ruleset enforcement for $repo" | ||
| else | ||
| log "DRIFT detected in ruleset enforcement for $repo" | ||
| fi | ||
| else | ||
| log "OK: Copilot review ruleset for $repo" | ||
| fi |
There was a problem hiding this comment.
sync_rulesets only checks whether enforcement is active, but it doesn’t validate the rest of the ruleset (rules/conditions/target). That means a repo can be reported as OK even if protections like deletion / non_fast_forward / copilot_code_review were changed or removed. Consider diffing the current ruleset JSON against the desired config (after normalizing away server-managed fields like id, timestamps, etc.) and treating any mismatch as drift (and applying an update in --apply mode).
scripts/sync-repo-settings.sh
Outdated
|
|
||
| # Check if the ruleset already exists | ||
| local existing | ||
| existing=$(gh api "repos/$OWNER/$repo/rulesets" --jq ".[] | select(.name == \"$ruleset_name\") | .id" 2>/dev/null || echo "") |
There was a problem hiding this comment.
The ruleset list call (gh api repos/$OWNER/$repo/rulesets) is likely paginated. Without --paginate, repos with enough rulesets may not return the target ruleset in the first page, causing false “missing” drift and potentially creating duplicates. Use gh api --paginate (and/or increase per_page) when listing rulesets.
| existing=$(gh api "repos/$OWNER/$repo/rulesets" --jq ".[] | select(.name == \"$ruleset_name\") | .id" 2>/dev/null || echo "") | |
| existing=$(gh api --paginate "repos/$OWNER/$repo/rulesets?per_page=100" --jq '.[] | select(.name == $ruleset_name) | .id' --arg ruleset_name "$ruleset_name" 2>/dev/null || echo "") |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/quality-checks.yml (1)
90-100:⚠️ Potential issue | 🟠 MajorValidate the ruleset schema, not just the top-level key.
This only proves
.rulesetsexists. A brokenrulesets.copilot_code_reviewobject would still pass CI, andscripts/sync-repo-settings.shLines 383-388 then short-circuit ruleset sync when.nameis empty.As per coding guidelines, "If baseline.json is updated, ensure the rulesets section is present and correctly structured to pass CI."🧪 Proposed fix
for section in repo_settings security branch_protection rulesets labels required_files; do if ! jq -e ".$section" config/baseline.json > /dev/null 2>&1; then echo "ERROR: Missing section '$section' in baseline.json" ERRORS=$((ERRORS + 1)) else echo "OK: section '$section' present" fi done + if ! jq -e ' + .rulesets.copilot_code_review + | type == "object" + and (.name | type == "string") + and (.enforcement | type == "string") + and (.target | type == "string") + and (.conditions.ref_name.include | type == "array") + and (.rules | type == "array" and length > 0) + ' config/baseline.json > /dev/null 2>&1; then + echo "ERROR: rulesets.copilot_code_review is missing required fields" + ERRORS=$((ERRORS + 1)) + fi # Validate label structure🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/quality-checks.yml around lines 90 - 100, The current check only verifies the top-level .rulesets key; change the validation to assert the ruleset object structure (e.g., .rulesets.copilot_code_review and its .name field is present and non-empty) so broken or empty ruleset objects fail CI; update the workflow loop to use jq checks like ensuring ".rulesets.copilot_code_review | has(\"name\") and .name != \"\" " (or equivalent) and add similar assertions for any other required nested keys referenced by sync logic (see rulesets.copilot_code_review and the code path in scripts/sync-repo-settings.sh that short-circuits when .name is empty) so the job errors when the nested schema is missing or malformed.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/sync-repo-settings.sh`:
- Around line 394-410: Currently the script only checks current_enforcement and
ignores other ruleset fields; instead fetch the full existing ruleset JSON via
gh api (the same endpoint used for $existing) and compare it to the desired
payload (the effective payload produced by jq '.rulesets.copilot_code_review'),
and treat any diff as drift. Replace the current_enforcement check (variables
current_enforcement/existing) with a deep compare between the existing ruleset
JSON and the effective ruleset JSON, set changes to reflect differences, and
when MODE="--apply" PUT the full effective payload to
"repos/$OWNER/$repo/rulesets/$existing" (using the same --input <(echo
"$effective" | jq '.rulesets.copilot_code_review')) while preserving the
existing logging behavior (log WARN on failure, log APPLIED on success, log
DRIFT when not applying).
- Around line 391-417: The code treats any gh api failure as a "missing" ruleset
and always logs "APPLIED" even when POST/PUT fail; fix by checking gh api exit
statuses and outputs rather than swallowing errors. For the initial lookup
(existing) run gh api capturing both output and exit code (e.g., capture output
into a var and check $?), if the call failed log a specific warning and skip
treating the repo as "missing"; for the PUT/POST calls (the gh api -X PUT/POST
lines) capture their exit codes and only set changes/log "APPLIED ..." when the
gh api call returns success; on failure log a clear WARN with the gh output and
do not mark as applied. Use the existing variable names (existing,
current_enforcement, MODE, effective, log) and the same gh api invocation points
to implement these checks.
---
Outside diff comments:
In @.github/workflows/quality-checks.yml:
- Around line 90-100: The current check only verifies the top-level .rulesets
key; change the validation to assert the ruleset object structure (e.g.,
.rulesets.copilot_code_review and its .name field is present and non-empty) so
broken or empty ruleset objects fail CI; update the workflow loop to use jq
checks like ensuring ".rulesets.copilot_code_review | has(\"name\") and .name !=
\"\" " (or equivalent) and add similar assertions for any other required nested
keys referenced by sync logic (see rulesets.copilot_code_review and the code
path in scripts/sync-repo-settings.sh that short-circuits when .name is empty)
so the job errors when the nested schema is missing or malformed.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: c0de2fab-dfc6-4f57-a0c4-6d7d26bcdfc1
📒 Files selected for processing (5)
.github/workflows/quality-checks.ymlCLAUDE.mdREADME.mdconfig/baseline.jsonscripts/sync-repo-settings.sh
- Use jq --arg for safe ruleset name matching (no injection risk) - Bail early if rulesets API call fails (don't treat errors as missing) - Compare full ruleset config (name, enforcement, target, conditions, rules) instead of just enforcement field - Only log APPLIED on successful API calls
Summary
sync_rulesets()function to the sync script that creates and verifies the Copilot code review ruleset on all repositoriesconfig/baseline.json(deletion protection, non-fast-forward, Copilot review)rulesetsto baseline schema validation in quality-checks.ymlTest plan
rulesetsschema sectionSummary by CodeRabbit
New Features
Documentation