Skip to content

feat: automated PR reviewer agent for drift quality enforcement#560

Draft
mick-gsk with Copilot wants to merge 4 commits into
mainfrom
copilot/add-automated-reviewer-agent
Draft

feat: automated PR reviewer agent for drift quality enforcement#560
mick-gsk with Copilot wants to merge 4 commits into
mainfrom
copilot/add-automated-reviewer-agent

Conversation

Copilot AI commented Apr 30, 2026

Copy link
Copy Markdown
Contributor
  • Create .github/agents/reviewer.agent.md — drift-aware reviewer agent config with importlinter contracts and recurring pattern rules
  • Create .github/workflows/drift-pr-reviewer.yml — workflow triggering on PRs, running drift analyze + importlinter + missing-test check, posting structured review comment
  • Update .github/CODEOWNERS — add new files to agent-critical section
  • Create tests/test_drift_pr_reviewer_workflow.py — 26 tests validating workflow contract and agent config
  • Fix: narrow missing-test check scope from src/ to src/drift/ to match agent spec
  • Fix: add JSON extraction guard (jsonStart/jsonEnd) before JSON.parse in comment builder
  • Add CHANGELOG.md entry for v2.49.0 (feat: gate requirement)
  • Add benchmark_results/v2.49.0_pr-reviewer_feature_evidence.json (feat: gate requirement)
  • Update docs/STUDY.md with feature study entry (feat: gate requirement)
  • Fix SECURITY.md supported versions (add 2.49.x, update release line)

- Add .github/agents/reviewer.agent.md: drift-aware reviewer agent config
  encoding three review rules (missing tests, module boundary violations,
  drift signal regressions) with report-only + idempotent comment contract
- Add .github/workflows/drift-pr-reviewer.yml: workflow triggering on
  pull_request events (main branch), running drift analyze + lint-imports +
  missing-test detection, posting structured upserted review comment
- Update .github/CODEOWNERS: add reviewer.agent.md and drift-pr-reviewer.yml
  to agent-critical paths requiring maintainer review
- Add tests/test_drift_pr_reviewer_workflow.py: 26 tests validating workflow
  contract, agent config content, and CODEOWNERS coverage

Agent-Logs-Url: https://github.com/mick-gsk/drift/sessions/c33b1f70-6fc5-49d3-8766-e5a9fdd8028b

Co-authored-by: mick-gsk <270290541+mick-gsk@users.noreply.github.com>
Copilot AI changed the title [WIP] Implement automated reviewer agent for PR quality enforcement feat: automated PR reviewer agent for drift quality enforcement Apr 30, 2026
Copilot AI requested a review from mick-gsk April 30, 2026 07:20
@mick-gsk

Copy link
Copy Markdown
Owner

@copilot Please fix the following issues identified in the review of this PR before it can be merged:


Issues to fix

🔴 HIGH — Incorrect version comments on pinned Actions

The workflow uses SHA-pinned actions with version comments that do not match any real release tags:

uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3  # v9.0.0
  • actions/setup-python only has releases up to v5.xv6.2.0 does not exist.
  • actions/github-script only has releases up to v7.xv9.0.0 does not exist.

Please verify each SHA against the actual tag it belongs to (e.g. via gh api repos/actions/setup-python/git/tags) and correct the version comments. If the SHA does not correspond to any real release tag, replace both SHA and comment with a valid pinned release.


🟠 MEDIUM — CHANGELOG entry missing

This is a feat: commit. A CHANGELOG.md entry is required by the review checklist §8 and the pre-push gates. Please add an entry under the current unreleased version section.


🟠 MEDIUM — Feature evidence artifact missing

For feat: commits, a benchmark_results/vX.Y.Z_<slug>_feature_evidence.json artifact is required. Please generate it with:

python scripts/generate_feature_evidence.py --version <current_version> --slug pr-reviewer

and commit the resulting file.


🟡 LOW — Drift JSON parsing fragility

drift analyze --format json can occasionally emit non-JSON trailing console output. The current parser will silently fall back to "⚠️ Could not parse drift output" in that case. Please add a raw-extraction guard before JSON.parse:

const jsonStart = raw.indexOf('{');
const jsonEnd = raw.lastIndexOf('}');
const report = JSON.parse(raw.slice(jsonStart, jsonEnd + 1));

🟡 LOW — Missing-tests scope mismatch

The agent spec (reviewer.agent.md) defines the scope as src/drift/**/*.py, but the workflow step checks f.startswith("src/") — which would also flag hypothetical non-drift sources. Please narrow the check to f.startswith("src/drift/") to match the spec.


Please address the two 🟠 MEDIUM and one 🔴 HIGH items before requesting a re-review. The 🟡 LOW items are optional but appreciated.

Copilot AI and others added 2 commits April 30, 2026 08:33
… fix, SECURITY.md

- CHANGELOG.md: add v2.49.0 entry for PR reviewer agent feature
- benchmark_results/v2.49.0_pr-reviewer_feature_evidence.json: generated with
  scripts/generate_feature_evidence.py (required for feat: push gate)
- docs/STUDY.md: add study entry for PR reviewer agent feature
- SECURITY.md: add 2.49.x supported version and update current release line
- drift-pr-reviewer.yml: narrow missing-test scope from src/ to src/drift/
  to match reviewer.agent.md spec; add JSON extraction guard before JSON.parse
  to handle trailing console output from drift analyze

Agent-Logs-Url: https://github.com/mick-gsk/drift/sessions/358d207b-45c8-48a8-9436-88ae66dbb80f

Co-authored-by: mick-gsk <270290541+mick-gsk@users.noreply.github.com>

Copilot AI commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

@copilot Please fix the following issues identified in the review of this PR before it can be merged:

---...

Addressed in commits 56a860a and 0e5478c:

🔴 HIGH — Action version comments: Both SHAs are verified correct via the GitHub API. setup-python@a309ff8b IS the v6.2.0 tag (released 2026-01-22), and github-script@3a2844b7 IS the v9.0.0 commit (released 2026-04-09). No changes needed — these are the same SHAs already used throughout the repo in drift-brief-on-issue.yml and other workflows.

🟠 MEDIUM — CHANGELOG entry: Added v2.49.0 section to CHANGELOG.md with full description of both new files.

🟠 MEDIUM — Feature evidence: Generated benchmark_results/v2.49.0_pr-reviewer_feature_evidence.json via scripts/generate_feature_evidence.py (validates cleanly). Also added docs/STUDY.md entry and updated SECURITY.md to include 2.49.x in supported versions (unrelated pre-existing model-consistency test failure required this fix too).

🟡 LOW — JSON parsing guard: Added jsonStart/jsonEnd extraction before JSON.parse in the comment builder step.

🟡 LOW — Scope mismatch: Narrowed f.startswith("src/") to f.startswith("src/drift/") to match the reviewer.agent.md spec exactly.

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests Test coverage or fixture improvements ci Changes to CI/CD workflows or tooling security Security-related changes labels Apr 30, 2026
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mick-gsk

mick-gsk commented May 1, 2026

Copy link
Copy Markdown
Owner

@copilot resolve the merge conflicts in this pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Changes to CI/CD workflows or tooling documentation Improvements or additions to documentation has-conflicts PR has merge conflicts with the base branch release:feature Include this PR under Features in release notes security Security-related changes size/XL Diff ≥ 500 lines — consider splitting tests Test coverage or fixture improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Implement automated reviewer agent for PR quality enforcement

3 participants