🧪 [Testing] Add tests for opencode_in_progress and retry OpenCode agent failures#65
Conversation
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Warning Review limit reached
More reviews will be available in 20 minutes and 28 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthrough
ChangesRetryable OpenCode Agent Failure Detection & Dispatch
🎯 3 (Moderate) | ⏱️ ~20 minutes
Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
OpenCode Review Overview
|
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
OpenCode Agent approved this PR.
The PR introduces a new test file for opencode_in_progress with thorough test cases, ensuring the function behaves as expected in various scenarios. The changes are well-contained and do not introduce any risks.
- Result: APPROVE
- Reason: The PR adds comprehensive tests for the
opencode_in_progressfunction, covering all edge cases and scenarios. No blockers or regressions identified. - Head SHA:
f28319d94ffbcca9e74b784b5ca7943a5c7c56f9 - Workflow run: 27595613198
- Workflow attempt: 1
|
@copilot resolve the merge conflicts in this pull request |
Ignoring comment directed at |
…-in-progress-15660145001259273042 # Conflicts: # tests/test_pr_review_merge_scheduler.py
…5660145001259273042' into add-tests-opencode-in-progress-15660145001259273042
|
@copilot resolve the merge conflicts in this pull request |
Ignoring comment directed at |
…-in-progress-15660145001259273042 # Conflicts: # .jules/bolt.md # tests/scripts/ci/test_opencode_review_normalize_output.py # tests/test_pr_review_merge_scheduler.py # tests/test_vibesec.py
|
@copilot 리뷰와 로그를 확인하여 버그를 해결하세요. |
Ignoring comment directed at |
|
@copilot 리뷰와 로그를 확인하여 버그를 해결하세요. 그리고 merge conflicts가 있으면 해결하세요. |
Ignoring comment directed at |
Resolved locally in |
로컬에서 merge conflict를 해결하고 테스트를 확인했습니다 ( |
|
@copilot 리뷰와 로그를 확인하여 버그를 해결하세요. 그리고 merge conflicts가 있으면 해결하세요. Base branch 최신 변경 사항을 fetch 해도 됩니다. Python GIL Issue 사유로 Python 3.14 이상 버전을 사용하십시오. DB를 쓰는 경우 모든 DB 이름 객체는 두 단어 이상 조합으로 구성된 snake case여야 합니다. OpenCode Agent가 실패하는 경우 등에도 대응하여 버그를 고치십시오. |
Ignoring comment directed at |
There was a problem hiding this comment.
OpenCode Agent requested changes.
Critical security regressions and performance optimizations needed in scanner/cli/vibesec.py and scripts/ci/opencode_review_normalize_output.py
- Result: REQUEST_CHANGES
- Reason: Security and performance regressions identified
1. CRITICAL scanner/cli/vibesec.py:1 - Security rule regression
- Problem: Removed critical security rule for automated scanning in RULES_CURSOR and RULES_CLAUDE
- Root cause: Regression in security guardrails for OWASP rule enforcement
- Fix: Restore the removed rule: 'When dealing with OWASP rules or security vulnerabilities, automatically run
vibesec scan .as a hook or use tools likesemgrepto verify the codebase.' - Regression test: Add test cases to verify the presence of automated scanning rules in the output of
vibesec init - Suggested diff:
@@ -48,6 +48,7 @@
- Never set CORS to allow all origins on authenticated endpoints.
- Add tests for cross-user access denial on every resource endpoint.
+- When dealing with OWASP rules or security vulnerabilities, automatically run `vibesec scan .` as a hook or use tools like `semgrep` to verify the codebase.
See https://github.com/Seongho-Bae/VibeSec for full rules and checklists.2. HIGH scanner/cli/vibesec.py:1 - Performance regression in file scanning
- Problem: Replaced optimized _process_dir_entries with less efficient inline implementation
- Root cause: Performance optimization removed from file traversal logic
- Fix: Restore the _process_dir_entries function for optimized directory traversal
- Regression test: Benchmark file scanning performance with large directories
- Suggested diff:
@@ -466,6 +466,25 @@
return _RULES_CACHE[ext]
+def _process_dir_entries(dir_path: str):
+ """Process entries in a directory, yielding files and returning subdirectories."""
+ dirs = []
+ try:
+ with os.scandir(dir_path) as it:
+ for entry in it:
+ try:
+ if entry.is_symlink():
+ continue
+ if entry.is_dir(follow_symlinks=False):
+ if entry.name not in SKIP_DIRS and not entry.name.startswith("."):
+ dirs.append(entry.path)
+ elif entry.is_file(follow_symlinks=False):
+ _, ext = os.path.splitext(entry.name)
+ if ext.lower() not in SKIP_EXTENSIONS:
+ yield Path(entry.path)
+ except (OSError, PermissionError):
+ continue
+ except (OSError, PermissionError):
+ pass
+ return dirs
+
def _collect_files(base_path: Path):3. HIGH scripts/ci/opencode_review_normalize_output.py:1 - Validation logic regression
- Problem: Broken validation chain with separate validation functions removed
- Root cause: Validation logic fragmented and less robust
- Fix: Restore the original validation function structure with clear separation of concerns
- Regression test: Add test cases for invalid input scenarios
- Suggested diff:
@@ -9,6 +9,30 @@
from typing import Any
+def _validate_metadata(
+ value: dict[str, Any],
+ expected_head_sha: str,
+ expected_run_id: str,
+ expected_run_attempt: str,
+) -> bool:
+ if value.get("head_sha") != expected_head_sha:
+ return False
+ if value.get("run_id") != expected_run_id:
+ return False
+ if value.get("run_attempt") != expected_run_attempt:
+ return False
+ return True
+
+
+def _validate_result_and_reason(value: dict[str, Any]) -> bool:
+ result = value.get("result")
+ if result not in {"APPROVE", "REQUEST_CHANGES"}:
+ return False
+ if not isinstance(value.get("reason"), str) or not value["reason"].strip():
+ return False
+ if not isinstance(value.get("summary"), str) or not value["summary"].strip():
+ return False
+ return True
+
+
def valid_control(
value: Any,
*,
- Head SHA:
7ad96953db21de92cefd5c83bd233e9e2a03c60a - Workflow run: 27610353324
- Workflow attempt: 1
|
@copilot 리뷰와 로그를 확인하여 버그를 해결하세요. 그리고 merge conflicts가 있으면 해결하세요. Base branch 최신 변경 사항을 fetch 해도 됩니다. Python을 사용하는 경우 Python GIL Issue 사유로 Python 3.14 이상 버전을 사용하십시오. DB를 쓰는 경우 모든 DB 이름 객체는 두 단어 이상 조합으로 구성된 snake case여야 합니다. OpenCode Agent가 실패하는 경우 등에도 대응하여 버그를 고치십시오. |
Addressed in 3704fe6. I fetched |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
tests/test_pr_review_merge_scheduler.py (1)
159-197: ⚡ Quick win
trigger_reviews=False재시도 실패 분기 테스트도 추가해 주세요.프로덕션 코드의
retryable_failure분기는 현재trigger_reviews=True케이스만 검증되고, 비활성화 시block사유 분기(“after agent failure”)는 직접 검증이 없습니다. 이 케이스 하나를 추가하면 회귀 방지 효과가 큽니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_pr_review_merge_scheduler.py` around lines 159 - 197, Add a second test case to verify the behavior when trigger_reviews is set to False for the retryable agent failure scenario. The test should use the same PR data structure as test_inspect_pr_retries_after_retryable_agent_failure but call inspect_pr with trigger_reviews=False and verify that the returned Decision contains reason "block" and message "after agent failure" instead of "review_dispatch" and "retrying OpenCode review after agent failure". This ensures the block path in the retryable_failure branch is properly tested when review triggering is disabled.scripts/ci/pr_review_merge_scheduler.py (1)
181-183: ⚡ Quick win재시도 판정이 고정 문구 1개에 과도하게 결합되어 있습니다.
현재는 리뷰 본문 문자열이 조금만 바뀌어도 재시도 분기가 바로 무력화됩니다. 최소한 매처 문자열을 상수화하고, 핵심 키워드 기반(또는 다중 marker)으로 완화해 두는 편이 안전합니다.
제안 diff
+RETRYABLE_FAILURE_MARKERS = ( + "opencode agent review evidence was missing or invalid", + "reason: timeout", +) + def is_retryable_opencode_failure_review(review: dict[str, Any]) -> bool: - body = (review.get("body") or "").strip() - return "OpenCode Agent review evidence was missing or invalid." in body + body = (review.get("body") or "").strip().lower() + return any(marker in body for marker in RETRYABLE_FAILURE_MARKERS)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/ci/pr_review_merge_scheduler.py` around lines 181 - 183, The retry determination in the is_retryable_opencode_failure_review function is too tightly coupled to a single hardcoded string literal, making it brittle to any text changes in the review body. Extract the matcher string "OpenCode Agent review evidence was missing or invalid." as a named constant at the module level, and consider relaxing the matching logic to check for key phrases or keywords rather than requiring an exact string match. This approach will make the retry logic more resilient to minor variations in the review message while remaining maintainable.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@scripts/ci/pr_review_merge_scheduler.py`:
- Around line 181-183: The retry determination in the
is_retryable_opencode_failure_review function is too tightly coupled to a single
hardcoded string literal, making it brittle to any text changes in the review
body. Extract the matcher string "OpenCode Agent review evidence was missing or
invalid." as a named constant at the module level, and consider relaxing the
matching logic to check for key phrases or keywords rather than requiring an
exact string match. This approach will make the retry logic more resilient to
minor variations in the review message while remaining maintainable.
In `@tests/test_pr_review_merge_scheduler.py`:
- Around line 159-197: Add a second test case to verify the behavior when
trigger_reviews is set to False for the retryable agent failure scenario. The
test should use the same PR data structure as
test_inspect_pr_retries_after_retryable_agent_failure but call inspect_pr with
trigger_reviews=False and verify that the returned Decision contains reason
"block" and message "after agent failure" instead of "review_dispatch" and
"retrying OpenCode review after agent failure". This ensures the block path in
the retryable_failure branch is properly tested when review triggering is
disabled.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 0c697916-032f-489e-899d-62fc869f1669
📒 Files selected for processing (2)
scripts/ci/pr_review_merge_scheduler.pytests/test_pr_review_merge_scheduler.py
Fixed in |
There was a problem hiding this comment.
OpenCode Agent approved this PR.
The changes include optimizations and test additions without introducing regressions or security issues. The PR is well-structured and follows best practices.
- Result: APPROVE
- Reason: No blockers found
- Head SHA:
12033969643dc1d4773d1a68b8bed215b523063a - Workflow run: 27611887233
- Workflow attempt: 1
🎯 What: Adds unit tests for the
opencode_in_progressfunction inscripts/ci/pr_review_merge_scheduler.pyand updates the scheduler so retryable OpenCode agent failures do not permanently block review dispatch.📊 Coverage: The tests cover multiple scenarios including:
statusCheckRollupcontexts.opencode-review.COMPLETED,SUCCESS,FAILURE,ERROR).IN_PROGRESS,PENDING,QUEUED).CheckRun's workflow name (OpenCode Review).opencodeas in progress.CHANGES_REQUESTEDreviews caused by OpenCode agent failures such as invalid or missing review evidence.✨ Result: Test coverage is improved for a critical CI/CD path, and the scheduler now re-dispatches OpenCode review when the prior current-head OpenCode review failed in a retryable way instead of treating that agent failure as a permanent block on auto-merge.
PR created automatically by Jules for task 15660145001259273042 started by @seonghobae