chore: stop nightly live tests from running on a daily schedule#478
Conversation
📝 WalkthroughWalkthroughThis PR adds resilient transient-error handling to live integration tests. A new ChangesTransient Upstream Failure Handling
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
@copilot please address the blocking signals on this PR and push a fix commit. Trigger: blocking-review Follow the PR's |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/nightly.yml:
- Around line 3-8: You removed the scheduled trigger and weakened CI in the
workflow by replacing the daily cron with only workflow_dispatch; revert the
change in the .github/workflows/nightly.yml so the original 'on:' schedule
(cron/daily) and original fail-fast/CI behavior are restored, or if the change
is required, add an explicit approval gate (e.g., require a repo
owner/maintainer sign-off in the PR description or add a manual approval step)
and document that approval in the PR; locate the change around the 'on:' block
and the workflow_dispatch entry to restore the original trigger and CI settings
or to add the required approval metadata before merging.
In `@tests/integration/test_free_channels_live.py`:
- Around line 216-218: Docstring and query use non-English text; replace the
Chinese docstring and the Chinese search query with English-only text. Update
the module-level docstring that currently reads the Baidu Tieba live test
description to an English equivalent (e.g., "Baidu Tieba live test — high
captcha/rate-limit risk; run outside default live."), and change the search
invocation where _get_channel("tieba") is used with _search_or_skip(...) to pass
an English query string instead of "AI 编程助手" (e.g., "AI coding assistant").
Ensure you only modify the string literals in the docstring and the call to
_search_or_skip while keeping function names _get_channel and _search_or_skip
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 85922237-1192-43d3-a0ae-8f0e62a547bf
📒 Files selected for processing (2)
.github/workflows/nightly.ymltests/integration/test_free_channels_live.py
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8b80aa1503
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@copilot please address the blocking signals on this PR and push a fix commit. Trigger: blocking-review 💡 Codex ReviewHere are some automated review suggestions for this pull request. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". Follow the PR's |
There was a problem hiding this comment.
Pull request overview
This PR reduces false-alarm failures from the live “free channels” integration suite by treating upstream/transient errors as skips and by changing the GitHub Actions workflow to run only on manual dispatch.
Changes:
- Add a
_search_or_skip()helper in the live integration tests to skip on specific typed upstream/transient exceptions. - Update the nightly workflow to remove the scheduled cron trigger and improve pytest reporting (
-ra, no-x).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
tests/integration/test_free_channels_live.py |
Routes live searches through a skip-on-transient wrapper and updates exception handling imports. |
.github/workflows/nightly.yml |
Removes the daily schedule and adjusts pytest flags for clearer manual-run output. |
The Nightly live channel suite hits ~14 real public endpoints from a shared CI datacenter IP and asserts each returns results. Public services routinely rate-limit (429), block datacenter traffic (403) or return transient 5xx, so at least one fails on nearly every nightly run -- producing a daily false-alarm failure email with no actionable signal. Remove the daily cron trigger; keep workflow_dispatch so the suite can still be run on demand from the Actions tab, where a human can read the result. The job still fails loudly on any real regression when run. Authorized by the repo owner (0xmariowu) in lieu of an automated approval gate. https://claude.ai/code/session_01WJdXSEnHXZ8bhuV33vBN1R
0xmariowu
force-pushed
the
claude/daily-auto-test-issue-GOxWD
branch
from
320d2b9 to
3363c76
Compare
0xmariowu
changed the title
Problem
The
Nightly — Live Integration Testsworkflow fired a failure email nearly every day. Root cause: it hits ~14 real public free endpoints (arxiv, reddit, dblp, pubmed, ddgs…) from a shared CI datacenter IP and asserts every one returns results. Public services routinely rate-limit (429), block datacenter traffic (403), or return transient 5xx — so at least one of the 14 fails on essentially every run. These are upstream conditions, not regressions in AutoSearch.Reproduced locally: arxiv
429, reddit403, dblp500.Change
Remove the daily
crontrigger fromnightly.yml.workflow_dispatchis kept, so the suite can still be run on demand from the Actions tab, where a human can read the result. When run, the job still fails loudly on any real regression — no test behaviour is changed.This is the minimal change that stops the daily false-alarm emails.
Trade-off
No daily auto-run means no automatic alert if a platform genuinely breaks our parsing. Run the workflow manually (or add a weekly cron later) when you want that regression check.
Workflow edit authorized by the repo owner (@0xmariowu) in lieu of an automated approval gate.
https://claude.ai/code/session_01WJdXSEnHXZ8bhuV33vBN1R