Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 61 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ jobs:
cd tests
python test_modules.py

- name: Run bot-blocking tests
run: |
cd tests
python test_bot_blocking.py

test-action:
runs-on: ubuntu-latest
name: Test Link Checker Action
Expand Down Expand Up @@ -66,13 +71,67 @@ jobs:
fail-on-broken: 'true'

- name: Test with silent codes for broken links
continue-on-error: true
continue-on-error: true
uses: ./
with:
html-path: 'tests/broken-links.html'
fail-on-broken: 'true'
silent-codes: '404,500' # Treat these as silent


# The ignore-patterns fixture is built here rather than kept in tests/ so
# that the counts below are exact and the other jobs keep scanning the
# same files they always have. Every host is under .invalid, which is
# reserved and never resolves, so the ignored case makes no requests at
# all and the un-ignored case fails fast.
- name: Create ignore-patterns fixture
run: |
mkdir -p ci-fixtures
cat > ci-fixtures/ignore-patterns.html <<'HTML'
<html><body>
<a href="https://fred.stlouisfed.org.invalid/">FRED</a>
<a href="https://fred.stlouisfed.org.invalid/series/UNRATE">UNRATE</a>
<a href="https://blocked.example.invalid/page">Blocked</a>
</body></html>
HTML

- name: Test ignore-patterns skips matching links
id: test-ignored
uses: ./
with:
html-path: 'ci-fixtures'
fail-on-broken: 'true'
ai-suggestions: 'false'
timeout: 10
ignore-patterns: |
https://fred\.stlouisfed\.org\.invalid/.*
# comments and blank lines are skipped

blocked\.example\.invalid

- name: Test ignore-patterns off reports the same links
id: test-not-ignored
uses: ./
with:
html-path: 'ci-fixtures'
fail-on-broken: 'false'
ai-suggestions: 'false'
timeout: 10

- name: Verify ignore-patterns counts
run: |
assert() {
if [ "$2" != "$3" ]; then
echo "::error::$1: expected '$3', got '$2'"
exit 1
fi
echo "ok: $1 = $2"
}
assert "ignored-count with patterns" "${{ steps.test-ignored.outputs.ignored-count }}" "3"
assert "broken-link-count with patterns" "${{ steps.test-ignored.outputs.broken-link-count }}" "0"
assert "broken-links-found with patterns" "${{ steps.test-ignored.outputs.broken-links-found }}" "false"
assert "ignored-count without patterns" "${{ steps.test-not-ignored.outputs.ignored-count }}" "0"
assert "broken-link-count without patterns" "${{ steps.test-not-ignored.outputs.broken-link-count }}" "3"

test-empty-directory:
runs-on: ubuntu-latest
name: Test Empty Directory
Expand Down
40 changes: 27 additions & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- Initial release of the AI-Powered Link Checker action
- `ignore-patterns` input: newline-separated regular expressions for URLs to skip entirely. Matching URLs are never requested, so they can be reported as neither broken nor redirected, and are counted separately via the new `ignored-count` output
- `update-existing-issue` input (default `true`): reuse the newest open issue with the same title and `broken-links` label, refreshing its body, instead of opening a duplicate on every run. New `issue-updated` output reports which path was taken
- Status `0` — a request that never completed — can now be listed in `silent-codes`. It applies to transport failures only (timeout, connection failure, a response broken mid-stream), not to malformed links or redirect loops

### Changed
- **Behaviour change:** with `create-issue: 'true'`, a recurring finding now refreshes one open issue rather than opening a new one per run. Set `update-existing-issue: 'false'` to restore the previous behaviour
- A crash in `link_checker.py` now fails the action with the checker's stderr in the job log, instead of aborting the step with no diagnostic
- `$GITHUB_OUTPUT` heredocs use a per-run delimiter, so scanned link text containing a line reading `EOF` can no longer truncate an output

### Fixed
- The `legitimate_domains` allowance in bot-blocking detection required the error string `Connection Error`, but the timeout handler passes `timeout`. A listed domain was therefore protected against connection errors and reported broken on timeouts. Both are now treated alike, keyed on which handler caught the failure rather than on the error text
- `silent-codes` was only consulted on responses that returned a status code, so it could never apply to timeouts or connection errors
- `compile_ignore_patterns` caught only `re.error`, so a pattern raising `OverflowError` (an oversized repetition count) or `RecursionError` aborted the whole run instead of being skipped as documented
- The checker's stderr was written to a file that was never displayed, so its warnings — including a skipped ignore pattern — could not reach the job log
- The test runner called `unittest.main(exit=False)` without inspecting the result, so the CI test job reported success even when tests failed
- `tests/test_bot_blocking.py` used a pre-migration `sys.path`, so it could not import the module under test; it now asserts rather than printing, and CI runs it
- Corrected the action's self-referencing links in issue bodies, PR comments, artifacts and `examples.md`, which still pointed at the pre-migration `QuantEcon/meta` path
- The generated issue body claimed later runs would refresh it in place even when `update-existing-issue` was `false`
- The temporary ignore-patterns file is removed via an `EXIT` trap, so it is not left behind when the step exits early

## [1.0.0] - 2025-10-01

### Added
- Initial stable release migrated from QuantEcon/meta repository
- Smart link validation with configurable timeouts
- AI-powered suggestions for broken and redirected links
- Bot-blocking detection and handling
Expand All @@ -17,21 +40,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- GitHub issue creation with detailed reports
- Workflow artifact generation with link analysis
- MyST Markdown and Jupyter Book compatibility

### Features
- Enhanced robustness compared to traditional link checkers
- Respectful rate limiting and improved timeout handling
- Redirect detection and improvement suggestions
- Comprehensive JSON output with detailed link information
- Integration with GitHub Issues API for automated reporting
- Performance optimizations for large documentation sites

## [1.0.0] - 2025-10-01

### Added
- Initial stable release migrated from QuantEcon/meta repository
- Respectful rate limiting and improved timeout handling
- Full compatibility with existing workflows
- Enhanced documentation and examples
- Enhanced documentation and examples
- Comprehensive test suite with Python module testing
- GitHub Marketplace listing
- Python requirements management with requirements.txt
- Python requirements management with requirements.txt
39 changes: 37 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,26 @@ jobs:

The action includes intelligent logic to reduce false positives for legitimate sites:

### Ignore Patterns

Some hosts throttle or block datacenter IP ranges outright, so they time out from a CI runner while working perfectly for readers. The automatic detection below catches many of these, but it cannot know which hosts a particular project depends on. Use `ignore-patterns` to declare them:

```yaml
- uses: QuantEcon/action-link-checker@v1
with:
html-path: '_site'
ignore-patterns: |
https://fred\.stlouisfed\.org/.*
# blank lines and comments are ignored
linkedin\.com
```

Patterns are Python regular expressions, one per line — newline-separated rather than comma-separated so that quantifiers such as `{1,3}` survive intact. Each is matched against the full URL with `re.search`, so a bare domain works as a substring without anchoring.

Ignored URLs are never requested at all. They therefore cannot be reported as broken *or* as redirects, and they are counted separately in the report and exposed as the `ignored-count` output. An invalid pattern is logged to the job log and skipped rather than failing the run.

If a project already maintains a Sphinx `linkcheck_ignore` list, those patterns can be pasted here directly. Both take Python regular expressions, so the syntax carries over unchanged; note that Sphinx anchors its patterns at the start of the URL while this action matches anywhere in it, so a pattern here may match slightly more than the same pattern does under Sphinx.

### Bot Blocking Detection
- **Major Sites**: Automatically detects common sites that block automated requests (Netflix, Amazon, Facebook, etc.)
- **Encoding Issues**: Identifies encoding errors that often indicate bot protection
Expand Down Expand Up @@ -202,6 +222,17 @@ Common codes to consider:
- `503`: Service Unavailable (temporary outages)
- `429`: Too Many Requests (rate limiting)
- `502`: Bad Gateway (temporary server issues)
- `0`: The request never completed — it timed out, the connection failed (DNS failure, connection refused), or the server broke the response mid-stream. Silencing `0` suppresses every unreachable host, so prefer `ignore-patterns` when only specific hosts are affected. It applies to transport failures only: a malformed link such as `https://` and a redirect loop also report status `0`, and those stay reported however `silent-codes` is set, since they are the project's own to fix.

### Recurring Reports

When `create-issue` is enabled on a schedule, the action reuses the newest open issue carrying the same `issue-title` and the `broken-links` label, refreshing its body with the latest run instead of opening another issue. A weekly cron on a persistent finding therefore produces one issue, not one per week.

Close the issue once the links are fixed; if the finding recurs afterwards, a fresh issue is opened. To restore the previous behaviour of always opening a new issue, set `update-existing-issue: 'false'`.

Two things follow from matching on the title and the label. Give each workflow its own `issue-title` if a repository runs more than one link check, or they will overwrite each other's report. And leave the `broken-links` label in place — if it is removed from the tracking issue, later runs stop finding it and start opening duplicates again.

Note that GitHub does not send notifications for an edit to an issue body, so a refreshed report is quiet by design. Watch the scheduled workflow itself if you want to be told about every run.

### Performance Tuning

Expand Down Expand Up @@ -287,11 +318,13 @@ permissions:
|-------|-------------|----------|---------|
| `html-path` | Path to HTML files directory | No | `./_build/html` |
| `mode` | Scan mode: `full` or `changed` | No | `full` |
| `silent-codes` | HTTP codes to silently report | No | `403,503` |
| `silent-codes` | HTTP codes to silently report (`0` = no response) | No | `403,503` |
| `ignore-patterns` | Regex patterns for URLs to skip entirely, one per line | No | *(none)* |
| `fail-on-broken` | Fail workflow on broken links | No | `true` |
| `ai-suggestions` | Enable AI-powered suggestions | No | `true` |
| `create-issue` | Create GitHub issue for broken links | No | `false` |
| `issue-title` | Title for created issues | No | `Broken Links Found in Documentation` |
| `update-existing-issue` | Refresh the open issue with this title instead of opening a duplicate | No | `true` |
| `create-artifact` | Create workflow artifact | No | `false` |
| `artifact-name` | Name for workflow artifact | No | `link-check-report` |
| `notify` | Users to assign to created issue | No | `` |
Expand All @@ -305,9 +338,11 @@ permissions:
| `broken-links-found` | Whether broken links were found |
| `broken-link-count` | Number of broken links |
| `redirect-count` | Number of redirects found |
| `ignored-count` | Number of links skipped by `ignore-patterns` |
| `link-details` | Detailed broken link information |
| `ai-suggestions` | AI-powered improvement suggestions |
| `issue-url` | URL of created GitHub issue |
| `issue-url` | URL of the created or updated GitHub issue |
| `issue-updated` | Whether an existing issue was reused rather than a new one opened |
| `artifact-path` | Path to created artifact file |

## Best Practices
Expand Down
Loading
Loading