Skip to content

visitAllLinks: production-faithful navigation for relative and hash links; linear crawl - #133

Merged
NullVoxPopuli merged 2 commits into
mainfrom
crawler-fidelity
Jul 28, 2026
Merged

NullVoxPopuli merged 2 commits into
mainfrom
crawler-fidelity

Conversation

@NullVoxPopuli

@NullVoxPopuli NullVoxPopuli commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Three fixes, found crawling AuditBoard's 265-page documentation app (follow-up to #131):

1. Relative hrefs navigate where production would — and warn

The click navigates by element.href, which the browser resolves against the test page's URL (e.g. /tests) — not the app's current route the way a production visit would (there, the address bar is the current route). Verified in the harness: clicking ../data-schema/index.md from /schema/getting-started/index.md sent the router to /data-schema/index.md instead of /schema/data-schema/index.md.

The crawler already resolves each link's target against currentURL() at the moment it encounters it — that's what the expected value is built from. Now it also points the anchor at that resolved target before clicking, so the click exercises the real properLinks-and-router path with the URL a production visitor would produce. (Absolute hrefs are left untouched.)

Because a relative href is an authoring hazard (it only behaves when the address bar is the app's current route), the rewrite also emits a console.warn naming the page, the offending href, and the exact root-absolute path to author instead — an action item for fixing the source document.

2. Hash-insensitive navigation assertion

currentURL() never includes a #hash, so page#section links always read as failed navigations (/hash-target doesn't start with /hash-target#down). The assertion now compares without the hash.

3. Linear crawl

visited was keyed on (current page, target) pairs, so every target was re-visited from every page that links to it. Shared nav links appear on every page, making the crawl quadratic in app size — ~13k visits for the 265-page app, which times out any sane assert.timeout. It's now keyed on the target alone: one visit per reachable URL, which is what "are all links visitable" means.

Tests: the test-app gains a nested route whose template has a relative <a href="other"> (the crawl hard-errors without fix 1; the test also asserts the warning fires and names the resolved target), a page#hash link to an otherwise-unlinked route (fails without fix 2), and an assertion that no target is visited twice. 8/8 passing with the fixes; the suite errors without them.

🤖 Generated with Claude Code

Three fixes, found crawling a 265-page documentation app:

1. Relative hrefs: the click navigates by element.href, which the
   browser resolves against the TEST PAGE's url (e.g. /tests), not the
   app's current route the way a production visit would (there, the
   address bar is the current route). Verified: '../data-schema/index.md'
   from /schema/getting-started/index.md sent the router to
   /data-schema/index.md instead of /schema/data-schema/index.md. The
   crawler already resolves the target against currentURL() when it
   encounters the link — now it points the anchor at that resolved
   target before clicking, so the click exercises the real
   properLinks-and-router path with the production URL.

2. The navigation assertion compared against the link's #hash, which
   currentURL() never includes — page#hash links always read as failed
   navigations. Compare without the hash.

3. 'visited' was keyed on (current page, target) pairs, so every target
   was re-visited from every page that links it — shared nav links
   appear on every page, making the crawl quadratic in app size
   (~13k visits for a 265-page app; times out). Key on the target
   alone: one visit per reachable URL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pointing the click at the resolved target keeps the crawl faithful, but
a relative href is an authoring hazard the app author should fix:
relative hrefs resolve against the browser's URL rather than the app's
current route, so they only behave in a real full-page visit. The
warning names the page, the href, and the exact root-absolute path to
author instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@NullVoxPopuli NullVoxPopuli added the bug Something isn't working label Jul 28, 2026
@NullVoxPopuli
NullVoxPopuli merged commit 6b11637 into main Jul 28, 2026
2 checks passed
@NullVoxPopuli
NullVoxPopuli deleted the crawler-fidelity branch July 28, 2026 20:17
@github-actions github-actions Bot mentioned this pull request Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant