Site Audit: crawler has no query-parameter handling (duplicate false positives, budget, no sprawl finding)

[Nordalux]

Summary

The site-audit crawler treats every distinct query string as a distinct page. There is no parameter handling anywhere in the crawl path — not for deduplication, not for budget, and no reporting of parameter sprawl as a finding.

This has two separate consequences: false-positive duplicate findings, and crawl budget spent on variants of the same page. I'd like to align on which of these (if any) you want addressed before writing code, per CONTRIBUTING.

Current behaviour

normalizeUrl resolves, strips the fragment, lowercases the host and sorts query params for the dedup key — it never drops one:

// src/server/lib/audit/url-utils.ts
parsed.hash = "";
parsed.searchParams.sort();
parsed.hostname = parsed.hostname.toLowerCase();

The queue predicate filters on origin, SSRF policy, robots.txt and the visited/queued sets, and nothing else:

// src/server/workflows/siteAuditWorkflowCrawl.ts
function shouldQueueCrawlLink(link, origin, robots, visited, queued): boolean {
  return (
    isSameOrigin(link, origin) &&
    isCrawlableUrl(link) &&
    robots.isAllowed(link) &&
    !visited.has(link) &&
    !queued.has(link)
  );
}

url-policy.ts is purely SSRF protection and does not look at the query string.

Credit where due: robots.txt is honoured via robots-parser, and that already mitigates a good share of faceted-navigation sprawl on platforms that ship sensible defaults. Checking a live Shopify store, the platform default disallows /collections/*sort_by* and */collections/*filter*&*filter*. So this is not the runaway-crawl scenario it might first look like.

What robots.txt does not cover, on any platform:

• Tracking parameters — utm_*, gclid, fbclid, msclkid. Nobody disallows these, and they change nothing about the page.
• Single-facet and pagination URLs where the platform default only blocks multi-facet combinations.
• Sites with no parameter rules at all, which is the common case outside the big platforms — ?replytocom=, ?orderby=, ?PHPSESSID=, calendar ?date= and so on.

Consequence 1 — false-positive duplicate findings

This is the part I'd argue is a straight bug rather than a preference.

/page and /page?utm_source=newsletter are the same page. The crawler enqueues both, stores both, and the cross-page checks in multipage.ts then report duplicate-title, duplicate-meta-description and duplicate-content against them. An audit tool reporting a page as a duplicate of itself is a wrong finding, and users can't tell it apart from a real one.

Consequence 2 — crawl budget

Every variant consumes a page from maxPagesPerAudit and a unit from the capacity budget. On a large site the crawl can hit its ceiling on variants before reaching pages that were never visited.

Consequence 3 — the finding that isn't reported

Parameter sprawl and faceted-navigation traps are a standard technical-SEO finding; comparable tools surface them. AUDIT_ISSUE_TYPES currently has no equivalent, even though detectUrlTemplate in url-utils.ts already collapses paths to templates and would give the grouping such a check needs.

Possible directions

Deliberately not picking one — these have quite different footprints and the call is yours.

A. Drop known tracking parameters from the dedup key. Narrow, no configuration, addresses consequence 1. ?utm_source= variants collapse onto the canonical URL instead of being crawled and reported as duplicates. Risk is low but non-zero: a site could theoretically serve different content per tracking parameter.

B. Report parameter sprawl as an audit issue. Addresses consequence 3 and arguably fits the product best — surface the trap rather than hide it. Would touch audit-issues.ts / types.ts, which #122 is currently working in, so it would want to wait for that to land.

C. Cap parameter variants per path template during the crawl. Addresses consequence 2, the largest behavioural change. Needs decisions: what limit, hard-coded or a launch-form option, and whether skipped variants get surfaced so the cap isn't silent.

A and B are additive and could ship independently. C is the one I'd least want to guess at.

Happy to implement whichever you'd take, or to leave it if you consider the current behaviour intentional.