[zordhalo]

Pages that opt out of indexing were scored like any other. On aeo.advancelabs.dev, /login (noindex, nofollow, not in the sitemap) was reached through the "Sign in" link and produced 4 failures including a false critical.

Behaviour

1. Excluded from scoring. Pages with noindex (meta robots, or X-Robots-Tag, including none; header agent prefixes count only for googlebot/bingbot) are removed from ctx.pages and, when structuredData lines up one-to-one with pages, from ctx.structuredData. ctx.crawl is untouched. They stay in pagesCrawled and appear in the new AuditReport.excludedPages as {url, reason: 'noindex'}, so nothing disappears silently. If every page is noindex, nothing is excluded.
2. nofollow honoured (crawler). A page with nofollow/none (meta or header) is fetched and recorded but its links are not followed; <a rel=nofollow> links are skipped. noindex alone still follows links.
3. Indexability rules keep seeing noindex pages via the new Rule.includesNoindexPages.

Per-rule decisions

• tech.indexable: full context, with a narrower definition of defect. A noindex page is flagged if it is the root, if the sitemap lists it (contradiction), or if there is no sitemap to show intent. A noindex page that a non-empty sitemap leaves out is treated as a deliberate opt-out (login, checkout). Without this, /login would still raise the critical.
• tech.canonical-present, tech.canonical-resolves: full context (canonical checks are inspection rules; unchanged behaviour).
• tech.sitemap-covers-pages: already skipped noindex pages; unchanged.
• Site and crawl-level rules (robots, sitemap, llms.txt, https, redirects, broken pages, hreflang targets over crawl.pages): read ctx.crawl, which is not filtered; unchanged.
• Every other per-page rule (titles, descriptions, headings, thin content, alt text, internal linking, viewport, OG, Twitter, structured data, all AEO and E-E-A-T rules): scored without noindex pages.

Tests: scoring noindex.test.ts (12) and crawler links/crawl tests. Changesets: scoring minor, types minor, crawler patch.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
aeo-toolkit-docs	Ready	Preview	Nov 8, 2026 2:44am UTC