Make indexability one predicate across hreflang, robots meta and sitemaps - #39
Merged
Merged
Conversation
…maps Three things had to agree about a non-home market and didn't: the hreflang cluster, the robots meta tag, and what the sitemap lists. Only the first existed, so turning NEXT_PUBLIC_MULTI_REGION_LIVE on would have put .in and .ai into the cluster while they still serve byte-identical Canadian content under a dark router — an orphaned duplicate, not a market. marketIndexable() in lib/region/seo.ts is now the single predicate. A market is indexable only once regionRoutingLive() gives it its own content, and .in additionally waits on its own launch flag. hreflang membership, the noindex in app/layout.tsx, and sitemap contents all derive from it, so the cluster-implies-indexable invariant holds by construction and self-heals when routing goes live rather than leaving a hack to remember. Also extracts lib/region/crawl-paths.ts. The crawl-metadata exemption list was written twice and had already drifted: geo-route.ts exempted /robots.txt, /sitemap.xml and /sitemaps/, waitlist.ts only the first two. Invisible while routing is off; at India launch .in/sitemap.xml would have resolved, listed six children, and 307'd every one to /waitlist — unsubmittable in Search Console with nothing in our logs. A redirected sitemap is worse than a missing one, so this list must not live in two places. organizationSchema keeps main's per-market ORG_ADDRESS (the .in node is the Bangalore entity, which is better than the addressCountry-only fallback this branch had) but makes areaServed follow it. main asserted the GTA city list unconditionally, so the .in node claimed a Bangalore address AND a Toronto service area — a self-contradicting entity that tells Google the India domain is really a Toronto business. areaServed is now keyed off the resolved addressCountry so a market added to ORG_ADDRESS later cannot silently inherit the wrong service area. tests/seo/organization-schema.test.ts covers this; there was no coverage of organizationSchema at all before. Adds SEO-AUDIT.md and the review brief that calls this work partial by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prerequisite for the ARC0 runbook §F flag flips. This must deploy before
NEXT_PUBLIC_REGION_ROUTING_LIVEis turned on.Why now
Flipping R against current
mainwakes the India waitlist gate, which exempts only/robots.txtand/sitemap.xml— not/sitemaps/. On.inthe sitemap index would resolve, list six children, and 307 every one to/waitlist: unsubmittable in Search Console, recorded by Google as a persistent fetch failure that takes weeks of re-crawls to clear, with nothing in our logs.lib/region/crawl-paths.tsmakes that exemption list single-source.What changed
marketIndexable()is now the one predicate. hreflang membership, thenoindexinapp/layout.tsx, and sitemap contents all derive from it, so "in the cluster ⟺ indexable" holds by construction. A market is indexable only onceregionRoutingLive()gives it its own content;.inadditionally waits on its own launch flag. Self-heals when routing goes live — no temporary hack to remember.lib/region/crawl-paths.ts— extracts the crawl-metadata exemption list that had drifted betweengeo-route.tsandwaitlist.ts.organizationSchema— keepsmain's per-marketORG_ADDRESS(the.innode is the Bangalore entity) but makesareaServedfollow it.mainasserted the GTA city list unconditionally, so the.innode claimed a Bangalore address and a Toronto service area — a self-contradicting entity implying the India domain is really a Toronto business. Now keyed off the resolvedaddressCountry, so a market added toORG_ADDRESSlater can't silently inherit the wrong service area.Measured problem this addresses
.ca/.ai/.inhomepages currently differ by ~110 bytes out of ~330,000 with the hostname normalised out — the same page. All three say "Toronto"/"GTA", all three self-canonicalize, none carriesnoindex.Tests
tests/seo/organization-schema.test.tsis new —organizationSchemahad zero coverage before.Gates: typecheck clean · 1450 tests pass · lint 0 errors · knip clean · migrations OK.
🤖 Generated with Claude Code