Repository navigation
Trim the crawl surface so bots cost less to serve - #86
Merged
Merged
Conversation
A station page is rendered per request, because the CSP nonce rules out caching it, and robots.txt welcomes every named AI crawler as well as the search ones. Listing 1615 stations therefore bought 1615 renders on every pass any of them made. The list is ranked by clicks and votes, so the cap drops to 600 and sheds the tail that was drawing least, and station entries ask for a monthly rather than weekly recrawl.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Part of getting back inside the Vercel free tier. Station pages cannot be CDN cached while the CSP nonce forces per-request rendering, so every crawler visit to one is a full render. The sitemap listed 1615 station pages and
robots.txtdeliberately names 21 AI crawlers as welcome on top of the search engines, so that count was multiplied by every bot making a pass.What
MAX_SITEMAP_STATIONSdrops from 2000 to 600. The selection is already ranked by clicks plus votes, so this keeps the stations people actually search for and sheds the tail that was drawing least.monthlyrecrawl rather thanweekly. A station's name, country and tags rarely move, so a weekly sweep was paying for a fresh render four times as often as the page earned.Browse pages (country, tag, language, combos) are untouched: there are far fewer of them and they are the ones that rank.
Trade-off
This is a deliberate reduction in indexed long tail. It is the lever available while the strict CSP stays, since the pages themselves cannot be cached. If the nonce is ever dropped in favour of ISR, the cap can go back up.
Verification
npm run lint,npm run typecheck,npm test(465 passing),npx prettier --check .andnpm run buildall pass locally. The sitemap tests are written against an explicit cap argument, so they are unaffected by the constant.