Repository navigation
Pool robots.txt opens /api/*; hub sitemap moves to /sitemap-en.xml - #81
Merged
Merged
Conversation
Pool: the pages fill their figures, the hardware catalog and the blocks table from /api/*, and Google does not render a fetch to a blocked URL, so Disallow: /api/ left /hardware and /blocks empty for Googlebot. Only the per-address and per-block lookups (/api/miner/, /api/solo/, /api/found/) stay closed. The indexing check now fails if robots.txt blocks an endpoint the pages render from. Hub: Search Console never fetched https://lazarus-xbt.xyz/sitemap.xml ("Couldn't fetch") while the pool's new sitemap-en.xml / sitemap-zh.xml were read on submit. The URL list is now /sitemap-en.xml, robots.txt points at it, and /sitemap.xml is an index of that one file, as on the pool site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pool site: robots.txt
Disallow: /api/blocked the JSON the pages are built from. Google does not render a fetch to a blocked URL, and the first fetch inpool.js(/api/pool?h=0) has no fallback, so for Googlebot the whole refresh stopped:/hardwareshowed no machines,/blockshad an empty table, and every stat read "—"./hardwareis "Crawled – currently not indexed" in Search Console (crawled 28 Sep, under this rule).New rules:
The three that stay closed are the per-address and per-block lookups: endless URLs, slow to build, and only fetched after a user looks something up. Nothing under
/api/is secret. The API needs no login, is open to any site (CORS*), and is documented on/apiand in/api/openapi.json.cloudflare/pool-site/tests/indexing.pynow fails if robots.txt blocks any endpoint the pages render from.Hub: sitemap at /sitemap-en.xml
Search Console has never downloaded
https://lazarus-xbt.xyz/sitemap.xml("Couldn't fetch", no download date), even though the file returns 200application/xml, including when requested as Googlebot. The pool's newsitemap-en.xml/sitemap-zh.xmlwere read the moment they were submitted.public/sitemap-en.xmlandrobots.txtpoints at it./sitemap.xmlbecomes an index listing that one file, as on the pool site.sitemap-lastmod.mjs,indexnow.mjsandtest-indexing.mjsnow readsitemap-en.xml, and the test also checks the index and the robots line.Checks run locally
test-indexing.mjs,sitemap-lastmod.mjs --check,test-naughtylist.mjsindexing.py,lastmod.py,language.mjspool/test_*.pyfilesAll pass.
🤖 Generated with Claude Code