Skip to content

Mempool explorer: sitemap of the real pages, robots.txt Sitemap line, noindex for the rest - #79

Open
brettridenour wants to merge 5 commits into
mainfrom
site/mempool-sitemap
Open

Mempool explorer: sitemap of the real pages, robots.txt Sitemap line, noindex for the rest#79
brettridenour wants to merge 5 commits into
mainfrom
site/mempool-sitemap

Conversation

@brettridenour

@brettridenour brettridenour commented Oct 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

mempool.lazarus-xbt.xyz/sitemap.xml used to return the explorer's HTML shell (200, text/html), robots.txt named no sitemap, and no explorer page carried a robots header. This PR fixes all three. One rule: a page is in the sitemap, or it answers X-Robots-Tag: noindex. The rule and its exclusions are documented in cloudflare/mempool-site/INDEXING.md.

What's in it

  • INDEXABLE in _worker.js is the single list: 5 pages, the ones about this chain and this pool: /, /mining, /mining/pool/lazarus, /blocks/1, /graphs/mempool (/blocks and /graphs redirect to the last two). Trimmed from 16 on 19 Oct on IT's advice: the stock mempool.space pages (/about, /docs/faq, API docs, terms, privacy, trademark, calculator) and the live widgets (/clock/*, /mempool-block/0, /rbf) are out.
  • sitemap.py builds dist/sitemap-en.xml from that list and dist/sitemap.xml, an index naming it (the layout the pool site and the hub use: Search Console reads a new sitemap URL on submit, while /sitemap.xml was never fetched). Each <lastmod> is the build time, since the explorer is one Angular app. There is no <priority>: Google ignores it. Pages serves both statically (_routes.json exclude, _headers cache rule).
  • shell() adds X-Robots-Tag: noindex to every page not in the list. That covers the stock pages and live widgets above, chart pages, other pools, /<lang>/ copies, /address, /tx, /block and the redirect stubs /graphs, /docs, /docs/api, /blocks, which the app sends to listed pages. The licence files get the header from _headers (/lazarus/NOTICE, LICENSE, COPYING.md) and from the worker (/3rdpartylicenses.txt). Scripts, styles and images carry no robots header.
  • robots.txt gains Sitemap: https://mempool.lazarus-xbt.xyz/sitemap-en.xml. Its Disallows are unchanged (/address/, /tx/, /api/ and the language-prefixed forms); lifting /address/ and /tx/ is still sequenced in Explorer: robots.txt blocks /address/ and /tx/, and the pages carry no noindex #46, and this header is that issue's step 1.

Guards

  • tests/sitemap.py (Python, no network): no duplicates or off-host URLs, nothing robots.txt disallows, no <priority>, sitemap.xml indexes sitemap-en.xml, robots.txt names sitemap-en.xml, and the list must be valid JSON.
  • tests/noindex.mjs (Node, no network): listed pages have no robots header; unlisted pages, language copies, address/tx/block pages and text files get noindex; scripts don't.
  • verify-mempool.py (run by deploy.sh mempool) checks the deployed robots.txt, both sitemaps, every listed page (200 with no robots header) and sample unlisted ones (noindex). /robots.txt leaves the node-vs-Pages byte comparison, since the node's nginx has no rule for it.
  • Not yet in CI: wiring both tests into the sites job needs a workflow-scoped token. That follow-up is pending.

Deploy: the explorer is not in the merge-to-main autodeploy. Ship with LAZARUS_NODE=<node> ./cloudflare/deploy.sh mempool, then delete the temporary bridge Worker lazarus-mempool-seo, after the deploy (see the comments below and INDEXING.md). Consider a mempool_pages autodeploy step for cloudflare/mempool-site/** and node/umbrel/mempool-theme/**.

Checked: both tests pass, and fail as intended on a disallowed URL, a duplicate and a malformed list. The four redirects and their self-canonicals were confirmed in a browser on 19 Oct, and the chart pages were judged thin (labels only, no prose). py_compile and bash -n are clean.

🤖 Generated with Claude Code

@brettridenour @claude
mempool explorer: sitemap.xml, named in robots.txt and checked in CI
f3d47dc 
mempool.lazarus-xbt.xyz/sitemap.xml fell through to the app shell
(200 text/html). sitemap.py now holds the explorer's page list (the 19
URLs our SEO advisor's crawl found) and build.sh writes dist/sitemap.xml
from it after postprocess.py. Every <lastmod> is the build time: the
explorer is one Angular app, so a build is when any page last changed,
and there are no hand-kept dates to go stale. Pages serves the file
statically (_routes.json exclude, _headers cache rule) and the worker's
robots.txt names it.

tests/sitemap.py runs in the `sites` CI job: no duplicate or off-host
URLs, nothing robots.txt disallows, and robots.txt still names the
sitemap. verify-mempool.py checks the deployed robots.txt and sitemap
on the candidate alone (the node's nginx serves neither, so they leave
the byte-for-byte comparison) and that every listed page answers 200.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettridenour

brettridenour commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

Live now via a temporary bridge Worker (19 Oct 2026, ~05:42 UTC), because the explorer is not in site-autodeploy and its build needs the node's :3006.

  • Worker lazarus-mempool-seo (version 1b3f8e31) on routes mempool.lazarus-xbt.xyz/robots.txt and mempool.lazarus-xbt.xyz/sitemap.xml, zone lazarus-xbt.xyz. Everything else on the hostname still goes straight to the lazarus-mempool Pages deployment eaf4eb4c, untouched.
  • It serves exactly what this PR produces: ROBOTS from _worker.js and the output of sitemap.py (lastmod frozen at 2026-10-19T05:41:34+00:00).

@AwokenLazarus once this PR is merged and deployed with deploy.sh mempool, please delete the bridge (Workers & Pages → lazarus-mempool-seo → Delete, or wrangler delete --name lazarus-mempool-seo) so the Pages build's own sitemap and robots.txt take over. It might also be worth adding a mempool_pages step to site-autodeploy for cloudflare/mempool-site/** and node/umbrel/mempool-theme/**.

@brettridenour @claude
mempool explorer: sitemap lists the 16 real pages, everything else no…
be1247c 
…index

A browser check showed four sitemap URLs are app redirects (/graphs,
/docs, /docs/api, /blocks land on /graphs/mempool, /docs/faq,
/docs/api/rest, /blocks/1), and two were licence text files. The list
of indexable pages now lives once, as INDEXABLE in _worker.js: sitemap.py
builds sitemap.xml from it, and the worker answers every other page
(chart pages, other pools, /<lang>/ copies, address/tx/block pages, the
redirect stubs) with X-Robots-Tag: noindex. The licence files get the
header from _headers (static) and the worker (3rdpartylicenses.txt).

INDEXING.md records what is left out and why, and the temporary bridge
Worker that serves this until the Pages deploy. tests/noindex.mjs
checks the header rule against a stand-in for the assets;
verify-mempool.py checks it on the deployed candidate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettridenour brettridenour changed the title Mempool explorer: sitemap.xml, named in robots.txt Mempool explorer: sitemap of the real pages, robots.txt Sitemap line, noindex for the rest Oct 19, 2026
@brettridenour

brettridenour commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

Bridge updated (19 Oct ~06:35 UTC, version f9bc9b4f). lazarus-mempool-seo now serves the corrected 16-URL sitemap and adds X-Robots-Tag: noindex to the unlisted pages under its routes: /graphs*, /docs*, /blocks*, /mining*, /tools*, /clock*, /mempool-block*, /lazarus/NOTICE|LICENSE|COPYING.md and /3rdpartylicenses.txt. It is generated from this branch's _worker.js and sitemap.py. /<lang>/ copies and /address|/tx|/block pages only get the header once this PR is deployed.

Verified live as Googlebot: the 16 sitemap pages return 200 with no robots header, the 13 sampled unlisted URLs return noindex, and /, /api/v1/blocks/tip/height and /resources/config.js are unaffected.

@AwokenLazarus the order still matters: deploy this PR, then wrangler delete --name lazarus-mempool-seo.

brettridenour and others added 3 commits October 19, 2026 01:31
@brettridenour @claude
mempool robots.txt: drop the redundant "Allow: /"
2ab20f9 
Everything not disallowed is allowed already. Google and Bing (RFC 9309)
pick the longest matching rule, so order never mattered to them, but a
first-match parser reading "Allow: /" above the Disallows would allow
every URL, address and tx pages included. Without the line, no Allow
sits below a Disallow and every parser reads the same rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettridenour @claude
mempool robots.txt: "Allow: /" on the second line
213a40c 
Brett's call, matching the pool's robots.txt: the Allow goes right under
User-agent, above the Disallows. Google and Bing pick the longest
matching rule, so the Disallows still apply. A first-match parser would
read it as allow-all; the worker's 429 on crawler /api/address/ lookups
stays the backstop for the electrs load the Disallows exist for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettridenour @claude
mempool sitemap: 5 pool pages in /sitemap-en.xml, /sitemap.xml indexe…
11aa42f 
…s it

On IT's advice the sitemap keeps only the pages about this chain and this
pool: /, /mining, /mining/pool/lazarus, /blocks/1 and /graphs/mempool (the
pages /blocks and /graphs redirect to). The stock mempool.space pages
(/about, /docs/faq, API docs, terms, privacy, trademark, calculator) and the
live widgets (/clock/*, /mempool-block/0, /rbf) leave INDEXABLE and so get
noindex. <priority> is gone; Google ignores it.

The pages are listed in /sitemap-en.xml, which robots.txt names, and
/sitemap.xml is an index of it, as on the pool site and the hub: Search
Console reads a new sitemap URL on submit, while /sitemap.xml was never
fetched. robots.txt is otherwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettridenour

brettridenour commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

19 Oct, 11aa42f: sitemap trimmed to 5 pool pages and moved to /sitemap-en.xml.

  • On IT's advice the sitemap now lists only /, /mining, /mining/pool/lazarus, /blocks/1 and /graphs/mempool. Under this PR's rule (in the sitemap or noindex), the stock mempool.space pages and the live widgets now answer X-Robots-Tag: noindex. <priority> is gone.
  • The page list is served at /sitemap-en.xml, which robots.txt names. /sitemap.xml is an index of it, matching the pool site and the hub. Search Console read the hub's new sitemap-en.xml on submit after never fetching its /sitemap.xml.
  • robots.txt is otherwise unchanged.

Bridge Worker lazarus-mempool-seo updated to version 912a516a so this is live now. It has 19 routes: the previous 13, plus /sitemap-en.xml, /about*, /rbf*, /terms-of-service*, /privacy-policy* and /trademark-policy*, so those pages get noindex before this PR deploys. The plan is unchanged: deploy this PR with deploy.sh mempool, then wrangler delete --name lazarus-mempool-seo.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@brettridenour