Skip to content

Pool robots.txt opens /api/*; hub sitemap moves to /sitemap-en.xml #81

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brettridenour merged 1 commit into from
Oct 19, 2026
Merged

Pool robots.txt opens /api/*; hub sitemap moves to /sitemap-en.xml #81

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cloudflare/map-site/public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://lazarus-xbt.xyz/sitemap.xml
Sitemap: https://lazarus-xbt.xyz/sitemap-en.xml
13 changes: 13 additions & 0 deletions cloudflare/map-site/public/sitemap-en.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://lazarus-xbt.xyz/</loc><lastmod>2026-10-12</lastmod><priority>1.0</priority></url>
<url><loc>https://lazarus-xbt.xyz/learn/</loc><lastmod>2026-10-12</lastmod><priority>0.9</priority></url>
<url><loc>https://lazarus-xbt.xyz/ecosystem/</loc><lastmod>2026-10-12</lastmod><priority>0.8</priority></url>
<url><loc>https://lazarus-xbt.xyz/key/</loc><lastmod>2026-10-12</lastmod><priority>0.8</priority></url>
<url><loc>https://lazarus-xbt.xyz/faq/</loc><lastmod>2026-10-12</lastmod><priority>0.7</priority></url>
<url><loc>https://lazarus-xbt.xyz/map/</loc><lastmod>2026-10-06</lastmod><priority>0.6</priority></url>
<url><loc>https://lazarus-xbt.xyz/naughtylist/</loc><lastmod>2026-10-12</lastmod><priority>0.6</priority></url>
<url><loc>https://lazarus-xbt.xyz/map/table/</loc><lastmod>2026-10-11</lastmod><priority>0.4</priority></url>
<url><loc>https://lazarus-xbt.xyz/stream/</loc><lastmod>2026-10-18</lastmod><priority>0.8</priority></url>
<url><loc>https://lazarus-xbt.xyz/blake2b/</loc><lastmod>2026-10-12</lastmod><priority>0.8</priority></url>
</urlset>
15 changes: 3 additions & 12 deletions cloudflare/map-site/public/sitemap.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://lazarus-xbt.xyz/</loc><lastmod>2026-10-12</lastmod><priority>1.0</priority></url>
<url><loc>https://lazarus-xbt.xyz/learn/</loc><lastmod>2026-10-12</lastmod><priority>0.9</priority></url>
<url><loc>https://lazarus-xbt.xyz/ecosystem/</loc><lastmod>2026-10-12</lastmod><priority>0.8</priority></url>
<url><loc>https://lazarus-xbt.xyz/key/</loc><lastmod>2026-10-12</lastmod><priority>0.8</priority></url>
<url><loc>https://lazarus-xbt.xyz/faq/</loc><lastmod>2026-10-12</lastmod><priority>0.7</priority></url>
<url><loc>https://lazarus-xbt.xyz/map/</loc><lastmod>2026-10-06</lastmod><priority>0.6</priority></url>
<url><loc>https://lazarus-xbt.xyz/naughtylist/</loc><lastmod>2026-10-12</lastmod><priority>0.6</priority></url>
<url><loc>https://lazarus-xbt.xyz/map/table/</loc><lastmod>2026-10-11</lastmod><priority>0.4</priority></url>
<url><loc>https://lazarus-xbt.xyz/stream/</loc><lastmod>2026-10-18</lastmod><priority>0.8</priority></url>
<url><loc>https://lazarus-xbt.xyz/blake2b/</loc><lastmod>2026-10-12</lastmod><priority>0.8</priority></url>
</urlset>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://lazarus-xbt.xyz/sitemap-en.xml</loc></sitemap>
</sitemapindex>
4 changes: 2 additions & 2 deletions cloudflare/map-site/scripts/indexnow.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
// Google does not use IndexNow; it reads the sitemap.
//
// node scripts/indexnow.mjs dry run: print what would be sent
// node scripts/indexnow.mjs --live submit every URL in public/sitemap.xml
// node scripts/indexnow.mjs --live submit every URL in public/sitemap-en.xml
// node scripts/indexnow.mjs --live /blake2b/ /learn/ submit only these paths
//
// The key is the public/<key>.txt file served at the site root; it is public by design.
Expand All @@ -21,7 +21,7 @@ const key = readFileSync(join(pub, keyFile), "utf8").trim();
const args = process.argv.slice(2);
const live = args.includes("--live");
const paths = args.filter((a) => a.startsWith("/"));
const fromSitemap = [...readFileSync(join(pub, "sitemap.xml"), "utf8").matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
const fromSitemap = [...readFileSync(join(pub, "sitemap-en.xml"), "utf8").matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
const urlList = paths.length ? paths.map((p) => `https://${HOST}${p}`) : fromSitemap;

const body = { host: HOST, key, keyLocation: `https://${HOST}/${keyFile}`, urlList };
Expand Down
12 changes: 6 additions & 6 deletions cloudflare/map-site/scripts/sitemap-lastmod.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
// Set each <lastmod> in public/sitemap.xml to the date of the last commit that changed that
// Set each <lastmod> in public/sitemap-en.xml to the date of the last commit that changed that
// page's HTML (or, for Worker-rendered pages, its source files). Run it in the commit that changes a page (npm run sitemap), so the sitemap never
// says "today" for pages that did not change: search engines only use lastmod while it stays true.
//
// node scripts/sitemap-lastmod.mjs rewrite public/sitemap.xml
// node scripts/sitemap-lastmod.mjs rewrite public/sitemap-en.xml
// node scripts/sitemap-lastmod.mjs --check exit 1 and list the URLs whose date is stale
import { execFileSync } from "node:child_process";
import { readFileSync, writeFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";

const here = dirname(dirname(fileURLToPath(import.meta.url)));
const sitemapPath = join(here, "public", "sitemap.xml");
const sitemapPath = join(here, "public", "sitemap-en.xml");
const check = process.argv.includes("--check");

// Pages the Worker renders have no HTML file; their words live in these sources.
Expand Down Expand Up @@ -46,11 +46,11 @@ const out = xml.replace(/<url><loc>([^<]+)<\/loc><lastmod>([^<]*)<\/lastmod>/g,

if (check) {
if (stale.length) {
console.error(`sitemap.xml lastmod is out of date (run: npm run sitemap):\n ${stale.join("\n ")}`);
console.error(`sitemap-en.xml lastmod is out of date (run: npm run sitemap):\n ${stale.join("\n ")}`);
process.exit(1);
}
console.log("sitemap.xml lastmod matches git");
console.log("sitemap-en.xml lastmod matches git");
} else {
writeFileSync(sitemapPath, out);
console.log(stale.length ? `updated:\n ${stale.join("\n ")}` : "sitemap.xml already current");
console.log(stale.length ? `updated:\n ${stale.join("\n ")}` : "sitemap-en.xml already current");
}
5 changes: 4 additions & 1 deletion cloudflare/map-site/scripts/test-indexing.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,10 @@ import assert from 'node:assert/strict';
import {readFileSync} from 'node:fs';
import worker from '../src/worker.js';
const publicDir = new URL('../public/', import.meta.url);
const xml = readFileSync(new URL('sitemap.xml', publicDir), 'utf8');
const index = readFileSync(new URL('sitemap.xml', publicDir), 'utf8');
assert.deepEqual([...index.matchAll(/<sitemap><loc>([^<]+)<\/loc>/g)].map(m=>m[1]), ['https://lazarus-xbt.xyz/sitemap-en.xml']);
assert.ok(readFileSync(new URL('robots.txt',publicDir),'utf8').includes('Sitemap: https://lazarus-xbt.xyz/sitemap-en.xml\n'));
const xml = readFileSync(new URL('sitemap-en.xml', publicDir), 'utf8');
const urls = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m=>m[1]);
assert.equal(urls.length, 10);
assert.equal(new Set(urls).size, 10);
Expand Down
7 changes: 6 additions & 1 deletion cloudflare/pool-site/tests/indexing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,12 @@ def handle_starttag(self, tag, attrs):
count += 1
assert count == 30, count # 15 documents in two languages
assert not (out/'static/index.html').exists(), 'raw template is publicly exported'
assert 'Disallow: /\n' not in (out/'robots.txt').read_text()
robots = (out/'robots.txt').read_text()
assert 'Disallow: /\n' not in robots
# The pages render from these; Google does not run a fetch to a blocked URL.
disallowed = [l.split(':', 1)[1].strip() for l in robots.splitlines() if l.startswith('Disallow:')]
for api in ['/api/pool', '/api/coinbaser', '/api/price', '/api/solo', '/api/miners', '/api/hardware', '/api/rentals', '/api/blocks', '/api/payouts', '/api/history']:
assert not any(api.startswith(d) for d in disallowed), (api, disallowed)
for route in ['/static/', '/static/index.html', '/static/index']:
assert f'{route} / 301' in (out/'_redirects').read_text()
assert (out/'404.html').exists(), 'Pages would enable SPA fallback without 404.html'
Expand Down
11 changes: 6 additions & 5 deletions pool/server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7184,11 +7184,12 @@ def robots_txt():
"User-agent: *\n"
"Allow: /\n"
"\n"
"# The API reference is a page a developer should be able to find; the JSON endpoints under\n"
"# it have nothing to read and would only burn crawl budget. Longest match wins, so /api\n"
"# stays allowed while everything below it is blocked.\n"
"Allow: /api\n"
"Disallow: /api/\n"
"# The pages fill their figures, the hardware catalog and the blocks table from /api/*, and\n"
"# Google does not render a fetch to a blocked URL, so those endpoints stay open. Only the\n"
"# lookups are closed: one URL per address or block, without end, and each slow to build.\n"
"Disallow: /api/miner/\n"
"Disallow: /api/solo/\n"
"Disallow: /api/found/\n"
"\n"
f"Sitemap: {_PUBLIC_SITE}/sitemap.xml\n"
)
Expand Down
Loading