Limit bootstrap secret downloads and clarify Cloudflare guidance

docs/cloudflare-origin-lockdown.md

@@ -49,7 +49,9 @@ The certificate Cloudflare presents in this setup (Cloudflare calls it global Au
 - It does not prove the request passed through the `zenon.info` zone. Another Cloudflare customer who points a proxied hostname of their own at this origin IP and uses Cloudflare features that override the SNI and `Host` header sent to the origin would present the same certificate. The zone-level protections configured in this account (the WAF custom rule, rate limiting, Bot Fight Mode) do not apply to such traffic.
 - The lockdown therefore removes direct-to-origin traffic; it does not replace the app's own authentication and abuse controls, which must stay in place.
 
-If account-specific origin authentication is ever required, Cloudflare's zone-level or per-hostname Authenticated Origin Pulls with a customer-uploaded certificate provides it: the origin then verifies a certificate that only this account holds. That needs a private CA, an API upload of the certificate, and a rotation procedure of its own, and is not part of this plan.
+The app separately limits authenticated secret downloads to 60 requests per node token per 15 minutes and returns `429` with `Retry-After` when the limit is reached. This in-memory bound does not protect a secret after its token is disclosed; rotate compromised node credentials and re-run bootstrap.
+
+If account-specific origin authentication is ever required, Cloudflare's zone-level or per-hostname Authenticated Origin Pulls with a customer-uploaded certificate provides it: the origin then verifies a certificate that only this account holds. That needs a private CA, certificate upload through the dashboard or API, and a rotation procedure of its own, and is not part of this plan.
 
 ## Procedure
 
@@ -59,7 +61,7 @@ Do the steps in this order. Steps 1 to 3 change nothing visible; step 4 turns en
 
 Zone `zenon.info` → **SSL/TLS** → **Origin Server** → **Authenticated Origin Pulls** → On.
 
-The dashboard toggle is zone-wide. That is fine: other hostnames in the zone receive the client certificate too, and their Traefik routers ignore it because they carry no `clientAuth` option. The zone-level and per-hostname modes with a customer-uploaded certificate exist only through the API; they are needed only for account-specific authentication (see [What This Guarantees](#what-this-guarantees)).
+The dashboard toggle is zone-wide. That is fine: other hostnames in the zone receive the client certificate too, and their Traefik routers ignore it because they carry no `clientAuth` option. The zone-level and per-hostname modes with a customer-uploaded certificate can be configured in the dashboard or through the API; they are needed only for account-specific authentication (see [What This Guarantees](#what-this-guarantees)).
 
 ### 2. Put the Cloudflare CA on the Coolify host
 
@@ -152,4 +154,4 @@ Coolify's Let's Encrypt resolver uses the HTTP-01 challenge on port 80, which is
 
 ## Follow-Up: Real Client IPs
 
-With the orange cloud on, Traefik's peer address is a Cloudflare edge, and the app's `TRUST_PROXY=uniquelocal` setting trusts only the proxy hop, so the login rate limiter (`AttemptLimiter`) currently keys on Cloudflare edge addresses. Once Authenticated Origin Pulls guarantees that every `testnet.zenon.info` connection came from Cloudflare's network, the `CF-Connecting-IP` header can be trusted for that router, because Cloudflare's edge sets that header itself on every proxied request, whichever zone the request passed through. That is the only downstream trust this plan places in the lockdown; nothing in the app should assume a request also passed this zone's WAF rules. The change is app-side (read the header only when the immediate peer is trusted and the header is present) and should be made after step 5 passes, not before.
+With the orange cloud on, Traefik's peer address is a Cloudflare edge, and the app's `TRUST_PROXY=uniquelocal` setting trusts only the proxy hop, so the login rate limiter (`AttemptLimiter`) currently keys on Cloudflare edge addresses. After step 5 confirms the intended router binding, a future app-side change may use `CF-Connecting-IP` for client-IP attribution and rate limiting on ordinary proxied requests, but only when the immediate proxy path is trusted. It must not use the header for identity or authentication: Cloudflare documents Worker subrequests where the value may be altered or replaced with a Worker address. Global AOP authenticates the Cloudflare edge, not the end user or this zone's WAF processing. This follow-up is not implemented by the procedure above.

src/server/bootstrap-secret-http.test.ts

@@ -0,0 +1,140 @@
+import assert from "node:assert/strict";
+import { spawn } from "node:child_process";
+import { createCipheriv, createHash } from "node:crypto";
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { createServer } from "node:net";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { once } from "node:events";
+import { fileURLToPath } from "node:url";
+import { it } from "node:test";
+import { BOOTSTRAP_SECRET_MAX_DOWNLOADS } from "./bootstrap-secret-limit.js";
+
+const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
+const fixtureAppSecret = "fixture-app-secret-for-tests";
+
+async function unusedPort(): Promise<number> {
+  const server = createServer();
+  await new Promise<void>((resolve, reject) => {
+    server.once("error", reject);
+    server.listen(0, "127.0.0.1", resolve);
+  });
+  const address = server.address();
+  if (!address || typeof address === "string") throw new Error("Expected a TCP port");
+  await new Promise<void>((resolve, reject) => server.close((error) => error ? reject(error) : resolve()));
+  return address.port;
+}
+
+function pillar(id: string, token: string) {
+  const wallet = { address: "test-address", keyFile: { fixture: true }, passwordCipher: "test-cipher" };
+  return {
+    id,
+    userId: id,
+    pillarName: id,
+    pillarWallet: wallet,
+    rewardWallet: wallet,
+    producerWallet: wallet,
+    producerIndex: 0,
+    statusTokenHash: createHash("sha256").update(token).digest("hex"),
+    statusTokenCipher: "test-cipher",
+    createdAt: "2026-01-01T00:00:00.000Z"
+  };
+}
+
+function seed(id: string, token: string) {
+  const nonce = Buffer.alloc(12, 1);
+  const key = createHash("sha256").update(fixtureAppSecret).digest();
+  const cipher = createCipheriv("aes-256-gcm", key, nonce);
+  const encrypted = Buffer.concat([cipher.update("fixture-network-key", "utf8"), cipher.final()]);
+  return {
+    id,
+    userId: id,
+    nodeName: id,
+    publicIp: "198.51.100.1",
+    p2pPort: 35995,
+    publicKey: "fixture-public-key",
+    enode: "fixture-enode",
+    multiaddr: "/ip4/198.51.100.1/tcp/35995",
+    networkPrivateKeyCipher: `${nonce.toString("hex")}:${cipher.getAuthTag().toString("hex")}:${encrypted.toString("hex")}`,
+    statusTokenHash: createHash("sha256").update(token).digest("hex"),
+    statusTokenCipher: "test-cipher",
+    createdAt: "2026-01-01T00:00:00.000Z"
+  };
+}
+
+it("bounds authenticated secret routes by node token without charging invalid or wrong-type requests", async () => {
+  const dataDir = await mkdtemp(path.join(tmpdir(), "testnet-secret-http-"));
+  try {
+    const token = "fixture-pillar-token";
+    const otherToken = "fixture-other-token";
+    const seedToken = "fixture-seed-token";
+    const state = {
+      users: [],
+      sessions: [],
+      pillars: [pillar("pillar-one", token), pillar("pillar-two", otherToken)],
+      seedNodes: [seed("seed-one", seedToken)],
+      settings: {
+        sporkAddress: "test-spork-address",
+        sporkWallet: { address: "test-spork-address", keyFile: {}, passwordCipher: "test-cipher" }
+      }
+    };
+    await writeFile(path.join(dataDir, "app-state.json"), JSON.stringify(state), { mode: 0o600 });
+
+    const port = await unusedPort();
+    const origin = `http://127.0.0.1:${port}`;
+    const child = spawn(process.execPath, ["--import", "tsx", "src/server/index.ts"], {
+      cwd: repoRoot,
+      env: { ...process.env, APP_SECRET: fixtureAppSecret, DATA_DIR: dataDir, PORT: String(port), NODE_ENV: "test" },
+      stdio: ["ignore", "pipe", "pipe"]
+    });
+    let output = "";
+    child.stdout.on("data", (chunk: Buffer) => { output += chunk.toString(); });
+    child.stderr.on("data", (chunk: Buffer) => { output += chunk.toString(); });
+
+    const get = (route: string, bearer?: string) => fetch(`${origin}${route}`, {
+      headers: bearer ? { Authorization: `Bearer ${bearer}` } : {}
+    });
+
+    try {
+      let ready = false;
+      for (let attempt = 0; attempt < 100; attempt += 1) {
+        if (child.exitCode !== null) throw new Error(`Server exited before readiness: ${output}`);
+        try {
+          const health = await get("/api/health");
+          if (health.ok) { ready = true; break; }
+        } catch { /* Wait for startup. */ }
+        await new Promise((resolve) => setTimeout(resolve, 50));
+      }
+      assert.ok(ready, `Server did not become ready: ${output}`);
+
+      assert.equal((await get("/api/bootstrap/producer.json")).status, 401);
+      assert.equal((await get("/api/bootstrap/producer.json", "invalid-token")).status, 401);
+      assert.equal((await get("/api/bootstrap/network-private-key", token)).status, 404);
+
+      for (let index = 0; index < BOOTSTRAP_SECRET_MAX_DOWNLOADS; index += 1) {
+        assert.equal((await get("/api/bootstrap/producer.json", token)).status, 200);
+      }
+      const blocked = await get("/api/bootstrap/producer-password.txt", token);
+      assert.equal(blocked.status, 429);
+      assert.ok(Number(blocked.headers.get("retry-after")) > 0);
+      assert.match(blocked.headers.get("cache-control") ?? "", /no-store/);
+      assert.equal((await get("/api/bootstrap/producer.json", otherToken)).status, 200);
+
+      assert.equal((await get("/api/bootstrap/producer.json", seedToken)).status, 404);
+      for (let index = 0; index < BOOTSTRAP_SECRET_MAX_DOWNLOADS; index += 1) {
+        assert.equal((await get("/api/bootstrap/network-private-key", seedToken)).status, 200);
+      }
+      const blockedSeed = await get("/api/bootstrap/network-private-key", seedToken);
+      assert.equal(blockedSeed.status, 429);
+      assert.ok(Number(blockedSeed.headers.get("retry-after")) > 0);
+    } finally {
+      if (child.exitCode === null) {
+        const exited = once(child, "exit");
+        child.kill("SIGTERM");
+        await exited;
+      }
+    }
+  } finally {
+    await rm(dataDir, { recursive: true, force: true });
+  }
+});

src/server/bootstrap-secret-limit.test.ts

@@ -0,0 +1,19 @@
+import assert from "node:assert/strict";
+import { describe, it } from "node:test";
+import {
+  BOOTSTRAP_SECRET_MAX_DOWNLOADS,
+  BOOTSTRAP_SECRET_WINDOW_MS,
+  BootstrapSecretDownloadLimiter
+} from "./bootstrap-secret-limit.js";
+
+describe("BootstrapSecretDownloadLimiter", () => {
+  it("admits the bounded per-token quota and returns a retry delay until the window expires", () => {
+    const limiter = new BootstrapSecretDownloadLimiter();
+    for (let index = 0; index < BOOTSTRAP_SECRET_MAX_DOWNLOADS; index += 1) {
+      assert.equal(limiter.admit("pillar-token-hash", 0), 0);
+    }
+    assert.equal(limiter.admit("pillar-token-hash", 0), BOOTSTRAP_SECRET_WINDOW_MS);
+    assert.equal(limiter.admit("another-token-hash", 0), 0);
+    assert.equal(limiter.admit("pillar-token-hash", BOOTSTRAP_SECRET_WINDOW_MS), 0);
+  });
+});

src/server/bootstrap-secret-limit.ts

@@ -0,0 +1,17 @@
+import { AttemptLimiter } from "./rate-limit.js";
+
+// The agent runs once a minute and a pillar fetches at most two secrets per run. This leaves
+// headroom for retries while bounding repeated downloads by a holder of a valid node token.
+export const BOOTSTRAP_SECRET_MAX_DOWNLOADS = 60;
+export const BOOTSTRAP_SECRET_WINDOW_MS = 15 * 60_000;
+
+export class BootstrapSecretDownloadLimiter {
+  private readonly limiter = new AttemptLimiter({
+    maxAttempts: BOOTSTRAP_SECRET_MAX_DOWNLOADS,
+    windowMs: BOOTSTRAP_SECRET_WINDOW_MS
+  });
+
+  admit(tokenHash: string, now = Date.now()): number {
+    return this.limiter.admit(tokenHash, now);
+  }
+}

src/server/index.ts

@@ -13,6 +13,7 @@ import { bootstrapInstallScript } from "./bootstrap-script.js";
 import { resolveGitRef } from "./git-refs.js";
 import { genesisSettingsKey, publishInputsKey, settingsSnapshot } from "./settings.js";
 import { AttemptLimiter } from "./rate-limit.js";
+import { BootstrapSecretDownloadLimiter } from "./bootstrap-secret-limit.js";
 import { checkCommit, checkGitRef, checkRepoUrl, loadRepoPolicy, redactUrl, releasePolicyErrors } from "./repo-policy.js";
 import { isPublicIp, probeSeedNode, validateSeedNodeIp } from "./seeders.js";
 import { DEFAULT_DEPLOYMENT_REPO, DEFAULT_GO_ZENON_REPO, readState, updateState } from "./storage.js";
@@ -58,6 +59,7 @@ const REPO_POLICY = loadRepoPolicy(process.env, [DEFAULT_GO_ZENON_REPO, DEFAULT_
 const LOGIN_WINDOW_MS = 15 * 60_000;
 const loginLimiterByAccountAndAddress = new AttemptLimiter({ maxAttempts: 10, windowMs: LOGIN_WINDOW_MS });
 const loginLimiterByAddress = new AttemptLimiter({ maxAttempts: 50, windowMs: LOGIN_WINDOW_MS });
+const bootstrapSecretDownloadLimiter = new BootstrapSecretDownloadLimiter();
 const MAX_CONCURRENT_LOGINS = 8;
 let loginsInFlight = 0;
 
@@ -683,7 +685,7 @@ function bootstrapManifest(request: express.Request, published: PublishedArtifac
 async function withBootstrapNode(
   request: express.Request,
   response: express.Response,
-  handler: (state: AppState, node: BootstrapNode) => Promise<void> | void
+  handler: (state: AppState, node: BootstrapNode, tokenHash: string) => Promise<void> | void
 ): Promise<void> {
   const token = bearerToken(request);
   if (!token) {
@@ -695,19 +697,27 @@ async function withBootstrapNode(
   const state = await readState();
   const pillar = state.pillars.find((candidate) => candidate.statusTokenHash === tokenHash);
   if (pillar) {
-    await handler(state, { nodeType: "pillar", pillar });
+    await handler(state, { nodeType: "pillar", pillar }, tokenHash);
     return;
   }
 
   const seedNode = state.seedNodes.find((candidate) => candidate.statusTokenHash === tokenHash);
   if (seedNode) {
-    await handler(state, { nodeType: "seed", seedNode });
+    await handler(state, { nodeType: "seed", seedNode }, tokenHash);
     return;
   }
 
   response.status(401).json({ error: "Invalid bootstrap token" });
 }
 
+function admitBootstrapSecretDownload(response: express.Response, tokenHash: string): boolean {
+  const retryAfterMs = bootstrapSecretDownloadLimiter.admit(tokenHash);
+  if (retryAfterMs === 0) return true;
+  response.setHeader("Retry-After", String(Math.ceil(retryAfterMs / 1000)));
+  response.status(429).json({ error: "Too many secret downloads; retry later" });
+  return false;
+}
+
 
 function historySample(report: NodeStatusReport): NodeStatusReport {
   // History only needs the numeric time series; drop per-peer detail, log lines, and the last
@@ -932,33 +942,36 @@ async function main() {
   });
 
   app.get("/api/bootstrap/producer.json", async (request, response) => {
-    await withBootstrapNode(request, response, (_state, node) => {
+    await withBootstrapNode(request, response, (_state, node, tokenHash) => {
       if (node.nodeType !== "pillar") {
         response.status(404).json({ error: "Seed nodes do not have producer wallets" });
         return;
       }
+      if (!admitBootstrapSecretDownload(response, tokenHash)) return;
       sendJsonFile(response, node.pillar.producerWallet.keyFile);
     });
   });
 
   app.get("/api/bootstrap/producer-password.txt", async (request, response) => {
-    await withBootstrapNode(request, response, (_state, node) => {
+    await withBootstrapNode(request, response, (_state, node, tokenHash) => {
       if (node.nodeType !== "pillar") {
         response.status(404).json({ error: "Seed nodes do not have producer wallets" });
         return;
       }
+      if (!admitBootstrapSecretDownload(response, tokenHash)) return;
       response.setHeader("Content-Type", "text/plain; charset=utf-8");
       response.setHeader("Cache-Control", "no-store");
       response.send(`${producerPassword(node.pillar)}\n`);
     });
   });
 
   app.get("/api/bootstrap/network-private-key", async (request, response) => {
-    await withBootstrapNode(request, response, (_state, node) => {
+    await withBootstrapNode(request, response, (_state, node, tokenHash) => {
       if (node.nodeType !== "seed") {
         response.status(404).json({ error: "Pillar nodes do not have managed network private keys" });
         return;
       }
+      if (!admitBootstrapSecretDownload(response, tokenHash)) return;
       response.setHeader("Content-Type", "text/plain; charset=utf-8");
       response.setHeader("Cache-Control", "no-store");
       response.send(`${decryptText(node.seedNode.networkPrivateKeyCipher)}\n`);