The UTXO snapshot over BitTorrent first (pinned info hash), bounded seeding, https as the fallback

[chrisguida]

Rebased on next; this branch includes #1's commits (stacked), the last commit is this one. Not a draft any more:
nothing here depends on a Knots release. The earlier version of this PR replaced the chain tarball with assumeutxo;
next did that already, with the 910,000 snapshot that 29.4.2 knows, so only the download part remains.

Why

Every new node downloads 9.6 GB from one server with a few download slots. With the swarm, other installing nodes and
a couple of seeders carry that, the server is the fallback, and a busy server costs nothing: the torrent needs no
slot. The pinned info hash is as strong a pin as the sha256: aria2 takes no piece whose hash is not in the metadata
the info hash names, and the sha256 and header checks follow as before.

What

• utxo_table gains two optional columns: the file's BitTorrent info hash and the URL of its .torrent file.
  tools/make-torrent.py FILE --webseed URL (pure Python, no dependencies) prints the sha256, the info hash and a
  magnet link; README "Publishing a snapshot" describes the two steps. No info hash is pinned for the 910,000 file in
  this PR: its URL is not live yet (404 today), so I could not compute it. Once the file is published, one run of the
  script gives the value for column 7; a .torrent next to the file (column 8) is optional but better, because its
  web seed lets a node without peers finish over https within the same download.
• Step 5: with an info hash, the download comes over BitTorrent first (aria2; the pinned .torrent if its info hash
  is the pinned one, else the magnet link from the info hash and four pinned public trackers). The pieces land in
  FILE.bt, never in the https file (a torrent's partial file is sparse and must not be "resumed" sequentially); a
  re-run resumes with aria2 re-checking the pieces. No download speed for 15 minutes ends the attempt and the https
  download takes over, with the slot waits as before. An https partial from an earlier run is left to https.
  --no-torrent: https only.
• After the sha256 and header checks the node seeds the file to other installing nodes for at most 2 hours or share
  ratio 2 (transient service alphapool-snapshot-seed, the saved metadata in /var/lib/alphapool/seed, root only).
  The seeder keeps the file open, so the node's load, which moves and deletes the file, does not stop it; the space is
  freed when it ends. If ufw is on, TCP 6881 is allowed meanwhile and removed after (and at uninstall). --no-seed
  turns it off; systemctl stop alphapool-snapshot-seed stops it early.
• The validated-start journal records the info hash and the .torrent URL, so a resume after a reboot uses them.
  --print-cloud-init passes --no-torrent / --no-seed on.
• README ("How the node gets its chain", "Waiting for a download slot", "Downloads"), CHANGELOG, SHA256SUMS.

Testing done

bash -n everywhere. utxo_file_ready, utxo_torrent_download and utxo_seed_start were run with aria2c, curl,
systemctl, systemd-run and ufw shimmed and a small fake snapshot with a real header: magnet success (file verified,
metadata saved, seeder started), torrent failure then https, --no-torrent, no info hash pinned (today's line:
https only, nothing changes), a .torrent with the wrong info hash (refused, magnet used), a .torrent with the
pinned one (used), an https partial from before (torrent steps aside), --no-seed, aria2 writing past --index-out
(no sparse file reaches https), a corrupted file (AP-401), and the seed helper's cleanup. Not done: a real swarm
download on a server; I will run one against a seeder of mine as soon as the 910,000 file is published, and post
the timings here.

[chrisguida]

Full run of this branch on a fresh DigitalOcean droplet (s-2vcpu-4gb: 2 vCPU, 4 GB, 80 GB, Ubuntu 24.04), with a
976,000 row added to utxo_table (info hash 3cf7e4d15841f116856f6f19bf2ac15b1dbae2c3, .torrent next to the file)
and a Bitcoin Knots 29.4.2 build that carries the 976,000 entry via --knots-dir (Guix build of
bitcoinknots/bitcoin#444's branch; sha256 of the tarball 5c26890d72daa499fe22b905de5cfb0a78e2445aedbde0726278f57672106a9d):

step	time
1-4 packages, user, Knots, gateway	20 s
5 the 9.5 GB snapshot over BitTorrent (one seeder on another droplet, the https web seed, a few outside peers)	5 min 40 s at 31-36 MB/s
5 sha256 + header check, seeder started	10 s
6-8 config, firewall, node start	10 s
9 block headers	1 min
9 loadtxoutset (167,095,701 coins)	18 min
install complete	25 min 36 s
9 catch-up of the 141 blocks since the snapshot, gateway gets its job	1 min 20 s
READY (gateway has AlphaPool's job)	26 min 57 s

For comparison on the same plan: the published 2026-10-05.2 (chain tarball) took 32 min 50 s to READY; next's
default from the 910,000 snapshot validates ~66,000 blocks after the load (about half a day).

The seeder on the installing node worked as designed (transient unit, metadata copy in /var/lib/alphapool/seed,
file kept open through the handover and deletion). One gap showed up and is fixed in the last commit: the seeder
starts in step 5, before step 7 turns ufw on, so on a server where ufw was off the seeding port was never allowed;
step_firewall now allows 6881-6889 while the seeder is active.

Still open from my side: the info hash for your 910,000 file (its URL is a 404 for me today). Once it is published,
tools/make-torrent.py utxo-910000.dat --webseed <url> gives the value, and I will seed it from a droplet of mine.

[chrisguida]

Rebased onto 632a912 on top of the rebased #1 (still stacked: its three commits come first). Two merges in install.sh: the column 7/8 comment goes under your new line above utxo_table, and uninstall removes both your alphapool-build.issue and the seed directory. One content change: your 976000 row now carries the torrent info hash 3cf7e4d15841f116856f6f19bf2ac15b1dbae2c3. Your utxo-976000.dat is byte-identical to the file the torrent was made from, so the info hash is the same; the https URL in the row stays the fallback and there is no .torrent URL column, so the magnet comes from the info hash and the pinned trackers. The 910000 row has no info hash yet, so that route is unchanged. torrent-next-test.sh: 16 of 16 pass.

[chrisguida]

Added the 910000 row's torrent info hash too (4c8f819): e1fb935bdbc0429c8c2521cb81d6c8f42ff26f8b, made from your published utxo-910000.dat (sha256 6ac02081…, 9,637,809,744 bytes) with tools/make-torrent.py, 4 MiB pieces, your https URL as the web seed. A seeder on a droplet of mine now carries both snapshots. torrent-next-test.sh still 16 of 16.