infra(archive-node): bare-metal buildout — provision → sync → first-party RPC origin

[JSONbored]

Archive node: from raw provisioned server to production RPC origin

Supersedes this issue's original scope, which described a pruned (128GB) subtensor-node service on Railway. The actual plan (decided in #2518/#2518) is a dedicated bare-metal archive node on Latitude.sh, not a pruned node on a managed platform. Server is already provisioned by the owner (2026-07 buildout, being built tonight alongside the indexer node).

Deliverable

A full, unpruned (--pruning=archive --sync=full) subtensor node, synced to head, serving as (1) the RPC origin for the indexer node's index-chain/indexer-rs + backfill services, and (2) the first TRUSTED_RPC_UPSTREAM_ORIGINS entry in the Worker's RPC proxy, with public archive nodes demoted to failover.

Sizing reference

docs.learnbittensor.org/subtensor-nodes/subtensor-node-requirements: 4 cores/16GB min; archive storage ~3.5TB+ as of block #7,929,000 (2026-03-28) — confirm the provisioned box has real headroom past that as the chain continues to grow. No official warp-sync snapshot URL exists — initial sync is either full genesis sync or a self-produced snapshot.

Steps (see sub-issues)

1. OS bootstrap & hardening
2. Install + configure subtensor (full archive), systemd unit, disk/IO tuning
3. Initial sync + self-snapshot to R2 (reproducible bootstrap for future replicas / geo-HA)
4. Decide + implement the network path to the indexer node (private VLAN / WireGuard / RPC-over-TLS — currently undocumented anywhere, a real gap flagged in the 2026-07-03 architecture research)
5. Wire into the Worker's RPC proxy as first-party origin
6. Monitoring + alerting (sync lag, peer count, disk usage, RPC latency) — process-restart alerting shipped, see infra(archive-node): subtensor txpool-background "crash" — root-caused as benign (scheduled restart), not a bug #4775
7. Ansible role (subtensor-archive) — see Decision: deep-history cold-tier data architecture (2-box: archive node + indexer/DB) — PG/Timescale vs +columnar sibling #2518-config-mgmt (JSO-2517)

Part of #2518 (#2518). Related: indexer-node epic. See ADR 0013 (docs/adr/0013-hybrid-deployment-topology.md) + runbook (deploy/README.md) — note both currently assume co-location with the indexer box, which the real 2-box plan doesn't have; the network-path sub-issue above is where that gets resolved.

maintainer-only.

[JSONbored]

Status update (2026-07-03): the bare-metal box is live and syncing — genuinely substantial progress since this was filed, tracked in detail in Linear (JSO-1926 and children):

• OS bootstrap/hardening: done (Ansible common role, metagraphed-infra repo).
• subtensor install/config/tuning: done, container running (--pruning=archive --sync=full, tuned ulimits/wasm-execution/cache sizes, verified against real --help output).
• Initial sync: in progress, climbing steadily from genesis toward the chain tip. Self-snapshot to R2 still pending (comes after sync completes).
• Network path to the indexer node: resolved via Tailscale (not a VLAN or WireGuard tunnel) — both boxes joined a shared tailnet, verified end-to-end with a real cross-box RPC call.
• Monitoring: Prometheus/Grafana/Alertmanager deployed and alerting to Discord (sync lag, peer count, disk thresholds); verified live with real test alerts.
• Ansible role (subtensor-archive): done, reproducible from a clean install.

Not yet done: wiring this box into the Worker's TRUSTED_RPC_UPSTREAM_ORIGINS (waiting on the box reaching steady-state sync before promoting it from failover to first-party origin).

[JSONbored]

2026-07-04 progress update:

Step 4 (network path to the indexer node) is resolved, not still a gap: a Tailscale ACL grant (tag:edge-indexer → tag:edge-archive:9944, jsonbored/tailscale-tailnet) lets the indexer node reach the archive node's subtensor RPC directly over the tailnet. Verified working: a cross-box RPC call from the indexer node to the archive node succeeded once the RPC bind picked up the box's real Tailscale IP.

Sync status: the full-archive subtensor node (--pruning=archive --sync=full) is live and syncing from genesis, currently ~21% (block ~1.83M of ~8.54M chain tip as of 2026-07-04), ETA roughly a week out at the current rate. Confirmed genuinely full-archive (not just configured-but-untested): state queries succeed at already-synced heights (tested blocks 1 and 1,000,000), fail cleanly at not-yet-synced heights.

Also found and fixed along the way: metagraphed-indexer-rs had a real bug where subxt's per-block metadata fetch could hang indefinitely against our own node specifically while it's mid-sync under heavy import churn (a known, still-open upstream gap, paritytech/subxt#2050). Fixed with a timeout+reconnect wrapper (ChainClient, dbbc8b9) — verified live against the actual reproducing condition. This doesn't make the node usable for indexing before it finishes syncing, but removes a real correctness landmine for when it does.

Remaining steps (5-7: Worker RPC-proxy wiring, monitoring/alerting, Ansible role) are tracked separately — monitoring is already partially live (Alertmanager → Discord relay deployed this session); Ansible role (roles/subtensor-archive) already exists in metagraphed-infra.

See #2956 for the follow-on Postgres/Redis/indexer cutover from Railway to this box, gated on this issue's sync completing.