Retire riv-dev1 Blockscout after riv-subgraphs-a reaches coverage parity

[DenisCarriere]

Context

Blockscout is now healthy and public on riv-subgraphs-a using dom826 (EPYC 9274F) and local TopoLVM storage. riv-dev1 remains active as a temporary safety copy while the new, fresh database backfills.

Activation evidence

• target endpoint: https://robinhood-blockscout.riv-subgraphs-a.pinax.io
• workload revision: b447f0afbfdc8c3c5b3d1fbafc9ae7baac5abbbc
• fleet revision: 8baa2a8580cd81d905ea9200be719e30f7c935b9
• 597-second active test: 78.9 committed blocks/sec and 68.9 net historical blocks/sec
• old sustained observation: approximately 19.8 committed blocks/sec and 9.3 net historical blocks/sec
• target pods Ready with zero restarts and zero pending block operations

Initial parity snapshot

At the handoff snapshot:

• riv-dev1 missing-range aggregate: 25,925,266 blocks
• riv-subgraphs-a missing-range aggregate: 26,986,591 blocks
• coverage gap: 1,061,325 blocks

At the observed net-rate difference, parity is expected in roughly five hours, but the live database values are the gate rather than the estimate.

Cutover gates

• riv-subgraphs-a missing-range aggregate is less than or equal to riv-dev1
• target Firehose catchup remains active without pending-operation accumulation
• target PostgreSQL, Redis, all pods, Flux Kustomizations, certificate, Gateway, frontend, and API health are Ready
• compare representative historical block, transaction, and API responses
• remove clusters/dev/blockscout.yaml from k8s-pinax-fleet through a PR
• verify riv-dev1 resources are pruned and the riv-subgraphs-a public endpoint remains healthy

Do not raise catchup concurrency during this comparison; 24 workers was slower than 16 on riv-dev1.

[DenisCarriere]

riv-dev1 removal status

The Kubernetes deployment removal is complete:

• fleet PR pinax-network/k8s-pinax-fleet#44 merged and reconciled
• workload cleanup PR pinax-network/k8s-blockscout#10 merged
• blockscout and blockscout-system namespaces are absent
• tenant/source/Kustomizations, RBAC, Gateway, certificate, workloads, Services, PVCs, and old DNS record are absent
• riv-subgraphs-a remains Ready and public on revision eb479906b38293ae5c09c19fd13338f4514f5999

Two Ceph PV objects/backing images remain as the final storage cleanup gate:

• pvc-3dd3138a-bdaa-4641-aa09-4b990fcfa889 — former 1 TiB PostgreSQL volume
• pvc-9c522284-c5b9-45d2-b0d0-3fa7f7a5f537 — former 20 GiB Redis volume

Both PVs are Released/Terminating with Delete reclaim policy and the external-provisioner finalizer intentionally retained. Ceph reports HEALTH_ERR: two full OSDs and eight full pools. CSI DeleteVolume calls time out, and the old node mappings remain in uninterruptible I/O despite the pod objects and namespaces being gone. Controller restarts and exact, scoped unmap attempts were exhausted; finalizers were not stripped because that would silently orphan the images.

Resume gate:

1. Restore enough Ceph headroom for RBD metadata operations.
2. Clear the exact stale mappings for the two volume IDs on dom977 and dom965.
3. Allow CSI deletion to complete or remove the two confirmed csi-vol-* images with scoped provisioner credentials.
4. Verify both PV objects and both RBD images are absent, then close this issue.

No Blockscout service remains on riv-dev1; this issue stays open only for physical Ceph image cleanup.