Complete Railway DB retirement after static weights cutover

[wallscaler]

Context

Railway main Postgres is about 494GB on a 500GB volume. UID 200 validator weights now come from a Polaris sandbox publisher that writes signed static artifacts to wallscaler/cathedral-weight-feed, served through the cathedral-weights-failover Cloudflare Worker at api.cathedral.computer.

Goal

Retire the large Railway DB/project safely without risking UID 200 validator health.

Current status

• Static weights path is live on https://api.cathedral.computer/v1/validator/weights/next.
• Railway app services and Postgres deployments are stopped. Volumes are still preserved.
• Polaris sandbox remains the live composer and publisher.
• A sandbox user timer refreshes SN39 metagraph_hotkeys every 4 minutes.
• A sandbox user timer publishes signed weight artifacts every 3 minutes. The original system timer still runs every 10 minutes.
• Worker version 6d920a59-4a12-4cd4-8469-01f18e5c347f refreshes stale cached vectors synchronously and falls back from GitHub Contents API to raw GitHub before the now-stopped Railway read service.

Checklist

• Watch https://api.cathedral.computer/v1/validator/weights/next for at least one full validator interval.
• Confirm returned vectors stay fresh, signed, and accepted by scaffold.validator_thin --offline.
• Confirm sandbox weight feed publish timers keep pushing signed vectors.
• Confirm UID 200 validator is active with permit and set weights after static cutover.
• Add sandbox SN39 metagraph poller so static composition no longer depends on Railway DB freshness.
• Stop remaining nonessential Railway app services.
• Stop Railway Postgres deployments while preserving volumes.
• Preserve final minimal export artifacts and sandbox DB URL in credentials storage.
• Soak the static path after Railway stop.
• Delete or abandon the 494GB Railway DB/project only after the static path is stable and rollback expectations are explicit.

Verification log

• 2026-07-08T04:21Z: release gate passed for canonical and legacy API paths, using api.cathedral.computer for all weight URL checks. UID 200 was fresh after a new set-weights transaction.
• 2026-07-08T04:24Z: Railway app deployments stopped: cathedral-publisher, cathedral-read, cathedral-submit, cathedral-v2-worker. cathedral-v2-beta and cathedral-testlane were already stopped.
• 2026-07-08T04:25Z: Railway Postgres deployments stopped: Postgres and Postgres-GlUA. Volumes remain present and are not deleted.
• 2026-07-08T04:26Z: scaffold.validator_thin --publisher-url https://api.cathedral.computer --once --offline accepted vector 6688a3ff....
• 2026-07-08T04:26Z: release gate auto checks passed with vector age about 103s and UID 200 update age about 420s.

Rollback notes

• Fast rollback for public weights is to keep using the static feed. It is independent of Railway after this cutover.
• If Railway app or DB compute must be restored, use Railway redeploy on the stopped service deployments captured in ~/Documents/INBOX/cathedral-railway-retire/railway-status-before-stop-2026-07-08T0406Z.json and related status snapshots.
• Do not delete Railway volumes until after a longer soak and an explicit final abandonment decision.

Links

• PR: Route validator weights through static failover feed #370
• Static feed repo: wallscaler/cathedral-weight-feed
• Follow-up storage replacement: Replace GitHub static weights feed with object-storage edge feed #372

[wallscaler]

Railway compute retirement checkpoint complete. Static API path and UID 200 checks passed after stopping Railway app and Postgres deployments. Volumes are preserved. Final deletion remains gated on soak and explicit abandonment decision.

[wallscaler]

Cathedral Railway final DB abandonment plan

Current safe state

• UID 200 weights are served from the static GitHub feed through Cloudflare.
• The Polaris sandbox composes and publishes new signed vectors.
• Railway app and Postgres deployments are stopped.
• Railway volumes are preserved and not deleted.
• Minimal live state export is preserved under /Users/dreamboat/Documents/TOOLS/.credentials/cathedral-sandbox-migration-2026-07-07/minimal-state/.
• Railway status snapshots are preserved under /Users/dreamboat/Documents/INBOX/cathedral-railway-retire/.

Soak gates before deletion

Run these checks over at least 24 hours:

1. https://api.cathedral.computer/v1/validator/weights/next returns 200.
2. X-Cathedral-Vector-Source is static-github or static-github-raw.
3. Vector expires_at remains in the future with at least 10 minutes of margin during normal operation.
4. python3 -m scaffold.validator_thin --publisher-url https://api.cathedral.computer --once --offline accepts the vector.
5. UID 200 remains a permitted validator and fresh within one tempo.
6. Sandbox timers are active:
   • cathedral-publisher.service
   • cathedral-weight-feed-publish.timer
   • user cathedral-weight-feed-publish-fast.timer
   • user cathedral-sn39-metagraph-poller.timer
7. Public feed repo receives fresh commits from the sandbox.
8. No production validator path depends on read.cathedral.computer, submit.cathedral.computer, or Railway Postgres.

Deletion sequence

Do not run this sequence until the soak gates pass and the final abandonment decision is explicit.

1. Capture a final Railway status snapshot.
2. Capture final Cloudflare Worker version and route status.
3. Capture final public feed commit hash and vector ID.
4. Confirm the local validator accepted a static vector after Railway stop.
5. Confirm sandbox DB has fresh metagraph_hotkeys rows.
6. Confirm minimal export files are readable and checksums are saved.
7. Delete or abandon Railway services and volumes in this order:
   • stopped app services
   • V2 worker services
   • secondary Postgres volume Postgres-GlUA
   • main 494GB Postgres volume Postgres
   • project only after services and volumes are gone
8. Immediately re-run the static API checks and thin validator check.
9. Close issue Complete Railway DB retirement after static weights cutover #371 only after post-deletion checks pass.

Rollback expectations

• Before volume deletion: Railway can be restored by redeploying stopped services.
• After volume deletion: rollback is the sandbox static feed only. The large Railway DB is intentionally gone.
• Keep the sandbox DB and minimal export as the recovery floor until object-storage feed replacement in Replace GitHub static weights feed with object-storage edge feed #372 is complete.

[wallscaler]

Index probe at 2026-07-14 18:36 MDT found the static artifact generator alive but canonical distribution broken.

• api.cathedral.computer/v1/validator/weights/next: 404 Railway Application not found
• legacy prefixed route: 404
• read.cathedral.computer, submit.cathedral.computer, and v2-beta.cathedral.computer: same Railway fallback 404
• raw wallscaler/cathedral-weight-feed vector: fresh, HTTP 200, signature accepted by the pinned Cathedral key
• release gate against the API: failed
• Cathedral validator UID 200: stale by about 25,045 blocks / 83.5 hours
• 10 of 11 other permitted validators were fresh

The artifact publisher is still updating. The immediate failures appear to be the Cloudflare/public route binding and the UID 200 validator process.

[wallscaler]

2026-07-19 live recheck during the Polaris Railway exit:

• Verda reports cathedral-v2-standby-fin03 running at 95.133.253.189.
• sandbox-v2-origin.cathedral.computer still resolves to that IP.
• Local readiness is HTTP 200 on both 127.0.0.1:8000/health/ready and 127.0.0.1:8080/health/ready.
• PostgreSQL and cathedral-publisher.service are active.
• User cathedral-v2-beta-origin.service is active with six Uvicorn workers.
• System weight publishing and user fast-publish, metagraph-poller, and CNF-prebake timers are active.
• v2-beta.cathedral.computer/health/ready still returns Railway Application not found, consistent with the route-binding failure recorded above.

Conclusion: do not stop or delete the Verda instance. It is the healthy Cathedral V2 origin. The remaining failure is at the public edge binding, already tracked by this P0 issue.