Skip to content

fix(data-api): 25 more D1-only handlers — some serving frozen data since the streamer stopped #4832

Description

@JSONbored

Context

Follow-up to #4826/PR #4830 (account_events analytics routes). That audit was scoped to
account_events-derived analytics routes only (nominators, weight-setters, stake-flow, etc.) and
missed a second, larger category: request handlers in workers/request-handlers/entities.mjs that
read blocks/extrinsics/chain_events/account_events/neurons/neuron_daily directly via
d1Runner(env), with zero tryPostgresTier call
. Found via a systematic cross-reference of every
export async function handle* against tryPostgresTier(env call sites in the same file.

The metagraphed-streamer container (the D1 chain-data ingestion path) stopped 16 hours ago
(confirmed via docker ps on meta-indexer-01-us-lax1) — this is expected, since blocks/extrinsics/
account_events already serve from Postgres for the ported routes. But the handlers below still read
D1 directly, so they are silently serving data frozen at the moment the streamer stopped, while
their sibling routes on the exact same underlying chain data (e.g. /blocks/{ref} vs
/blocks/{ref}/extrinsics) correctly show live Postgres data. This is a live, active, currently-serving
correctness bug — not a hypothetical.

D1's neurons/neuron_daily are a separate case: refresh-metagraph.yml still dual-writes to both
D1 and Postgres (explicitly, per its own "#4771 verification window" comment — which is itself stale,
since METAGRAPH_NEURONS_SOURCE has already been flipped to "postgres" for the 5 live-state
handlers it covers). So these reads are NOT stale, just not yet on the resilient/scalable Postgres
path, and D1's neuron_daily (8.57M rows) is very likely the dominant contributor to D1 sitting at
9.13/10 GB (91%) right now.

Tier 1 — frozen D1 reads (blocks/extrinsics/chain_events/account_events), high priority

All read via d1Runner(env) with no tryPostgresTier, serving data frozen since the streamer stopped:

  • handleBlockExtrinsics (GET /blocks/{ref}/extrinsics) — loadBlockExtrinsics, extrinsics table
  • handleBlockEvents (GET /blocks/{ref}/events) — loadBlockEvents, chain_events table
  • handleBlocksSummary (GET /blocks/summary) — loadBlocksSummary, blocks table
  • handleAccountExtrinsics (GET /accounts/{ss58}/extrinsics) — loadAccountExtrinsics, extrinsics table
  • handleSudo (GET /sudo) — loadExtrinsics filtered call_module='Sudo', extrinsics table
  • handleGovernanceConfigChanges (GET /governance/config-changes) — loadExtrinsics filtered call_module='AdminUtils', extrinsics table
  • handleRuntime (GET /runtime) — loadRuntimeVersionHistory, blocks table (spec_version transitions)
  • handleSubnetEvents (GET /subnets/{netuid}/events) — loadSubnetEvents, account_events table
  • handleSubnetEventSummary (GET /subnets/{netuid}/event-summary) — loadSubnetEventSummary, account_events table
  • handleAccountSubnets (GET /accounts/{ss58}/subnets) — loadAccountSubnets, account_events table
  • handleAccount (GET /accounts/{ss58}) — loadAccountSummary, account_events table
  • handleSubnetAlphaVolume (GET /subnets/{netuid}/alpha-volume) — loadSubnetAlphaVolume, account_events table
  • handleSubnetWeights (GET /subnets/{netuid}/weights) — loadSubnetWeights, account_events table (NOT handleSubnetWeightSetters, its sibling — that one's already ported)

Tier 2 — neurons/neuron_daily reads, not stale but not on the resilient path

  • handleSubnetHistory, handleChainPerformance, handleChainYield, handleChainTurnover,
    handleSubnetTurnover, handleSubnetMovers, handleValidatorHistory, handleNeuronHistory
    all neuron_daily
  • handleChainConcentration, handleSubnetYield, handleAccountPortfolio, handleAccountsList
    all live neurons

NOT in scope (intentionally D1-native, confirmed via ADR/#4772)

handleSubnetIdentityHistory, handleChainIdentityHistory, handleAccountIdentity,
handleAccountIdentityHistory (subnet_identity_history/account_identity), handleSubnetHyperparams,
handleSubnetHyperparamsHistory (subnet_hyperparams), handleAccountPositionHistory
(account_position_daily) — all explicitly designed as permanent D1 tables, not part of the chain-data
cutover. Do not port these.

Proposed fix

Same pattern as PR #4830: add the missing Postgres SQL (mirroring each D1 loader's query) to
workers/data-api.mjs, wire tryPostgresTier(env, request, "<FLAG>") into each handler ahead of its
D1 fallback, matching each loader's exact return shape. Tier 1 reuses the existing
METAGRAPH_EXTRINSICS_SOURCE/METAGRAPH_ACCOUNT_EVENTS_SOURCE flags (already "postgres"). Tier 2
reuses METAGRAPH_NEURONS_SOURCE (already "postgres").

Given the size, land as two or three focused PRs (Tier 1 blocks/extrinsics-derived, Tier 1
account_events-derived, Tier 2 neurons/neuron_daily-derived) rather than one, mirroring #4830's own
scope discipline.

Priority

High for Tier 1 (active data-correctness bug, silently serving stale data with no error). Medium for
Tier 2 (not stale, but blocks D1 shrinkage — neuron_daily's 8.57M rows are very likely why
metagraphed-health D1 is at 9.13/10 GB right now) and is also a hard prerequisite for #4772 (D1
retirement) since D1 can't be deleted while any handler still depends on it for reads.

Links/resources

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions