Investigate D1 coverage gap for AdminUtils.sudo_set_weights_version_key

[JSONbored]

Context

This is a low-priority offshoot of the call_args shape-parity audit tracked in #4669 (itself following from #4668's live-verification of the D1→Postgres serving cutover, ADR 0013 Sequencing step 3). While walking all 105 call types D1 and Postgres both currently ingest, 13 came back "unresolved" because neither store had a recent occurrence to compare. 12 of those 13 are genuinely dormant on both sides (0 occurrences in the 365-day window in D1 and in Postgres) — e.g. Sudo.sudo, SubtensorModule.faucet, SubtensorModule.commit_crv3_weights. AdminUtils.sudo_set_weights_version_key is the one exception:

AdminUtils.sudo_set_weights_version_key (Postgres HAS a recent in-window occurrence at block 8584440, yet D1's own filtered feed returns zero rows for this call type — this one looks like a genuine D1 coverage gap, not a retention/rarity issue)

Re-querying D1's dedicated AdminUtils feed directly against production just now reproduces the same result:

$ curl -s "https://api.metagraph.sh/api/v1/governance/config-changes?call_function=sudo_set_weights_version_key&limit=10"
{"ok":true,"data":{"extrinsic_count":0,"extrinsics":[],...}}

That route is handleGovernanceConfigChanges (workers/request-handlers/entities.mjs:3849, routed via GOVERNANCE_CONFIG_CHANGES_PATH_PATTERN in workers/config.mjs:266) — it's loadExtrinsics hardcoded to callModule: "AdminUtils" against the same D1 extrinsics tier the cutover flag gates. Its own comment explains why this matters more than it would for most call types:

workers/request-handlers/entities.mjs:3846-3848: "most AdminUtils calls (77 of ~83) don't emit their own dedicated event, so the extrinsic + its decoded call_args is the reliable source, not chain_events."

In other words, for this call family the extrinsics tier isn't just a source, it's the only first-party place this occurrence could be corroborated from chain data — a D1-side gap here is exactly the case that comment is warning about, not a cosmetic difference.

This is unrelated to the call_args shape divergence (D1 SS58/hex strings vs. indexer-rs's raw newtype arrays) that's the actual blocker for flipping METAGRAPH_EXTRINSICS_SOURCE (still "d1" per wrangler.jsonc:88, METAGRAPH_BLOCKS_SOURCE already flipped to "postgres" at wrangler.jsonc:87). If anything it's a point in Postgres's favor: Postgres has data D1 is missing, not the reverse.

A plausible mechanism exists in D1's own ingestion path, worth checking first. scripts/fetch-events.py's extrinsics_for_block() silently drops any extrinsic that raises during decode:

# scripts/fetch-events.py:712-734
for extrinsic_index, ext in enumerate(extrinsics):
    try:
        value = ext.value if ext is not None else None
        if not isinstance(value, dict):
            continue  # an undecodable extrinsic — skip this row only
        ...
    except Exception:
        continue  # shape drift on one extrinsic → skip it, keep the rest

and _extrinsic_call() (scripts/fetch-events.py:567-585) independently swallows its own exceptions, returning (None, None, None) rather than raising — so a row for this call could either (a) never have been written at all (a per-extrinsic decode exception at line 733-734), or (b) exist with call_module/call_function left NULL (an internal _extrinsic_call failure), which would also make it invisible to a call_function=sudo_set_weights_version_key filter without being a true "zero rows" situation. Both are testable hypotheses, not conclusions — this issue is to actually determine which (if either) applies, or whether it's simpler (e.g. block 8584440 sits outside D1's current retention/backfill window).

Requirements

• Determine which of the following actually explains the gap, with direct evidence (not another audit-style extrapolation):
  1. True ingestion gap — D1's poller (scripts/fetch-events.py) never saw/wrote a row for this extrinsic at all (block outside its scan window, a continue at fetch-events.py:716 or :734 was hit for this specific extrinsic, or the cursor/backfill never covered block 8584440).
  2. Silent null-attribution — a row exists in D1 for the right (block_number, extrinsic_index) but with call_module/call_function NULL because _extrinsic_call() (fetch-events.py:567-585) hit its own internal exception path, so it exists but is invisible to the call_function filter.
  3. Retention/pruning — the row was written but has since aged out of D1's retained window (check D1's current min/max block_number for the extrinsics tier and whether 8584440 falls inside it).
• Confirm the corresponding Postgres row (block 8584440, AdminUtils.sudo_set_weights_version_key) independently via the indexer box to pin the exact extrinsic_index, then look up the identical (block_number, extrinsic_index) coordinate directly against D1 (not just the call_function-filtered route) to distinguish case 2 from cases 1/3.
• Do not bundle a fix into this ticket. This is a diagnosis-only ticket per its stated priority; if the root cause turns out to be a one-line, low-risk fix (e.g. an obvious metadata-version branch gap in _extrinsic_call), open it as its own separate, focused PR rather than folding a code change in here (house rule: one focused change per PR).
• Whatever the finding, fold it into the parity work as an annotation, not a new tracked blocker — update Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669's "unresolved call types" note (or wherever the parity harness from Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669/feat(api): serving-cutover for blocks + extrinsics from D1 to Postgres #4668 ends up living) to record this call type's specific status, distinct from the 12 genuinely-dormant ones.
• Explicitly state whether this affects the METAGRAPH_EXTRINSICS_SOURCE cutover decision (expectation: no — Postgres already has the data, so this is D1 under-coverage, not a Postgres shape or completeness problem).

Deliverables

• A short written diagnosis (posted as a comment on this issue and/or Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669) stating which of the three mechanisms above applies, including the exact wrangler d1 execute / psql / curl commands run and their raw output — mirroring the evidentiary style already used in Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669's status update.
• If mechanism (1) or (2): a minimal captured repro of the specific extrinsic's decoded ext.value shape (or lack thereof) that demonstrates exactly where D1's pipeline drops or mis-attributes it — no test file required for a diagnosis-only ticket, but a saved query artifact/comment is sufficient.
• An update to Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669 (or its successor parity-harness doc) reclassifying AdminUtils.sudo_set_weights_version_key out of the generic "genuinely rare/dormant on both sides" bucket into an annotated line describing the actual cause found here.

Expected outcomes

• A one-paragraph, evidence-backed root-cause statement exists for why GET /api/v1/governance/config-changes?call_function=sudo_set_weights_version_key returns extrinsic_count: 0 in production while Postgres has a confirmed row at block 8584440.
• Re-running that same query (or the equivalent direct D1 SQL) after investigation either still returns zero rows with a documented, understood reason attached, or returns the row because an actual fix (tracked separately, per Requirements) landed.
• Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669 (or wherever the parity findings are consolidated) reflects this call type's specific status rather than lumping it with the 12 genuinely-dormant ones.
• No change to the METAGRAPH_EXTRINSICS_SOURCE cutover readiness assessment — this issue is confirmed non-blocking for that decision either way.

Links/resources

• Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669 — "Reconcile call_args shape between indexer-rs and D1" (parent tracking issue; this ticket's audit findings originate there)
• feat(api): serving-cutover for blocks + extrinsics from D1 to Postgres #4668 — the PR that introduced METAGRAPH_BLOCKS_SOURCE/METAGRAPH_EXTRINSICS_SOURCE and the tryPostgresTier fallback mechanism
• fix(ui): make multisigCallHash/proxyRealAccount shape-agnostic (#4669) #4672 — multisigCallHash/proxyRealAccount made shape-agnostic (prior work in the same parity effort)
• fix(ui): render indexer-rs's nested-call shape in the extrinsic detail page #4676 — nested-call rendering fix for indexer-rs's shape in the extrinsic detail page (prior work in the same parity effort)
• ADR 0013 (docs/adr/0013-hybrid-deployment-topology.md) — Sequencing step 3, "serving cutover," the umbrella this work sits under
• scripts/fetch-events.py:689-735 (extrinsics_for_block) and :567-585 (_extrinsic_call) — D1's ingestion path and its silent-skip/null-attribution behavior
• workers/request-handlers/entities.mjs:3843-3891 (handleGovernanceConfigChanges) — the D1 "filtered extrinsics feed" the audit and this issue both query
• workers/config.mjs:262-267 (GOVERNANCE_CONFIG_CHANGES_PATH_PATTERN) — route definition (/api/v1/governance/config-changes)
• wrangler.jsonc:87-88 — the two existing cutover flags (METAGRAPH_BLOCKS_SOURCE: "postgres", METAGRAPH_EXTRINSICS_SOURCE: "d1")
• deploy/postgres/schema.sql — Postgres extrinsics.call_args jsonb column definition, indexer box: sudo docker exec -i metagraphed-indexer-postgres-1 psql -U metagraphed -d metagraphed for direct verification of block 8584440
• Evidence citation: Postgres AdminUtils.sudo_set_weights_version_key occurrence at block 8584440 (from the Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669 audit); live reproduction via curl https://api.metagraph.sh/api/v1/governance/config-changes?call_function=sudo_set_weights_version_key → extrinsic_count: 0 (run during this issue's drafting, 2026-07-09/10)

Priority

low — informational data-quality/coverage finding on the D1 side that does not block the call_args shape-parity work or the METAGRAPH_EXTRINSICS_SOURCE cutover decision; fold into the parity harness as an annotation rather than tracking standalone.

[linear-code]

JSO-3376

[JSONbored]

Diagnosis: mechanism (1), true ingestion gap — root cause identified and independently reproducing live right now

Confirmed via direct D1 query (wrangler d1 execute equivalent, run against the metagraphed-health database) before D1 became unresponsive to further queries mid-investigation (see below): block 8584440 itself is absent from D1's blocks table, not just this one call type's filtered feed. Its immediate neighbors are scattered, not a clean cutoff — 8584438 is present, 8584439 through at least 8584446 are missing, then presence resumes further out. This rules out case (3) (retention/pruning): a pruned range would be a contiguous cutoff from the oldest end, not a scattered hole in the middle of an otherwise-dense recent range, and 8584440 is nowhere near D1's retention boundary.

It also rules out case (2) (silent null-attribution in _extrinsic_call()): handleBlockIngest (workers/api.mjs:624-731) writes a batch's block rows and extrinsic rows in the same db.batch([...blockStmts, ...extrinsicStmts]) call. If the batch containing block 8584440 had failed, both the block row and every extrinsic row in it would be missing together — which is exactly what's observed (the block itself is gone, not present-with-null-call_function). A per-extrinsic decode exception in _extrinsic_call() would leave the block row intact with a null-attributed extrinsic row; that's not this.

Root cause, caught live

This is not specific to 8584440 — it's an actively recurring ingestion-gap mechanism, reproducing on meta-indexer-01-us-lax1 as of this comment:

• metagraphed-streamer's logs (docker logs metagraphed-streamer --since 3h) show 154 ingest_write_failed (HTTP 500) rejections and 15 stream reconnects in the last 3 hours alone, ongoing continuously since at least 05:21 UTC today with zero container restarts — this is sustained, not a blip.
• push() (scripts/stream-events.py:197-284) retries a failed ingest synchronously, blocking inside the same callback that services the finalized-head subscription (scripts/stream-events.py:366-396), with backoff up to 60s between attempts. A long retry stall appears to starve the underlying chain-RPC WebSocket's keepalive, producing the BrokenPipeError/JSONDecodeError "stream error" reconnects visible in the same log window.
• finalized_only=True head subscriptions don't backfill notifications missed during a reconnect gap — any block finalized while the streamer is disconnected/resubscribing is silently, permanently skipped by this pipeline. That produces exactly the scattered, few-blocks-at-a-time hole pattern seen around 8584440, rather than a clean range cutoff.
• Since the automatic backstop poller was retired 2026-07-04 (\.github/workflows/backfill-events.yml:6), nothing catches these gaps automatically — only a manual workflow_dispatch backfill run does, and evidently none has covered this range.

This is the same symptom #4654 hardened (uncaught exception → clean ingest_write_failed 500, 2026-07-09) but the underlying D1 write contention that trips it looks to have gotten worse, not resolved — 40+ minutes of near-continuous failure observed directly in this investigation, well past "occasional." A wider, currently-active investigation into this (including direct D1 429 "overloaded" responses on trivial reads, and a broader recent-block coverage sweep showing 38-61% missing in several 1000-block windows near the chain tip vs. 0% missing in an older window) is being filed as its own issue, since it's a live reliability problem well beyond this one call type.

Answer to this issue's specific questions

• Mechanism: (1), true ingestion gap. Not (2) null-attribution, not (3) retention/pruning.
• Does this affect the METAGRAPH_EXTRINSICS_SOURCE cutover decision? No — confirmed non-blocking, as expected. This is a D1-side ingestion gap; Postgres (fed by the separate indexer-rs pipeline, not this streamer) has the row.
• Folding this into Roadmap: complete the D1 -> Postgres serving cutover for chain data (ADR 0013) #4669's unresolved-call-types note as: "genuine D1 ingestion gap, root-caused to a streamer subscription-gap/D1-write-contention mechanism, tracked in [new issue]" rather than lumping with the 12 genuinely-dormant call types.