Skip to content

feat(indexer): detect and alert on ledger gaps using oldestLedger - #1041

Merged
DevTobis merged 2 commits into
DevTobis:mainfrom
kelvinokwudili52-stack:feat/indexer-ledger-gap-detection-54
Oct 25, 2026
Merged

DevTobis merged 2 commits into
DevTobis:mainfrom
kelvinokwudili52-stack:feat/indexer-ledger-gap-detection-54

Conversation

@kelvinokwudili52-stack

Copy link
Copy Markdown
Contributor

Summary

When the indexer resumes polling after an outage or downtime longer than Stellar RPC's retention window (~7 days), startLedger can fall behind oldestLedger. Previously, GetEventsResult only passed through events and latestLedger, leaving unrecoverable gaps untracked and risking false reconciliation breaks.

This change:

  • Passes through oldestLedger from the RPC getEvents response in services/indexer/src/rpc.ts.
  • Adds chain_event_gap table and migration in @ballast/db (packages/db/src/schema.ts) to store lost ledger ranges (from_ledger, to_ledger) for future data-lake backfills.
  • Adds indexer_gap alert type to @ballast/observability and sets cooldown per gap range.
  • Updates pollOnce in services/indexer/src/poller.ts to:
    • Detect when startLedger < oldestLedger.
    • Clamp startLedger to oldestLedger.
    • Log the lost range with structured logger.warn.
    • Fire an indexer_gap alert with fromLedger, toLedger, and oldestLedger.
    • Record the gap in chain_event_gap.
    • Update getLastIngestedLedger to consider recorded gaps so subsequent polling cycles do not re-alert on the same gap.
  • Adds unit tests in services/indexer/src/poller.test.ts verifying gap detection, clamping, alert firing, database recording, and non-re-alerting on subsequent cycles.

Closes #54

@drips-wave

drips-wave Bot commented Oct 18, 2026

Copy link
Copy Markdown

@kelvinokwudili52-stack Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@DevTobis

Copy link
Copy Markdown
Owner

Rebased onto main and slimmed. Main already clamps to the RPC's oldest ledger via getHealth(), returns skippedFrom/skippedTo and fires event_gap_skipped, so the oldestLedger passthrough, the indexer_gap alert and the gap-aware getLastIngestedLedger were dropped (they conflicted with the indexer cursor and duplicated the alert). Migration renumbered 0009 -> 0013. Kept: the chain_event_gap table, plus pollOnce now records the skipped range there (idempotent on the range index) with tests.

@DevTobis
DevTobis merged commit a432aaf into DevTobis:main Oct 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Detect and alert on gaps in indexed ledgers using the RPC's oldestLedger

2 participants