You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(indexer): detect and alert on ledger gaps using oldestLedger - #1041
When the indexer resumes polling after an outage or downtime longer than Stellar RPC's retention window (~7 days), startLedger can fall behind oldestLedger. Previously, GetEventsResult only passed through events and latestLedger, leaving unrecoverable gaps untracked and risking false reconciliation breaks.
This change:
Passes through oldestLedger from the RPC getEvents response in services/indexer/src/rpc.ts.
Adds chain_event_gap table and migration in @ballast/db (packages/db/src/schema.ts) to store lost ledger ranges (from_ledger, to_ledger) for future data-lake backfills.
Adds indexer_gap alert type to @ballast/observability and sets cooldown per gap range.
Updates pollOnce in services/indexer/src/poller.ts to:
Detect when startLedger < oldestLedger.
Clamp startLedger to oldestLedger.
Log the lost range with structured logger.warn.
Fire an indexer_gap alert with fromLedger, toLedger, and oldestLedger.
Record the gap in chain_event_gap.
Update getLastIngestedLedger to consider recorded gaps so subsequent polling cycles do not re-alert on the same gap.
Adds unit tests in services/indexer/src/poller.test.ts verifying gap detection, clamping, alert firing, database recording, and non-re-alerting on subsequent cycles.
@kelvinokwudili52-stack Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.
You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀
Rebased onto main and slimmed. Main already clamps to the RPC's oldest ledger via getHealth(), returns skippedFrom/skippedTo and fires event_gap_skipped, so the oldestLedger passthrough, the indexer_gap alert and the gap-aware getLastIngestedLedger were dropped (they conflicted with the indexer cursor and duplicated the alert). Migration renumbered 0009 -> 0013. Kept: the chain_event_gap table, plus pollOnce now records the skipped range there (idempotent on the range index) with tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the indexer resumes polling after an outage or downtime longer than Stellar RPC's retention window (~7 days),
startLedgercan fall behindoldestLedger. Previously,GetEventsResultonly passed througheventsandlatestLedger, leaving unrecoverable gaps untracked and risking false reconciliation breaks.This change:
oldestLedgerfrom the RPCgetEventsresponse inservices/indexer/src/rpc.ts.chain_event_gaptable and migration in@ballast/db(packages/db/src/schema.ts) to store lost ledger ranges (from_ledger,to_ledger) for future data-lake backfills.indexer_gapalert type to@ballast/observabilityand sets cooldown per gap range.pollOnceinservices/indexer/src/poller.tsto:startLedger < oldestLedger.startLedgertooldestLedger.logger.warn.indexer_gapalert withfromLedger,toLedger, andoldestLedger.chain_event_gap.getLastIngestedLedgerto consider recorded gaps so subsequent polling cycles do not re-alert on the same gap.services/indexer/src/poller.test.tsverifying gap detection, clamping, alert firing, database recording, and non-re-alerting on subsequent cycles.Closes #54