feat(kalshi): opt-in parallel backfill passes - #303
Merged
Conversation
The trades / markets / events backfill passes run sequentially by default, which stretches one full cycle to ~10-15min once all three are walking (vs ~3min for trades alone on the previous single-pass shape). The passes are HTTP-bound and write to independent CH tables + cursor scopes, so concurrency is safe. Adds `KALSHI_BACKFILL_PARALLEL=true` opt-in. When set, the three passes run via `Promise.allSettled` within one cycle. A failure in one pass no longer cancels the others — partial-cycle progress is durable per-pass — and the first failure propagates with its pass label intact for cycle-error attribution. Default stays sequential so behavior is unchanged for existing deployments until operators explicitly opt in. Trade-off in concurrent mode: the BatchInsertQueue's `lastFlushError` health is module-level, so one pass's flush failure forces the others' next-page health check to abort too. That's "fail safe" — overly eager rather than silently dropping rows — and worth the simplicity vs per-pass queue isolation. `runPasses()` is now exported so the parallel/sequential scheduling can be unit-tested directly. Tests assert: - Parallel kicks off all three passes before any completes. - Sequential runs strictly one-at-a-time. - Parallel + one pass failing: other passes still complete. - Sequential + one pass failing: subsequent passes halted. - Errors carry the failing pass label. - Drained / poisoned short-circuits apply in both modes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The trades / markets / events backfill passes currently run sequentially within one cycle, which stretches one full cycle to ~10–15min once all three are walking (vs ~3min for trades alone on the previous single-pass shape). The passes are HTTP-bound and write to independent CH tables + cursor scopes, so concurrency is safe.
Adds
KALSHI_BACKFILL_PARALLEL=trueopt-in. When set, the three passes run viaPromise.allSettledwithin one cycle. Default stays sequential — opt in only after observing parallel cycles healthy.Trade-off
The
BatchInsertQueuelastFlushErrorhealth flag is module-level, so in concurrent mode one pass's flush failure forces the others' next-page!queue.isHealthy()check to abort too. That's "fail safe" — overly eager rather than silently dropping rows — and the simpler design vs per-pass queue isolation.Code references
services/kalshi/backfill.ts:71—isParallelEnabled()env-var reader.services/kalshi/backfill.ts:146—runPasses()extracted + exported. Switches onparallelbetween sequentialfor awaitandPromise.allSettled.services/kalshi/backfill.ts:195—allSettledfailure aggregation: first rejection rethrows with its pass label intact, additional failures are logged.services/kalshi/backfill.test.ts:660onward — 9 new tests covering scheduling, error propagation, and sentinel short-circuits in both modes..env.example:48— documents the newKALSHI_BACKFILL_PARALLELflag.🤖 Generated with Claude Code