fix(mongodb): page snapshot by _id keyset windows

[olucasandrade]

Plan

.claude/fix-plans/mongodb-snapshot-keyset-streaming-20260924-212132.md

Problem

MongoSnapshot.fetchDocs appended every document into []bson.Raw before any row was processed. Package comments claimed keyset discipline, but there was no pagination — a large collection re-snapshot could OOM the process mid-recovery (InvalidResumeToken → outage).

Introduced by

bbd2b0d

Implementation

• Replace full-collection fetchDocs with _id keyset windows ($gt + Limit), default batch size 256 (matches Postgres backfill).
• Process each window (watermark ShouldEmit + append) before fetching the next; peak buffering is one window.
• Optional CheckpointStore progress keys {sourceID}:{coll}:snapshot persist the last scanned _id after each completed window; cleared on snapshot_complete. Distinct from change-stream resume tokens; empty progress is never treated as a resume-token Save.
• Wire SetProgressStore(ckStore) from the Mongo re-snapshot path in internal/cmd/mongo.go.
• findFn still returns []bson.Raw per call; callers pass keyset filter + limit via opts.

Validation

• CGO_ENABLED=0 go test ./internal/source/mongodb -count=1 (pass)
• New tests: multi-window consumption (findFn called >1×, peak ≤ batch); crash mid-snapshot resumes after stored _id.

Residual risk

• Mid-window crash re-emits docs from the incomplete window (at-least-once; EventLog IdempotencyKey dedup).
• Non-comparable / heterogeneous _id types across a collection can break $gt ordering (same class of risk as any Mongo keyset scan).
• Progress Save frequency is per window; very small windows increase checkpoint write load.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 315d07c1-2b8b-4210-b70e-1c0f240bae33

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cloudflare-workers-and-pages]

Deploying kaptanto with    Cloudflare Pages

Latest commit:	3fa217f
Status:	✅  Deploy successful!
Preview URL:	https://a0dcce87.kaptanto.pages.dev
Branch Preview URL:	https://fix-mongodb-snapshot-keyset.kaptanto.pages.dev

View logs