A Backfill replaces the Session or changes nothing (#69) - #77
Open
jerome-queck wants to merge 1 commit into
Open
A Backfill replaces the Session or changes nothing (#69)#77jerome-queck wants to merge 1 commit into
jerome-queck wants to merge 1 commit into
Conversation
`bin/backfill` deleted a Session's records before it ingested, so a run that never reached the ingest left the Session worse off than not running the command at all. On 2026-08-21 that cost 7,238 records of a live capture that cannot be taken again. The delete also swept the catalogue's own `sessions` document, which carries the same `session_key` every record carries, so the 2026 catalogue fell from 121 Sessions to 120 and nothing could even name the Session that had been emptied. What the stores already hold is now marked rather than deleted, and discarded only once the ingest has written the records replacing it. An ingest that does not finish — or that finishes without writing anything, which upstream's insert path does happily, logging the failed write and exiting zero — deletes what it wrote, takes the marks off and leaves the Session as it was. `meetings` and `sessions` are the catalogue's, they say what a Session *is* rather than what it recorded, and they are now outside every statement the command sends. A Backfill over a Session the stores already hold says so with the count before the ingest starts, which is a warning worth printing now that the destruction is the last thing to happen rather than the first: stopping the run costs nothing. ADR-0013 records the reasoning and the two mechanisms that lose to marking — a staging database, which MongoDB cannot swap in without copying the Session across twice, and an `_id` boundary, which upstream's clock-with-a-bump id generator quietly makes unsound. `test/backfill.test.sh` runs the wrapper itself against a JSON store that evaluates the very snippets `bin/lib/mongo.sh` sends, so the interrupted case is asserted where it lives — in the stores — with no MongoDB and no container stack anywhere near it. Assisted-by: Claude Opus 5 (reasoning: high) Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #69.
A Backfill either replaces the Session it names or leaves it exactly as it was. There is no longer
a window in which the Session is neither, and the catalogue is no longer collateral.
What went wrong, and the half nobody expected
bin/backfilldeleted before it ingested, so an ingest that did not run left the Session worse offthan not running the command — 7,238 records of a live capture, gone, on 2026-08-21.
The second half is the one that mattered more: the delete swept
session_keyacross everycollection, and the catalogue's own
sessionsdocument carriessession_keytoo. So the Sessionlost its name along with its data, 2026 fell from 121 catalogued Sessions to 120, and because the
picker builds its list from the catalogue, the Session could not even be shown as known-but-not-here.
meetingsandsessionsarebin/catalogue's (ADR-0009) and are now outside every statementbin/backfillsends, under any ordering.Marking, not staging
The records being replaced are marked
_supersededbefore the ingest and deleted after it, so whatthe ingest wrote is exactly what is not marked and both endings are exact: a finished ingest
discards the marked half, an unfinished one deletes the unmarked half and takes the marks off. The
field is
_-prefixed, which is the prefix upstream's query API strips from every document itreturns, so nothing above the stores can see it.
ADR-0013 argues it against the two alternatives. A staging database has to be copied across rather
than renamed — a million and a half documents moved twice for a job somebody runs at a keyboard. An
_idboundary is free and is a trap: upstream's_idis a millisecond clock bumped forward by oneper document, so a run writing that many documents leaves ids twenty-five minutes ahead of wall
clock and the next run's can start below them.
One thing found on the way:
insert_data_synccatchesBulkWriteError, logs it and returns, so theingest can exit zero having written nothing. The discard is conditioned on finding a record that
arrived, not on the exit status.
Proving it without a MongoDB
test/backfill.test.shnow runsbin/backfillitself — only what is under it is faked, and thefake MongoDB (
test/lib/mongo-store.ts) evaluates the very snippetsbin/lib/mongo.shsendsagainst a JSON store. So the assertions are about the stores rather than about the wrapper's
internals: nothing is deleted before the ingest has run, an interrupted ingest leaves four records
where four records were, the half it wrote is not left behind, the catalogue's rows survive both
endings, and a second Session in the same collections is untouched. The fake reproduces the reported
bug when given the old snippet, so the cases are not vacuous.
Full suite, both typechecks and shellcheck green.
#69 — bin/backfill discards before it ingests: 4 of 4 criteria delivered, nothing beyond the
brief.
Assisted-by: Claude Opus 5 (reasoning: high)
Co-authored-by: Claude noreply@anthropic.com