feat: set transaction isolation to REPEATABLE_READ in DatabaseContext… - #164
Open
mabduelaziz wants to merge 1 commit into
Open
feat: set transaction isolation to REPEATABLE_READ in DatabaseContext…#164mabduelaziz wants to merge 1 commit into
mabduelaziz wants to merge 1 commit into
Conversation
…2 to ensure consistent snapshot reads during data dumps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OSMOSIS apidb read task: modify transaction isolation level from READ COMMITTED to REPEATABLE READ
Background
We are trying to use osmosis as a tool to dump a PBF snapshot file used by routing engines. But we are seeing issues with dangling references in the dump. We have narrowed down the issue to the apidb read task reading
nodes,ways, andrelationsin sequence within a single transaction. Under PostgreSQL's defaultREAD COMMITTED, each statement gets its own snapshot. If a writer commits between the nodes scan and the ways scan, the dump can include a way that references nodes it never read, producing a PBF with dangling references that downstream consumers reject.Fix
The apidb read task opens a transaction at
REPEATABLE READand holds it for the full dump. The whole transaction sees one snapshot, so nodes and ways are read from a consistent point in time — no dangling references possible.Since READ_COMMITTED and REPEATABLE READ don't use locks on PostgreSQL, this change doesn't require any additional locks.
Outcomes under concurrent writes
Under concurrent writes, the dump either includes both nodes and ways or orphan nodes. It is not possible to have dangling references. This is valid OSM data.
Notes
REPEATABLE READuses MVCC).