feat(rds): transparent stale-connection reconnect in RdsServiceGateway - #152
Merged
Conversation
RdsServiceGateway (single-connection mode) held one psycopg2 connection for the life of the process with no liveness check, so a long-running consumer whose connection was dropped server-side would fail its next query with 'connection already closed' — and the except-block rollback would itself throw on the dead socket, masking the original error. Adds reconnect() (rebuilds the single connection or the pool from db_uri) and classifies connection-level errors; get_data and update_database now reconnect and retry once on such an error only. Non-connection errors (IntegrityError, DataError, etc.) keep their exact prior behavior, and rollback calls are guarded so a dead-connection rollback can't mask the cause. update_database commits once at the end of each path, so a whole-operation retry cannot double-write.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features
get_data/update_databasefailed withconnection already closed. Callers now get transparent recovery: on a connection-level error the gateway reconnects and retries the operation once.Fixes
exceptblocks calledconn.rollback(), which itself raises on a dead socket and hid the original failure. Those rollbacks are now guarded againstOperationalError/InterfaceError.Behavior / safety
psycopg2.OperationalError/InterfaceError(or messages like "connection already closed", "server closed the connection", "ssl connection has been closed", "connection not open", "terminating connection") trigger the reconnect+retry. Every otherpsycopg2.Error(IntegrityError, DataError, ProgrammingError, …) keeps its exact prior behavior — no retry.raise_on_errorsemantics are unchanged for non-connection errors.update_databasepath performs all work then commits once at the very end, so a connection-level failure means nothing was committed and re-running the operation cannot double-write.reconnect()rebuilds the single connection (or the pool) from the existingdb_uri; it is a no-op in test mode.Testing
tests/test_connect.py: 160 passed, ruff clean. New tests cover reconnect (single mode + test-mode no-op), connection-error classification,get_datareconnect-and-retry-once, and IntegrityError NOT triggering a reconnect for bothget_dataandupdate_database.Context
End-user impact
No user-facing surface. Operationally: services using RdsServiceGateway across long-lived processes stop failing on transient/idle connection drops.