postgres: serialize tracked schema apply with an advisory lock#1846
Draft
ns-kosti wants to merge 2 commits into
Draft
postgres: serialize tracked schema apply with an advisory lock#1846ns-kosti wants to merge 2 commits into
ns-kosti wants to merge 2 commits into
Conversation
## What Adds an opt-in `track_schema_checksums` intent flag to the OSS Postgres database provider. When enabled, each schema file's checksum is recorded in a `foundation.schema_checksums` table and the file is only re-applied when its contents change, instead of being re-applied on every `ns deploy`. ## Why The provider currently re-applies every `.sql` file on every deploy. Schema files are expected to be idempotent (e.g. `fn_ensure_table`, `CREATE INDEX IF NOT EXISTS`), so this is safe but wasteful. The checksum table lets us skip files that haven't changed. ## Design - **Opt-in, default false** — existing databases are unaffected until they enable it. - **Checksum** = `sha256:` + hex of the raw file bytes. No canonicalization, so any byte change re-applies; this avoids the correctness risk of a SQL-aware normalizer at the cost of an occasional harmless re-run. - **Table** = `foundation.schema_checksums(path TEXT PRIMARY KEY, checksum TEXT, updated_at TIMESTAMPTZ)`. - **Written only after a successful apply**, so `updated_at` stays constant across redeploys when nothing changed. - **Path validation** — empty and duplicate paths are rejected, since `path` is the primary key. - **Single batched read** of stored checksums (`WHERE path = ANY($1)`); reads and writes retry on retryable errors. - **Logs** each schema file it applies (skipped files are silent). ## Caveat The table records that a file's bytes were applied successfully once; it is not an assertion that the live schema matches the file. With the `fn_ensure_*` helpers, editing a definition changes the checksum (so the file re-runs) but the helper may still no-op against an existing object. It's a re-execution cache. ## Tests Unit tests cover checksum determinism, byte-sensitivity, the `sha256:` prefix, and path validation. End-to-end skip/re-apply coverage is tracked in #1845.
When track_schema_checksums is enabled, concurrent deploys of the same database could race while creating the checksum ledger and applying DDL. Pin a single pooled connection and hold a foundation-namespaced session advisory lock for the ledger ensure, checksum read, DDL apply and checksum write, so only one deploy mutates a given database's schema at a time. The lock is session-scoped on a pinned connection to keep DDL non-transactional (e.g. CREATE INDEX CONCURRENTLY remains possible). On unlock failure, or when the acquire result is uncertain, the connection's session is ended so Postgres releases the lock rather than stranding it on a pooled connection. Only the opt-in tracked path takes the lock; default behavior is unchanged.
ns-kosti
marked this pull request as draft
June 24, 2026 13:57
ns-kosti
force-pushed
the
postgres/skip-unchanged-schema
branch
from
June 24, 2026 14:00
09b7528 to
a46ed8e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #1844 (checksum ledger). Targets that branch so the diff is advisory-lock only.
What
When
track_schema_checksumsis enabled, concurrent deploys of the same database could race while creating the checksum ledger and applying DDL. This serializes the tracked schema apply with a Postgres session advisory lock:pg_advisory_lock) across the ledger ensure, checksum read, DDL apply, and checksum write.pg_advisory_xact_lock) so DDL stays non-transactional —CREATE INDEX CONCURRENTLYand similar remain possible.pg_advisory_lockstacks per session, and a dead connection can't be reused. The wait is bounded by context.Notes
Test
go build ./library/oss/...,go vet,gofmtgo test ./library/oss/postgres/prepare/database/helpers/(incl. newTestAdvisoryLockKey)