feat(duckdbservice): set late_materialization_max_rows=6000 on all workers - #1018
Merged
Conversation
…rkers Apply SET GLOBAL late_materialization_max_rows = 6000 when a worker opens its DuckDB pair. The setting is session-scoped (LOCAL) in DuckDB, so SET GLOBAL is required for the value to reach every session connection on the shared connector; a plain SET would only affect the warmup connection.
Test Impact PlanDeterministic summary of how this PR changes tests, CI runners, and coverage-risk signals. Summary
Signals
Coverage risk: neutral or increased No coverage-reduction warnings detected. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Sets
late_materialization_max_rows = 6000on every DuckDB worker, applied inOpenDuckDBPair(the worker-process DB factory induckdbservice/duckdb_pair.go), so it covers all workers regardless of how they were spawned (local process pools and K8s pods both go through this path).Why SET GLOBAL
late_materialization_max_rowsis session-scoped (LOCAL) in DuckDB 1.5.x — verified against the bundled bindings:SETon the warmup connection: fresh connection on the same connector still sees the default (50)SET GLOBAL: fresh connection sees the new valueduckdb_settings()reportsscope = LOCALSo a one-time
SET GLOBALat open is what propagates the cap to every session-pool connection serving user queries. Users can still override it per session if needed.Test
TestOpenDuckDBPairSetsLateMaterializationMaxRows(TDD: failed with50before the change, passes after) opens a real worker pair and asserts the value from a fresh connector connection — the same shape as a session-pool connection — so it also guards the SET GLOBAL scope assumption.Verification
go test ./duckdbservice/— passgo vet,gofmt— cleanjust lintcould not run locally (golangci-lint not installed); relies on CI