Skip to content

Reduce ranking spills and repeated maintenance after lease loss - #403

Merged
CountableNewt merged 7 commits into
mainfrom
codex/tsw-114-production
Sep 13, 2026
Merged

CountableNewt merged 7 commits into
mainfrom
codex/tsw-114-production

Conversation

@CountableNewt

@CountableNewt CountableNewt commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Production ranking stopped publishing during repeated Coordinator lease loss. A matched diagnostic window recorded roughly 1.77 GB of candidate-query temporary writes with only 6.16 MB of new WAL. This change reduces candidate spill and prevents maintenance work from repeatedly blocking publication.

  • Materialize compact ranking keys before hydrating selected payloads, preserving the original ordering, moderation, limits and statement snapshot.
  • Preserve graph startup grace/backoff across lease-host replacement. Keep its six-hour successful cadence.
  • Move disposable mention/profile/metadata expiration out of publication into independently paced 500-row batches, preserving active fetches and required live-item metadata.
  • Attribute lease failures using allowlisted service names, query IDs and bounded transaction/query ages; no SQL text or parameters in telemetry.
  • Retain the earlier incremental rollup experiment with both database tracking and runtime activation disabled. Its forced-collision benchmark still fails the CPU gate; this PR does not activate it or claim its savings.

Validation: 342 Wire tests with PostgreSQL integration, 152 OperationsCore tests serially, 11 IndexingWorker tests, and two independent-cleanup runtime tests passed. Candidate legacy-query parity covers 32 baseline/external/language/limit combinations. At 100,000 synthetic rows and unchanged 4 MB work_mem, temporary writes fell from about 479 MB to 18 MB and runtime fell about 45%; buffer lookups increased due to selected-key hydration. Production performance remains a rollout gate. Full parallel Operations tests exposed existing shared-fixture lock contention; serial execution passed.

Rollout: Development first, required CI and service/migration checks, then Production. Keep the 16 GB cap, durability, backup policy and replica counts unchanged while comparing publication freshness, actionable queue age, p95 latency, temporary writes and I/O pressure. The bundled experimental migration installs disabled triggers; verify they remain disabled. Worker/Redis aggregation work is a separate follow-up.

Refs TSW-92, TSW-114.

@linear-code

linear-code Bot commented Sep 13, 2026

Copy link
Copy Markdown

TSW-114

TSW-92

@CountableNewt CountableNewt changed the title Recompute only changed and due Wire signal rollups Reduce ranking spills and repeated maintenance after lease loss Sep 13, 2026
@CountableNewt
CountableNewt marked this pull request as ready for review September 13, 2026 17:50
@CountableNewt
CountableNewt merged commit 48dee4a into main Sep 13, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant