Skip to content

perf(database): add contact-tag/embedding indexes and orphan-attachment cleanup - #1197

Open
realcodesiman wants to merge 2 commits into
mainfrom
perf/db-indexes-and-attachment-retention
Open

realcodesiman wants to merge 2 commits into
mainfrom
perf/db-indexes-and-attachment-retention

Conversation

@realcodesiman

Copy link
Copy Markdown
Contributor

Implements P0 item A1/A2 and P2 item A8 (partial) from the system-design-review audit.

A1 — ContactToTag index

ContactToTag only had the composite PK (contactId, tagId) — every "contacts with tag X" query (segments, broadcast audience, filters) scanned poorly at scale. Added ContactToTag_tagId_contactId_idx on (tagId, contactId).

A2 — AIEmbedding HNSW index

AIEmbedding.embedding (vector(1536)) only had a btree workspaceId index — similarity search was a full seq scan + cosine per row. Its sibling table AIConversationEmbedding already has an HNSW index; added the equivalent AIEmbedding_embedding_idx (hnsw, vector_cosine_ops).

Both index creations use CREATE INDEX CONCURRENTLY IF NOT EXISTS in the generated migration, and were applied and verified against a real local Postgres (timescale/timescaledb-ha:pg18-all).

A8 — orphaned attachment cleanup (partial)

Message/Attachment are hypertable rows with no FK to their parent (compressed-chunk deletes can't carry an FK) — so attachments can outlive their message. Added apps/worker/src/schedule/handlers/purge-orphaned-attachments.ts, a new scheduled job (guarded by an in-process flag + distributedLock, batched 1000/run) that reconciles orphaned attachment rows. The retention period for Message itself is an explicit business decision, not made here — this only closes the orphan-attachment gap the audit called out as independently actionable.

Verification

  • pnpm --filter @chatbotx.io/database check-types && test && test:db — 729 + 16 integration tests pass against real local Postgres
  • pnpm --filter worker check-types && test — 2342/2342 pass
  • Migration applied cleanly with pnpm --filter @chatbotx.io/database db:migrate against local Postgres

@github-actions github-actions Bot added the improvement Refactor or performance improvement label Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Refactor or performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant