perf: CPU optimization for log ingestion pipeline - #3
Merged
Conversation
Phase 1 of CPU optimization. SSE stream previously called processLogBatch for every single incoming message, causing massive DB overhead. Now: - New LogBuffer class accumulates SSE messages - Flushes every 2 seconds or at 50 logs (whichever first) - Force-flush at 500 logs (overflow protection) - Profile state updates throttled to 10s intervals - Removes per-log DB read-then-write for profile state - Expected 10-50x reduction in DB ops for SSE-sourced logs
Phase 2 of CPU optimization:
- buildEventHash: replace JSON.stringify(fullObject) with fast field
concatenation using pipe-delimited stable fields — avoids serializing
the entire log object structure
- New DedupCache: in-memory LRU Set per profile (10K entries)
- O(1) Set membership check before hitting DB
- Only queries DB for hashes not in cache (near-zero DB dedup
queries for active streams)
- Auto-evicts oldest entries when over capacity
- Safe: empty cache on restart falls back to DB queries
Phase 3 of CPU optimization: - Device upserts: replaced per-device individual UPDATE/INSERT loop with single batch INSERT ... ON CONFLICT UPDATE using SQL expressions. Uses COALESCE(NULLIF(...)) to preserve existing non-null values and GREATEST() for lastSeenAt. Reduces N DB queries per batch to 1. - Poller interval: increased default from 30s to 300s (5 minutes). SSE stream now provides real-time data; poller serves as fallback safety-net only. Users can override via POLL_INTERVAL_SECONDS env var.
github-actions Bot
pushed a commit
that referenced
this pull request
Apr 18, 2026
## 1.0.0 (2026-04-18) ### Features * add activity detection and profile management services ([d9ba5fa](d9ba5fa)) * add alert system with auto-tagging and webhook notifications ([b687b37](b687b37)) * add application layout with sidebar navigation and switchers ([7fabee1](7fabee1)) * add brand logo/favicon and use across sidebar, layout, and README ([97639b2](97639b2)) * add dashboard overview widgets and stat cards ([c2c1470](c2c1470)) * add dashboard pages for all application views ([93ab668](93ab668)) * add database layer with Drizzle ORM and PostgreSQL schema ([7dff5ad](7dff5ad)) * add DNS log ingestion engine with real-time processing ([10f7e97](10f7e97)) * add NextDNS API client, types, and core utilities ([b100338](b100338)) * add REST API routes for all application endpoints ([40e6262](40e6262)) * add reusable chart components for data visualization ([e9c9b82](e9c9b82)) * add root layout and global stylesheet ([2884fc2](2884fc2)) * add semantic-release for automated versioning and changelogs ([3b52bc2](3b52bc2)) * add session-based authentication with login/logout flow ([0b8f826](0b8f826)) * add shadcn/ui component library primitives ([290d9a5](290d9a5)) * add Zustand dashboard store for client-side state ([444021a](444021a)) ### Bug Fixes * add loading states and cursor-pointer to all dashboard pages ([7d0d013](7d0d013)) * address PR review feedback for semantic-release setup ([245628a](245628a)) * **ci:** add Node.js 22 for semantic-release compatibility ([02b21ee](02b21ee)) * correct DATABASE_URL port and host in .env.example ([8677a22](8677a22)) * create .next/cache dir with correct ownership for non-root user ([97a0b16](97a0b16)) * improve UX with cursor pointer on interactive elements and delete loading state ([e1d03fa](e1d03fa)) * point dashboard empty state to /profiles instead of non-existent /settings ([eff8d0c](eff8d0c)) * prevent pino-pretty crash in Docker standalone builds ([d01f5f7](d01f5f7)), closes [#4](#4) * sync bun.lock with package.json ([f4f0c2a](f4f0c2a)) * trigger docker workflow on semantic-release tags ([d0638af](d0638af)) ### Performance Improvements * CPU optimization for log ingestion pipeline ([#3](#3)) ([c1abe62](c1abe62))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
3-phase CPU optimization for the log ingestion pipeline — reduces DB operations by 10-50x during active SSE streaming.
Phase 1: SSE Batching
LogBufferclass accumulates SSE messages, flushes every 2s or 50 logsPhase 2: Hash & Dedup Cache
buildEventHash()uses fast field concatenation instead ofJSON.stringifyDedupCache(10K entries/profile) — O(1) lookup, skips DB dedup queries for known hashesPhase 3: Batch Device Upserts + Poller Reduction
INSERT ... ON CONFLICT UPDATEper batch instead of N individual queriesFiles Changed
src/lib/ingestion/log-buffer.ts(new)src/lib/ingestion/dedup-cache.ts(new)src/lib/ingestion/sse-streamer.tssrc/lib/ingestion/log-processor.tssrc/lib/ingestion/ingestion-manager.tsCHANGELOG.md(new)Test Plan
tsc --noEmit)