Skip to content

perf(ingest): reduce database work and buffer MQTT bursts - #179

Merged
MrAlders0n merged 1 commit into
devfrom
perf/ingest-throughput
Sep 28, 2026
Merged

MrAlders0n merged 1 commit into
devfrom
perf/ingest-throughput

Conversation

@MrAlders0n

Copy link
Copy Markdown
Member

Changes

  • Move packet processing out of the MQTT callback into bounded workers, keeping messages from each observer ordered.
  • Cache successful capability writes and skip endpoint lookups for suppressed copies.
  • Log queue overflow, release rejected MQTT messages, and drain workers before the final presence flush.

Validation

  • Build, formatting, vet, full tests with PostgreSQL, and ingest/presence/hub race checks passed.
  • Local load tests delivered all 15,000 live hearings and stored all 5,000 observations at 2,000 MQTT messages/sec.
  • With added database latency, the baseline delivered 460/3,000 hearings; this candidate delivered all 3,000 without drops or reconnects.
  • Initial changes reduced Beacon CPU by 21% in the original synthetic workload. In a separate mixed workload, skipping unused endpoint lookups reduced CPU another 6% against the prior candidate (three-run medians).

Buffers remain bounded. Sustained overload can still drop messages; this does not add durable delivery. No API or schema changes.

@MrAlders0n
MrAlders0n requested a review from 446564 as a code owner September 28, 2026 02:49
@MrAlders0n
MrAlders0n merged commit 1887fcc into dev Sep 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant