perf: speed up SMTP ingestion hot path#10
Merged
Conversation
The EnvFilter fallback hardcoded "hedwig=info", which silently overrode log.level from the config file (with_env_filter replaces the with_max_level setting entirely). A server configured with level = "warn" still emitted a formatted INFO line per delivered message, which costs measurable CPU at high ingestion rates. Derive the fallback filter from the configured level instead. The HEDWIG_LOG_LEVEL environment variable still takes precedence.
Formatting-only cleanup left behind by earlier commits; no code changes.
extract_domain_from_path ran mailparse::addrparse plus a pest grammar parse (email_address_parser) twice per message - once for MAIL FROM and once for RCPT TO - just to pull out the lowercased domain for filter matching. Profiling at 73k msg/s showed ~4% of total CPU inside the pest parser alone. The SMTP command parser has already validated the address syntax by the time these callbacks run, so parsing the full address grammar again is redundant. Split on the last '@' instead, which also handles quoted local parts containing '@'. Filter semantics are unchanged: paths without a domain still return None and are handled by the existing allow/deny logic.
get, get_meta, delete and delete_meta all issued a stat (statx) before the actual read or unlink, doubling the syscall count for every storage operation. delete_meta and get additionally used the synchronous Path::exists, which performs blocking I/O on the async runtime. Perform the read or unlink directly and treat NotFound as the miss case, which is also TOCTOU-safe: the previous check-then-act pattern could still race with a concurrent delete and error out.
put() JSON-serialized the entire StoredEmail, escaping every body byte, and get() paid the matching unescape on read. At 80k msg/s the escape/copy work showed up as several percent of total CPU on both the ingest and delivery paths. Write files as a single-line JSON envelope (message id, from, to, queued_at) behind a HEDWIG1 magic prefix, followed by the message body verbatim. Reads detect the magic and fall back to parsing the legacy whole-file JSON format, so spools written by older versions still replay after an upgrade. Also excludes .meta.json files when listing emails in a status directory; list(Deferred) previously tried to parse metadata files as stored emails.
The DATA receive loop rescanned the entire accumulated buffer for <CRLF>.<CRLF> on every incoming read, making terminator detection O(n^2) in message size. For 64 KiB messages arriving in many TCP segments this showed up as a ~13ms p999 ingestion latency and a throughput cliff relative to smaller messages. Track how much of the buffer has already been searched and resume from there (minus a 4-byte overlap so a terminator split across reads is still found), using memmem for the substring search. The oversize-discard path reuses the same incremental search. Also declares the tokio "time" feature in the smtp crate, which previously only compiled through feature unification with the server crate, and adds end-to-end session tests covering chunked DATA delivery and dot-unstuffing.
The claim that the command parser pre-validates address syntax was
wrong: parse_email_address accepts nearly any bytes inside angle
brackets. With the previous commit, junk addresses that the old
pest parser rejected (multiple bare '@', embedded whitespace,
invalid domain characters) could yield a domain and satisfy an
allow filter that used to reject them as unparsable.
Restore None-for-junk semantics with cheap character-level checks:
local part must be non-empty without whitespace/control characters
(bare '@' only inside a quoted string), and the domain must be
dot-separated alphanumeric/hyphen labels. Display-name forms
("Name <user@example.com>") are handled explicitly, which the
first cut silently only passed by accident.
Also adds a 5000-case differential test asserting the incremental
DATA-terminator scan agrees with a from-scratch scan of the whole
buffer across random buffers and read chunkings, plus unit tests
pinning valid/junk address shapes.
Deploying hedwig with
|
| Latest commit: |
23e5822
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://4516bbee.hedwig-1mq.pages.dev |
| Branch Preview URL: | https://perf-ingest-hot-path.hedwig-1mq.pages.dev |
Owner
Author
|
Follow-up pushed after a closer confidence review of the parser-adjacent changes (cde795e):
164 tests passing; 1 KiB throughput unchanged (94.0k msg/s, within noise). |
External review (codex) found two filter regressions in the cheap
domain extraction:
- Valid quoted local parts with whitespace ("blocked user"@x) and
internationalized domains returned None, so deny rules that
previously matched them were bypassed.
- Malformed dot-atoms (foo..bar@, .foo@, foo.@) that the old pest
parser rejected produced a domain and could satisfy allow rules.
Hand-rolled validation cannot cheaply reproduce the full grammar,
so stop trying: the fast path now accepts only strict dot-atom
local parts with LDH domain labels - a conservative subset of the
full parser - and everything else falls back to the original
mailparse + email_address_parser code path. Filter semantics are
therefore identical to the pre-change code for every input, while
typical traffic still skips the expensive parse.
Adds regression tests for the reported bypasses and a differential
test asserting the fast path always agrees with the full parser
whenever it produces a domain. Throughput at 1 KiB is unchanged
(94k msg/s).
Owner
Author
|
External review (codex cli, gpt-5.6-sol) on the two riskiest changes: DATA scanner: no issues found. Overlap math, shed path, size-limit interactions, and multi-message state resets all verified; the 5000-case differential test passed. Domain extraction: two confirmed filter regressions, fixed in 23e5822:
Resolution: gave up on hand-rolled grammar parity. The cheap check is now a conservative fast path (strict dot-atom + LDH labels — a provable subset of the full parser) and anything unusual falls back to the original mailparse+pest path, making filter semantics byte-for-byte identical to main for every input. A differential test enforces that the fast path agrees with the full parser whenever it fires. Throughput unchanged (94k msg/s at 1 KiB). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Profiling ingestion under load (16 connections, 1 KiB messages, fs storage, outbound disabled) showed the server spending most of its CPU on per-message overhead rather than protocol work: a log-level config bug forcing a formatted INFO line per message, a full pest grammar parse just to extract filter domains, doubled syscalls in fs storage, JSON-escaping entire message bodies in the spool, and an O(n²) rescan of the DATA buffer on every TCP read.
Changes (one commit each)
EnvFilterfallback hardcodedhedwig=info, silently overridinglog.levelfrom config.extract_domain_from_pathran mailparse + a pest grammar parse twice per message; the command parser has already validated syntax, so split on the last@instead.get/get_meta/delete/delete_metastat'd before acting (two of them with blockingPath::existson the async runtime); act directly and treatNotFoundas the miss case (also TOCTOU-safe).HEDWIG1magic prefix followed by the message bytes verbatim, instead of JSON-escaping the whole body on write and unescaping on read. Legacy whole-file-JSON spools are still readable, so existing queues replay after upgrade. Downgrading with new-format files queued is not supported. Also fixeslist()trying to parse.meta.jsonfiles as emails.memmem. Adds the first end-to-end session tests (chunked DATA delivery, split terminator, dot-unstuffing) and declares the missingtokio/timefeature in the smtp crate.The store-and-forward design is unchanged: every message is durably written to the spool before
250 OK, disk remains the source of truth for the queue, and the job channel still carries only IDs.Benchmarks
Loopback, 16 connections, 10s runs, fs storage on tmpfs, outbound disabled, release build (
smtp_bench: minimal std-only SMTP load client, one full transaction per message):The large-body tail collapse comes from the terminator scan fix; the throughput gains stack across all five commits (per-commit deltas at 1 KiB: +7% log fix, +4% domain extraction, +16% stat removal, ~0% spool format, +4% scan fix — the spool format's win grows with body size).
Verification
cargo test --workspace(161 tests),cargo clippy --workspace --all-targets,cargo fmt— clean on every commit.metrics::tests::email_counters_incrementis flaky in parallel runs on main as well (global counter race); passes in isolation.