Stop the audit WAL open path from loading every event into memory - #54
Merged
Conversation
Production ran out of memory in a loop for five hours today: 36 OOM kills between 07:17 and 12:12 UTC, each one at ~3.5 GB anon RSS on a 3.8 GB host, with the server dying one to two minutes after it started listening. The cause is in the WAL open path, not in anything that changed today. Opening the WAL builds an index from event id to the whole decoded event, and keeps it for the life of the process. It exists to answer one question in AppendIdempotent: has this id already been written, and with these exact bytes? Holding the events to answer that makes the index cost several times the file on disk. The production WAL had grown to 510 MB and 996,257 events since 2026-06-16, so the index alone was multiple gigabytes, and the box could no longer boot the server it had been running for weeks. Store a sha256 of the canonical event bytes per id instead. That is exactly what the comparison needed, it is what the chain hash already relies on being deterministic, and it drops the index for the production WAL from gigabytes to roughly 120 MB. Residual, stated rather than hidden: the index is still O(events). At the observed rate of about 15k events a day it reaches a few hundred megabytes within a year, so WAL rotation or retention is still owed. This change buys the headroom to design that instead of doing it during an outage. Tested: a new test builds a WAL with large event bodies and fails if opening it retains more than a quarter of the file in heap; on the old code it retained more than the whole file. A second test pins the behaviour the index exists for, so the cheaper index cannot be bought by dropping replay detection. Full suite and go vet green; audit and store packages green under -race.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
生产今天 OOM 循环了五个小时:07:17 到 12:12 UTC 之间 36 次 OOM kill,每次都在 3.5 GB anon RSS(整机 3.8 GB),服务器监听后一两分钟就被杀。
原因在 WAL 的 open 路径,不是今天任何一个变更。开 WAL 会建一张 id 到整个已解码事件的索引并持有到进程结束,而它只回答
AppendIdempotent里的一个问题:这个 id 是不是已经写过,而且字节完全一样?为了回答这个而把事件本体留在内存里,索引就要花掉磁盘文件的好几倍。生产的 WAL 自 2026-06-16 起已经长到 510 MB、996257 条,索引本身就是几个 GB,机器再也起不来它已经跑了几周的服务。改成每个 id 存一份 canonical 事件字节的 sha256。这正是那次比较需要的东西,也正是 chain hash 已经依赖的确定性,索引在生产这份 WAL 上从几 GB 降到约 120 MB。
残余风险如实写明:索引仍然是 O(事件数)。按实测约每天 1.5 万条,一年内会到几百 MB,所以 WAL 轮转或保留策略仍然欠着。这个改动买的是从容设计它的余地,而不是在故障里做。
测试:新测试构造带大 body 的 WAL,如果 open 之后保留的堆超过文件的四分之一就失败(旧代码保留得比整个文件还多)。第二个测试钉住索引存在的意义,防止用砍掉重放检测来换便宜的索引。全量测试与 go vet 通过,audit 与 store 包在 -race 下通过。