Huge memory usage spike on startup

[katafrakt]

It was reported by some of my colleagues, and after that I noticed it myself, that on a large project during the startup Expert uses large amount of memory in peaks. Talking about memory usage reported by MacOS's Activity Monitor (so not just RAM) we are talking about up to 10GB on the manager node and 2.5GB on Engine node. After doing the whole startup the memory usage goes down to about 1.5GB on manager and similar on Engine.

My investigation traced it to the fact that we index the code in Engine and then we send the whole large list to Manager, which then sends it through couple of GenServers until it's batched and saved to SQLite.

I did a prototype with Codex, batching the index results with backpressure and calling garbage collection in few places aggressively. With that the manager node stayed under 600MB quite consistently. This leads me to believe that this is a direction worth pursuing. The prototype itself was a large change and not without issues on it's own, so I'm first creating this tracking issue, because it will probably be better to split the improvements into few more focused PR. Also I wonder what others think.

[doorgan]

Yes, batching the engine -> manager lists is a good improvement, we should do that
Also in general I think we should take any opportunity we can to reduce memory and disk usage on indexing

[sophiathedev]

@katafrakt @doorgan I had a look at this on main (2afc03c). The full entries list still comes back over RPC and goes through the store and SQLite GenServers before the 4,000-entry batching kicks in, so batching the transfer makes sense to me too. One catch is we can't just feed chunks into the existing write APIs: replace_all clears the whole index, and apply_index_update clears entries by path, so splitting a file across batches could wipe out entries from an earlier batch. I'd start with a focused PR for batched transfer with backpressure, keeping rollback and only committing the manifest once the write succeeds. That would target manager memory first, with streaming on the engine side as a separate follow-up. I haven't reproduced the 10 GB peak or benchmarked this approach yet. Is your prototype available somewhere, @katafrakt? Would be useful to see what issues you ran into and avoid duplicating work.