Memory Optimization Proposal: Implementing Stream-based Context Ingestion to avoid OOM

[professional-slacker]

Currently, the agent loads entire codebases into the V8 heap as monolithic strings. This leads to FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory during large string operations (confirmed in Runtime_StringSplit).
crash.txt

Instead of monolithic loading, I propose a stream-aware ingestion layer (a "gatekeeper" proxy at the stdin/file stream level).
This is Stream-aware Gateway.
This would:
Control data granularity before it hits the Node.js runtime.

Maintain a constant memory footprint regardless of codebase size.

Allow for more sophisticated context management (e.g., chunking code by semantic units/AST boundaries).

I understand this requires refactoring the context flow, but I believe it's a necessary step for scaling to massive codebases. Happy to discuss the implementation details further.

Regars,

[kevincodex1]

this is interesting lets dig deeper on this

[professional-slacker]

I tried a simple comparison on Linux.
makedummyfile.js
bench.js

$ node makedummyfile.js

Running this generates a 500MB large_dummy.txt file (uses dd on Linux; Windows would use fsutil, though it hasn’t been tested there).

$ node bench.js
--- Benchmark Start ---
[Brute Force] Time: 417.59ms, Mem: 500.00MB
[Stream] Time: 1684.03ms, Mem: 3.83MB

As you can see, using streams drastically reduces memory usage,
but naturally, the processing becomes slower.

This could be mitigated to some extent by adjusting the amount of data sent to Node.js,
or by using a wrapper implemented in a native language that doesn’t depend on the Node runtime,
which may offer better performance.

I’m not sure where exactly this mechanism should be integrated within openclaude.

[kevincodex1]

this looks cool will check this out

[professional-slacker]

Follow-up posted here:
#402 (comment)