Let a deployment bound what DuckDB holds

[skearnes]

Summary

Corpus created its DuckDB connection with no configuration, so a caller could not set memory_limit without reaching into _connection. In a container that is not a small gap: DuckDB reads the cgroup, so it sizes its default limit from the container's own cap rather than the host's — but that limit bounds DuckDB's buffers, not the process, and the default claims about 80% of the cap for them. The occurrence index build holds up to 2.8 GB beside that accounting, so the default is over the cap by construction, and reaching a cap is a kill: no exception, no log line, exit 137.

Measured in a container over the full corpus (2,428,291 reactions), calling check_index():

cap	memory_limit	outcome
4 GiB	3.1 GiB (DuckDB's default)	OOM-killed, exit 137
4 GiB	2 GiB	OOM-killed, exit 137
4 GiB	1 GiB	OutOfMemoryException at 195s, peak RSS 3.91 GiB
8 GiB	6.3 GiB (DuckDB's default)	OOM-killed at 85s
12 GiB	5 GiB	OutOfMemoryException at 194s, peak RSS 8.65 GiB
12 GiB	6500MiB	18,847,978 occurrences, peak RSS 9.29 GiB

The catchable exception check_index() documents only arrives when the limit leaves room; the docstring promised it unconditionally.

Changes

• Corpus(..., memory_limit=...) passes the setting to duckdb.connect.
• _warn_when_the_cap_leaves_no_headroom reads the cgroup at open (v2, then v1, treating max and the top-of-range sentinel as no cap) and warns when it leaves less than 4 GB above the limit.
• check_index and the README's deployment section say what a cap has to leave, and the floor reads 5–6.5 GB rather than 5 GB — 5 GiB raised where 6500MiB finished.

Testing

uv run pytest -n auto — 1189 passed, 2 skipped.

Eleven new tests cover the setting reaching DuckDB, the size parser, max and the v1 sentinel reading as no cap, and the warning firing on a cap with no headroom while staying quiet on one with headroom and on a machine with no cgroup to read.

End to end in an 8 GiB container, the configuration that was killed above:

WARNING execute.py:547: this container may hold 8.0 GB and DuckDB's memory_limit is 6.3 GB,
leaving 1.7 GB for everything else the process holds, which building the occurrence index
has been measured to want 4.0 GB of. Reaching the cap is a kill rather than an exception,
so open the Corpus with a memory_limit that leaves that much, or give the container more.

Notes

The default is unchanged: no memory_limit means DuckDB decides, as before. The warning is the only new behavior on that path, and it fires only where a cgroup cap is readable.

Wall-clock figures above are inflated — the container read the artifacts and spilled through a bind mount. Resident sizes, which are what the change is about, are unaffected.

🤖 Generated with Claude Code

Greptile Summary

The PR lets callers configure DuckDB's memory limit and adds deployment guidance for budgeting memory under container caps.

• Passes an optional memory_limit through Corpus to DuckDB.
• Detects effective caps across ancestor cgroups and warns when insufficient process headroom remains.
• Adds focused parsing, cap-detection, warning, and configuration tests.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
ord_schema/search/execute.py	Adds configurable DuckDB memory limits, cgroup-cap detection, and a headroom warning during Corpus initialization.
ord_schema/search/execute_test.py	Adds unit coverage for memory-limit propagation, setting parsing, ancestor cap selection, and warning behavior.
ord_schema/search/README.md	Documents measured memory requirements and recommended container memory-limit configuration.

_{Reviews (2): Last reviewed commit: "Read the cgroup this process is in, not ..." | Re-trigger Greptile}