Repository navigation
Memory limit from the container, streamed exports, trigram search index - #4
Merged
Merged
Conversation
…ch index - At start, read the cgroup memory limit and keep the Go heap at 80% of it (GOMEMLIMIT still wins). The 256 MB container used to hit its limit thousands of times during a large export and get throttled. - Exports write each file straight to the response: flat and nested JSON are generated incrementally (byte-identical to encoding/json, tested), PO, Android, iOS and .properties write through a buffer. - Search: FTS5 trigram index with one row per key (name, description, every text). Writes only mark keys dirty; the index catches up in the background, and searches check dirty keys directly meanwhile, so results are always exact. Matches are confirmed with a Unicode case-insensitive contains (LIKE folds ASCII only, so 'AÇÃO' now finds 'ação'). - Migration 3 indexes existing keys on first start. Platify (160k keys, 5 languages) on the dev server: editor page 30 ms, search 127 ms (was 2.5 s), full ZIP export 6.6 s (was 19 s), 91 MB peak RAM; the search index adds 54 MB to the database.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A benchmark against Tolgee with the real Platify project (160k keys, 5 languages) showed LowLex losing on text search and on the full ZIP export. Both came from the 256 MB container: the Go heap grew to the limit (4,907
memory.events maxduring one export) and the kernel throttled the process.Changes
GOMEMLIMITstill wins. Logged asmemory limit 256 MB: keeping the heap under 204 MB.formats.Writewrites each file straight to the response or ZIP entry. Flat and nested JSON are generated incrementally and are byte-identical toencoding/json(new test covers quotes, control characters, U+2028/9, invalid UTF-8, and keys that are both a text and a prefix likea/a.b).lowlex_contains, a Unicode case-insensitive substring test (SQLite's LIKE folds ASCII only). One- and two-character terms keep the LIKE scan.Benchmark on the dev server (same tool, same Tolgee, before → after)
LowLex peak RAM during the run: 91 MB. The index adds 54 MB to the Platify database (130 → 184 MB) and took ~65 s to build after the upgrade. Locally (load test, 160k keys): search 31–53 ms once the index is idle.