Found while fixing #249.
Observed
A single committed Delete whose key is the largest admissible length (just under 64 MiB) takes, on an M-series Mac, Release build:
| phase |
time |
| commit (Sync) |
0.17 s |
| reopen with WAL replay |
0.24 s |
| flush |
40.7 s |
| reopen from the manifest |
7.2 s |
| point get |
0.96 s |
SstCodec.Encode and Decode of the same one-entry SST take only 0.55 s and 0.34 s, so the SST format is not the cost.
Cause
FileMeta.SmallestKey and LargestKey are int[] (src/Pants.Core/Storage/Internal/FileMeta.cs:19). Every key bound is copied byte by byte with LINQ (LocalDiskStore.StageSst: allKeys[0].Select(value => (int)value).ToArray(), plus about 14 similar conversions). It is then serialized as a JSON number array into the manifest journal, the snapshot, and the manifest.json mirror. A 64 MiB key becomes about 128 M JSON numbers, written several times per flush and parsed again on reopen.
Expected
Flushing and reopening large keys should cost roughly the same as the SST codec. Keep the pinned on-disk manifest representation (Midge compatibility), but:
- hold key bounds in memory as
byte[] and convert only at the serialization boundary
- serialize with a custom converter that writes straight to the output, instead of building intermediate arrays
- avoid serializing the same bounds more than once per edit
Acceptance criteria
Found while fixing #249.
Observed
A single committed
Deletewhose key is the largest admissible length (just under 64 MiB) takes, on an M-series Mac, Release build:SstCodec.EncodeandDecodeof the same one-entry SST take only 0.55 s and 0.34 s, so the SST format is not the cost.Cause
FileMeta.SmallestKeyandLargestKeyareint[](src/Pants.Core/Storage/Internal/FileMeta.cs:19). Every key bound is copied byte by byte with LINQ (LocalDiskStore.StageSst:allKeys[0].Select(value => (int)value).ToArray(), plus about 14 similar conversions). It is then serialized as a JSON number array into the manifest journal, the snapshot, and themanifest.jsonmirror. A 64 MiB key becomes about 128 M JSON numbers, written several times per flush and parsed again on reopen.Expected
Flushing and reopening large keys should cost roughly the same as the SST codec. Keep the pinned on-disk manifest representation (Midge compatibility), but:
byte[]and convert only at the serialization boundaryAcceptance criteria
PantsSstEntryAdmissionTests: commit the largest admissibleDeletekey, flush it, reopen, and read it back. It was dropped from Reject oversized point-delete keys at staging #348 because it takes about 53 s locally and timed out or exhausted memory on the CI runners. It should complete in a few seconds on every OS in CI.