Skip to content

manifest: key bounds held as int[] make flushing and reopening a large key take tens of seconds #347

Description

@smiggleworth

Found while fixing #249.

Observed

A single committed Delete whose key is the largest admissible length (just under 64 MiB) takes, on an M-series Mac, Release build:

phase time
commit (Sync) 0.17 s
reopen with WAL replay 0.24 s
flush 40.7 s
reopen from the manifest 7.2 s
point get 0.96 s

SstCodec.Encode and Decode of the same one-entry SST take only 0.55 s and 0.34 s, so the SST format is not the cost.

Cause

FileMeta.SmallestKey and LargestKey are int[] (src/Pants.Core/Storage/Internal/FileMeta.cs:19). Every key bound is copied byte by byte with LINQ (LocalDiskStore.StageSst: allKeys[0].Select(value => (int)value).ToArray(), plus about 14 similar conversions). It is then serialized as a JSON number array into the manifest journal, the snapshot, and the manifest.json mirror. A 64 MiB key becomes about 128 M JSON numbers, written several times per flush and parsed again on reopen.

Expected

Flushing and reopening large keys should cost roughly the same as the SST codec. Keep the pinned on-disk manifest representation (Midge compatibility), but:

  • hold key bounds in memory as byte[] and convert only at the serialization boundary
  • serialize with a custom converter that writes straight to the output, instead of building intermediate arrays
  • avoid serializing the same bounds more than once per edit

Acceptance criteria

  • Add an engine-level test to PantsSstEntryAdmissionTests: commit the largest admissible Delete key, flush it, reopen, and read it back. It was dropped from Reject oversized point-delete keys at staging #348 because it takes about 53 s locally and timed out or exhausted memory on the CI runners. It should complete in a few seconds on every OS in CI.
  • Manifest bytes stay byte-for-byte identical, as proven by the existing Midge manifest compatibility fixtures.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:perfPerformance-critical data structure / algorithm parityarea:storageStorage engine internals (flush, hybrid persistence, I/O, recovery)priority:P3Deferred: blocked or intentionally postponed until an explicit prerequisite is met

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions