Skip to content

Read Rabin-Karp input in bounded blocks - #231

Merged
vrtmrz merged 7 commits into
mainfrom
perf/stream-rabin-karp-input
Nov 20, 2026
Merged

vrtmrz merged 7 commits into
mainfrom
perf/stream-rabin-karp-input

Conversation

@vrtmrz

@vrtmrz vrtmrz commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Read large Blobs in blocks of up to 1 MiB so the V3 Rabin-Karp splitter can produce Chunks as the input is read.

Empty Blobs finish without a read. For the common case of small Markdown notes, a Blob that fits in one block is read directly with arrayBuffer() to avoid creating a slice and a block iterator.

The rolling fingerprint and UTF-8 boundary checks continue across block boundaries. The splitter only reads another range when it needs more input. Commonlib pins the published octagonal-wheels@0.1.57 for iterateBlobBlocks.

Verification

  • Installed the updated lockfile with the reviewed npm 11.18.0 CLI.
  • Passed the Commonlib type check, 4,212 unit tests, 10 boundary tests, and 31 release checks.
  • Built and installed the packed Commonlib in an isolated consumer and passed the package boundary, bundle, and storage-path checks.
  • A Chromium Worker benchmark over 5,000 synthetic Markdown files (1–16 KiB) took a median of 5.76 seconds with direct reads, compared with 6.40 seconds at the previous PR head. Each version ran three times.

@vrtmrz
vrtmrz merged commit 4b129f1 into main Nov 20, 2026
3 checks passed
@vrtmrz
vrtmrz deleted the perf/stream-rabin-karp-input branch November 20, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant