Skip to content

fix: size compressed stream reads for LZW expansion; issue remainders (#21, #18, #16) - #23

Merged
canesin merged 5 commits into
mainfrom
fix/stream-budgets-docs
Sep 25, 2026
Merged

canesin merged 5 commits into
mainfrom
fix/stream-budgets-docs

Conversation

@canesin

@canesin canesin commented Sep 25, 2026 •

Copy link
Copy Markdown

Description

Closes the remainder of the open issues after v0.8.0 (#22).

Fixes #21. Fixes #18.

How has this been tested?

RED before the fix — TestReadStreamAcceptsIncompressibleCompressedState, a 32.7 MB state (20,000 nodes with 512 random meta bytes plus 20 MiB random user state) compressed to 43.7 MB:

net_test.go:1414: compressed stream of 43704906 bytes (32680445 decompressed): unexpected error: compress.Buf: msgpack: truncated input

GREEN after. TestReadStreamBoundsPlainMessage still refuses a >40 MiB plaintext message. The large-state test skips under the race detector (no concurrency; 16 s there).

Review loop: round 1 (independent reviewer) verified the LZW bound (Go's writer: ≤ 1.5N + 1.5N/3838 + 5 bytes; 40 MiB → ≤ 62.93 MB ≤ maxCompressedStreamBytes 63.02 MB; measured worst practical input 1.409×) and every stream path, and requested the discriminating #21 tests and the encrypted sender limit above. Round 2 requested an exact-boundary sender test, a plaintext cap edge test (TestReadStreamPlaintextCapEdge: exactly 40 MiB accepted, +1 refused) and compress-only-when-smaller. Mutation checks: sender limit on plaintext length, >= for >, plaintext cap +1 and always-compress each fail their test. Round 3 (verdict MERGE) confirmed every stream form and upstream interop (upstream accepts plaintext streams whatever its compression setting) and raised three Low findings, fixed: the initiator's bare EOF, the inconsistent error identity, and no test pinning maxCompressedStreamBytes to the LZW worst case (TestCompressedStreamBudgetCoversLZWWorstCase: 62,930,968 needed ≤ 63,021,056; mutating the budget to 45 MB or 1.5N+1 fails it). Mutating out the error response or the user-state check fails its test. Round 4 (verdict MERGE, delta review) verified nothing is written before any ErrMessageTooLarge return, that upstream initiators already handle errMsg, and the LZW arithmetic against compress/lzw (clear every 3,838 data codes; comment and test corrected); README notes that Mori refuses to send user state above 20 MiB, which upstream receivers accept. make ci passes (fmt, go fix, vet, lint, cross, deps, tidy, tests on amd64 and 386, race, bench smoke).

Checklist

  • make ci passes.
  • Behavior changes have tests, docs (SECURITY.md bounds table) and a CHANGELOG entry under Unreleased.

- readStream bounds a compressed stream message by the largest valid LZW
  form of a maxDecompressedBytes payload (1.5x plus clear/end codes,
  ~60 MiB) instead of the 40 MiB plaintext cap; plaintext messages keep
  the 40 MiB cap through a limited source. Senders compress regardless of
  gain, so v0.8.0 refused legitimate push/pull states of ~29-40 MiB with
  incompressible content. RED: a 32.7 MB state (20,000 nodes with random
  meta + 20 MiB random user state) compressed to 43.7 MB and failed with
  'compress.Buf: msgpack: truncated input'.
- Regression tests for #21 (compress.Buf and node fields declaring
  multi-gigabyte lengths are refused with bounded allocation) and #16
  (LocalNode reports StateLeft after Leave).
- CONTRIBUTING.md and PR template adapted from upstream hashicorp#375 (#18).

Fixes #21
Fixes #18
- #21 tests declare lengths inside the field limits (60 MiB compress.Buf,
  1 MiB Meta) with per-case allocation budgets; mutating readInto to
  allocate the declared size fails both (62.9 MB and 1.06 MB allocated),
  where the previous version still passed.
- Senders refuse, with ErrMessageTooLarge, encrypted stream messages whose
  ciphertext exceeds the 20 MiB receivers accept (push/pull state or
  SendReliable). RED: SendReliable of 20 MiB returned 'read/write on closed
  pipe' after a 10 s timeout.
- The incompressible-state test skips under the race detector (16 s there,
  no concurrency to check).
- readStream no longer allocates a limited source for encrypted messages;
  PR template uses an absolute CONTRIBUTING link; CHANGELOG range and
  encrypted-limit notes corrected.
@canesin
canesin merged commit e9bdd5a into main Sep 25, 2026
8 checks passed
@canesin
canesin deleted the fix/stream-budgets-docs branch September 25, 2026 21:21
@canesin canesin mentioned this pull request Sep 25, 2026
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant