Add Zstandard compression support to @tootallnate/zstd-wasm + CI#38
Merged
Conversation
Rebuild the WASM from upstream zstd 1.5.7's full single-file library
(encoder + decoder) instead of the decoder-only amalgamation, and add
encode-side bindings mirroring the existing decoder API:
- ZstdEncoder low-level streaming encoder (push/finish/encode)
- ZstdCompressStream CompressionStream-shaped TransformStream
- compressBytes one-shot helper
- DEFAULT_COMPRESSION_LEVEL (3); level (1-22) configurable everywhere
The C wrapper gains cctx_new/free, cctx_set_level, compress_stream
(ZSTD_compressStream2 with an endOp directive), and cstream_{in,out}_size.
The build still uses wasi-sdk with no WASI imports; multithreading is
disabled and error strings stripped to match the prior build.
Also add a CI workflow (.github/workflows/ci.yml) that builds and tests
the monorepo on pushes and PRs. It installs the system `zstd` binary
(the new tests cross-validate WASM output against it) and relies on the
checked-in WASM binaries, matching the release workflow's setup.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 34845d3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Both packages declared a `test` script (`vitest run`) but ship no test files, so vitest exited 1 and broke the new monorepo CI. Add `--passWithNoTests`, matching the existing ffmpeg-wasm convention, so turbo's `test` task is a no-op until real tests are added.
TooTallNate
changed the title
@tootallnate/zstd-wasm + CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds compression support to
@tootallnate/zstd-wasm(previously decode-only) and introduces a CI workflow that builds and tests the monorepo.Compression
combine.py, still compiled with wasi-sdk with no WASI imports, multithreading disabled, error strings stripped. Resultingzstd.wasmis ~546 KB (was ~136 KB).ZstdEncoder— low-level streaming encoder (push(chunk, onOutput)/finish(onOutput)/encode(plain)).ZstdCompressStream—TransformStreammirroring the platformCompressionStream; finalizes the frame on close.compressBytes(wasm, plain, level?)— one-shot helper.DEFAULT_COMPRESSION_LEVEL(3); compression level (1–22) configurable on all three entry points.cctx_new/cctx_free,cctx_set_level,compress_stream(ZSTD_compressStream2with anendOpdirective), andcstream_{in,out}_size.minor).CI
.github/workflows/ci.ymlrunspnpm build+pnpm testacross the monorepo on pushes tomainand on PRs.zstdbinary, which the new tests use to cross-validate WASM-produced frames. Uses the checked-in WASM binaries (native build steps fall back to committed artifacts when the toolchain is absent), so no wasi-sdk download is needed.Drive-by fix
@tootallnate/bffntand@tootallnate/ddsdeclared atestscript (vitest run) but ship no test files, so vitest exited 1. Added--passWithNoTeststo both (matching the existingffmpeg-wasmconvention) so the new CI is green.Testing
packages/zstd-wasm: 16/16 tests pass (5 existing decode + 11 new compression), including cross-validation against systemzstdv1.5.7, 4 MB roundtrips, compression-level monotonicity, empty inputs, and stream chaining.pnpm build+pnpm testis green across the monorepo (153/153 turbo tasks).