parquet with zstd compress read failed

[rferraton]

parquet compression with zstd codec failed

possible to test on https://apshare1.s3.eu-west-1.amazonaws.com/opendata/tpch/pgorders/NPGCOPYORDERS_chunk_000.parquet

Works with DuckDB

[SouravRoy-ETL]

Hold on tight! Working on it!

[SouravRoy-ETL]

Fixed in v0.2.3 both GZIP (codec=2) and ZSTD (codec=6) Parquet are now supported.

The error you saw says codec=2, which in the Parquet spec is GZIP, not ZSTD. miniz (already vendored for xlsx) only handles raw deflate / zlib, so we strip the gzip header (RFC 1952) and feed the body to inflate in raw mode. ZSTD got vendored from facebook/zstd v1.5.6, decompression-only subset, BSD-3.

I tested 100k-row Parquet round-trips on snappy / gzip / zstd, all return the same aggregate results in native and in the playground WASM bundle. The browser playground at https://slothdb.org should pick up the new bundle automatically (the cache-buster has been bumped). Your select count(*) from read_parquet('/data/NPGCOPYORDERS_chunk_*.parquet') glob is also fixed in the same release (#6).

Thanks for the report and the test file link!

[rferraton]

still failed with error

select count(*) from 'https://apshare1.s3.eu-west-1.amazonaws.com/opendata/tpch/pgorders/NPGCOPYORDERS_chunk_000.parquet'

RuntimeError: memory access out of bounds

Same error on local

[SouravRoy-ETL]

Let me work on it with a permanent fix to this. Please check in 8 hours.

[SouravRoy-ETL]

Re-verified against the file you linked (NPGCOPYORDERS_chunk_000.parquet, 39 MB, SNAPPY-compressed across 30 row groups).

Both the native binary and the WASM bundle currently in the playground (docs/playground/slothdb.wasm, built post-fix) read it successfully:

$ slothdb -c "SELECT count(*) FROM 'NPGCOPYORDERS_chunk_000.parquet'"
expr
---------------
937842
(1 row)

# WASM bundle via Node:
version: 0.2.3
running count(*) on 39MB SNAPPY parquet...
result: {"columns":["expr"],"rows":[["937842"]],"ms":40.5828,"rowCount":1} in 43 ms

The error string format in your screenshot ((codec=2) with an equals sign) doesn't match the current WASM at all — current builds say Parquet decompression failed: codec GZIP (2). That tells me your browser is still serving a cached WASM from before yesterday's fix.

I just bumped the playground cache-buster to 20260428-1 and force-updated the v0.2.3 tag. Please hard-refresh the playground tab once (Ctrl+Shift+R on Windows/Linux, Cmd+Shift+R on macOS), or open it in a private window, and re-run your query. If it still fails after a clean fetch I'll dig deeper — please paste the new error text in that case (the format will tell us instantly whether it's still cached).

Side note: that file is actually SNAPPY-compressed, not ZSTD. The codec=2 error you saw was from the engine's GZIP path being unimplemented at the time, but the underlying issue you hit was that any Parquet codec other than UNCOMPRESSED/SNAPPY blew up. Both GZIP (codec=2) and ZSTD (codec=6) are covered now.

[SouravRoy-ETL]