Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- `vortex.zstd` segments compressed with a shared (trained) dictionary now decode, via the native `libzstd` dictionary support, instead of being rejected. The upstream `zstd.vortex` compatibility fixture is read end-to-end and matches the Rust reference. ([#104](https://github.com/dfa1/vortex-java/issues/104))
- Writing a nullable `Utf8`/`Binary` column no longer throws `NullPointerException` (or silently drops nulls): nullable string columns now carry their validity like nullable primitives and round-trip through `vortex.masked`. As a result they decode as `MaskedArray` (validity + values child) rather than a bare `VarBinArray`. ([#168](https://github.com/dfa1/vortex-java/pull/168))
- CSV export now handles nullable columns (`MaskedArray`): null rows export as an empty field instead of failing with "unsupported array type for CSV export". ([#168](https://github.com/dfa1/vortex-java/pull/168))
- Zone-map pruning now compares filter values in the *column's* type domain rather than by the boxed value's type. A predicate whose value is boxed at a different width (e.g. `Integer` on an `I64` column) — or any value on a `U64` column — previously pruned nothing and silently degraded to a full scan; it now prunes correctly (unsigned columns by unsigned order). As part of this, a filter value genuinely incomparable to its column (e.g. a `String` against a numeric column) now raises `VortexException` during the scan instead of silently disabling pruning — a behaviour change for callers that relied on the previous silent full scan. ([#159](https://github.com/dfa1/vortex-java/issues/159))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,10 @@ private static Double numericSum(Array arr) {
// ── Java side ─────────────────────────────────────────────────────────────

private static Long stringByteLength(Array arr) {
if (!(arr instanceof VarBinArray v)) {
// Nullable Utf8 columns decode as a MaskedArray over a VarBin values child (null rows
// contribute zero-length entries); unwrap to count the same bytes Rust reports.
Array values = arr instanceof MaskedArray m ? m.inner() : arr;
if (!(values instanceof VarBinArray v)) {
return null;
}
if (!(v.dtype() instanceof DType.Utf8)) {
Expand Down Expand Up @@ -394,7 +397,7 @@ private static Long stringByteLength(Array arr) {
"varbin.vortex",
"varbinview.vortex",
"zigzag.vortex",
// zstd.vortex excluded: ZstdEncoding dictionary mode not yet implemented
"zstd.vortex",
})
void rust_vs_javaReader_statsMatch(String fixture, @TempDir Path tmp) throws Exception {
// Given
Expand Down
Loading