Part of #5.
_build_write_kwargs is only reached on the non-chunked path (src/daflip/services.py:590-594). The chunked writers take no compression argument at all: _write_chunk_parquet constructs pq.ParquetWriter(output_file, schema) (services.py:252) and _write_chunk_csv calls to_csv with no compression.
Result: daflip convert big.sas7bdat out.parquet --input-chunk-size 1000000 --compression zstd writes snappy, the PyArrow default, with no indication the flag was dropped. This matters most on exactly the large files that force chunking in the first place, where the difference between snappy and zstd is substantial on disk.
Fix: thread compression and compression_level into _handle_chunked_conversion and pass them to ParquetWriter(...) and to_csv(...).
Part of #5.
_build_write_kwargsis only reached on the non-chunked path (src/daflip/services.py:590-594). The chunked writers take no compression argument at all:_write_chunk_parquetconstructspq.ParquetWriter(output_file, schema)(services.py:252) and_write_chunk_csvcallsto_csvwith nocompression.Result:
daflip convert big.sas7bdat out.parquet --input-chunk-size 1000000 --compression zstdwrites snappy, the PyArrow default, with no indication the flag was dropped. This matters most on exactly the large files that force chunking in the first place, where the difference between snappy and zstd is substantial on disk.Fix: thread
compressionandcompression_levelinto_handle_chunked_conversionand pass them toParquetWriter(...)andto_csv(...).