Part of #5.
output_chunk_size is a CLI flag, is passed through the controller, and is validated by _validate_chunking_support (src/daflip/services.py:144) which will raise NotImplementedError for an unsupported output format. It is then never read again. The docstring at services.py:544 admits this: "Size of chunks for writing (currently unused)".
So the flag can only ever fail; it can never do anything. Worse, output row-group size is a real thing users want to control for Parquet.
Fix: either implement it as the Parquet row-group size (currently one row group per input chunk) and the CSV append batch size, or remove the flag.
Part of #5.
output_chunk_sizeis a CLI flag, is passed through the controller, and is validated by_validate_chunking_support(src/daflip/services.py:144) which will raiseNotImplementedErrorfor an unsupported output format. It is then never read again. The docstring atservices.py:544admits this: "Size of chunks for writing (currently unused)".So the flag can only ever fail; it can never do anything. Worse, output row-group size is a real thing users want to control for Parquet.
Fix: either implement it as the Parquet row-group size (currently one row group per input chunk) and the CSV append batch size, or remove the flag.