Part of #5.
infer_and_export_schema documents nrows as "Number of rows to read for schema inference (default: 10000)" and says it reads only the first nrows for large files (src/daflip/services.py:679, 685). In practice nrows is only applied for CSV (services.py:709). Every other pandas-backed format falls through to _read_dataframe, which for SAS calls pd.read_sas(input_file, encoding=sas_encoding) with no row limit.
Running daflip schema on a 71 GB .sas7bdat therefore attempts to load the whole file into memory, which is both the opposite of what the docstring promises and, on a large enough file, fatal.
Fix: honour nrows for the chunk-capable formats by pulling a single chunk from a chunked reader (SAS supports this via chunksize), and for formats where a row limit is genuinely unavailable, say so rather than silently doing a full read.
Part of #5.
infer_and_export_schemadocumentsnrowsas "Number of rows to read for schema inference (default: 10000)" and says it reads only the firstnrowsfor large files (src/daflip/services.py:679, 685). In practicenrowsis only applied for CSV (services.py:709). Every other pandas-backed format falls through to_read_dataframe, which for SAS callspd.read_sas(input_file, encoding=sas_encoding)with no row limit.Running
daflip schemaon a 71 GB.sas7bdattherefore attempts to load the whole file into memory, which is both the opposite of what the docstring promises and, on a large enough file, fatal.Fix: honour
nrowsfor the chunk-capable formats by pulling a single chunk from a chunked reader (SAS supports this viachunksize), and for formats where a row limit is genuinely unavailable, say so rather than silently doing a full read.