Profile large CSVs... and warn on upload

[mccalluc]

Right now we're loading the whole file, and at some point this will impact usability.

• Generate fake CSVs, and vary:
  • number of rows
  • number of columns
  • bit depth / data type
• Profile performance on:
  • schema inference
  • sampling (of public datasets)
  • ...?
• Make sure none are super-linear.
• Figure out a way to do an early warning for large files: Just read as text and count \n?

[mccalluc]

Profiling results:

read all cells

from pathlib import Path
from time import perf_counter
import polars as pl

def fake_csv(rows, cols, value):
    header = ",".join(f"col_{c}" for c in range(cols)) + "\n"
    body = rows * (",".join(str(value) for _ in range(cols)) + "\n")
    return header + body

for rows in [10**(i) for i in range(4,8)]:
    for cols in [10**(j) for j in range(1,3)]:
        Path("/tmp/fake.csv").write_text(fake_csv(rows, cols, "0123456789abcdef"))
        start = perf_counter()
        pl.scan_csv(
            "/tmp/fake.csv",
            infer_schema_length=None,
            raise_if_empty=False,
        ).collect_schema().items()
        elapsed = perf_counter() - start
        print(f"{rows=}, {cols=}, {elapsed=}")

integers: 1

rows=10000, cols=10, elapsed=0.009321584133431315
rows=10000, cols=100, elapsed=0.053610207978636026
rows=100000, cols=10, elapsed=0.03279683296568692
rows=100000, cols=100, elapsed=0.34232974983751774
rows=1000000, cols=10, elapsed=0.31613133382052183
rows=1000000, cols=100, elapsed=3.3993930839933455
rows=10000000, cols=10, elapsed=3.1633327081799507
rows=10000000, cols=100, elapsed=34.136864291038364

strings: 0123456789abcdef

rows=10000, cols=10, elapsed=0.009124707896262407
rows=10000, cols=100, elapsed=0.06569833285175264
rows=100000, cols=10, elapsed=0.0665871249511838
rows=100000, cols=100, elapsed=0.6285352918785065
rows=1000000, cols=10, elapsed=0.6623431250918657
rows=1000000, cols=100, elapsed=6.331358416937292
rows=10000000, cols=10, elapsed=6.666578625096008
rows=10000000, cols=100, elapsed=96.20990795781836

comment out infer_schema_length=None

Small CSVs load faster, but surprisingly, large CSVs are, if anything, slower.

Just handle.readline()

Starts to slow down above 100_000_000:

rows=10000, cols=10, elapsed=0.0012739999999999974
rows=10000, cols=100, elapsed=0.006728000000000012
rows=100000, cols=10, elapsed=0.011370999999999964
rows=100000, cols=100, elapsed=0.064444
rows=1000000, cols=10, elapsed=0.112514
rows=1000000, cols=100, elapsed=0.645324

(And then crashed.)

Suggestions

• 100M values, either ints or short strings, can be read in 4-7 seconds. So let's conservatively just check the filesize before even reading, and warn if greater than 100M.
• I don't see any reason to drop infer_schema_length=None right now.
• We might add other load-time checks in the future, perhaps number of columns.