pyarrow is a large, C++-backed dependency (contributed heavily to the slow-first-import findings in #22, and now sits in the plexos/server/client extras per #23's packaging split). arro3 is a much smaller, Rust-backed minimal Arrow implementation — worth evaluating as a lighter alternative for GAT's actual usage.
Current pyarrow usage (verified, not guessed)
Exactly three call sites across the whole codebase:
gat/server/routes.py — Arrow IPC serialization for HTTP responses: pa.Table.from_pandas(df), pa.BufferOutputStream(), pa.ipc.new_stream(sink, table.schema).gat/client/connection.py — the corresponding deserialization: pa.ipc.open_stream(resp.content).gat/backends/duckdb_backend.py — df.to_parquet(..., compression="zstd"), which goes through pandas' own parquet engine dispatch, not a direct pyarrow API call.
Two different replacement paths
- (1) and (2), the Arrow IPC transport layer — straightforwardly arro3's core strength (Arrow C Data Interface + IPC serialization in a much smaller package). Likely a clean swap.
- (3), parquet writing — harder.
pandas.to_parquet(engine=...) is hard-wired to recognize pyarrow or fastparquet by name; arro3 isn't (currently) a registered pandas parquet engine, so this call site can't just swap the import. Two options: (a) keep pyarrow scoped narrowly to just this one parquet-writing path, or (b) — possibly the better fix — write parquet via DuckDB's own native writer (COPY ... TO 'file.parquet' / relation.write_parquet(...)) instead of routing through pandas at all, which sidesteps needing any parquet-engine package here since duckdb is already a hard dependency everywhere _write_parquet is called.
Scope
Exploration only — needs confirming arro3's current API maturity/stability for (1)/(2), and deciding on (3)'s approach, before any implementation PR. If (3) moves to DuckDB-native writing, pyarrow could potentially be dropped from plexos/server entirely and arro3 (much smaller) used only where client/server actually need Arrow IPC.
pyarrow is a large, C++-backed dependency (contributed heavily to the slow-first-import findings in #22, and now sits in the
plexos/server/clientextras per #23's packaging split). arro3 is a much smaller, Rust-backed minimal Arrow implementation — worth evaluating as a lighter alternative for GAT's actual usage.Current pyarrow usage (verified, not guessed)
Exactly three call sites across the whole codebase:
gat/server/routes.py— Arrow IPC serialization for HTTP responses:pa.Table.from_pandas(df),pa.BufferOutputStream(),pa.ipc.new_stream(sink, table.schema).gat/client/connection.py— the corresponding deserialization:pa.ipc.open_stream(resp.content).gat/backends/duckdb_backend.py—df.to_parquet(..., compression="zstd"), which goes through pandas' own parquet engine dispatch, not a direct pyarrow API call.Two different replacement paths
pandas.to_parquet(engine=...)is hard-wired to recognizepyarroworfastparquetby name; arro3 isn't (currently) a registered pandas parquet engine, so this call site can't just swap the import. Two options: (a) keep pyarrow scoped narrowly to just this one parquet-writing path, or (b) — possibly the better fix — write parquet via DuckDB's own native writer (COPY ... TO 'file.parquet'/relation.write_parquet(...)) instead of routing through pandas at all, which sidesteps needing any parquet-engine package here since duckdb is already a hard dependency everywhere_write_parquetis called.Scope
Exploration only — needs confirming arro3's current API maturity/stability for (1)/(2), and deciding on (3)'s approach, before any implementation PR. If (3) moves to DuckDB-native writing, pyarrow could potentially be dropped from
plexos/serverentirely and arro3 (much smaller) used only whereclient/serveractually need Arrow IPC.