Is your feature request related to a problem? Please describe.
When shipping DataFrames over the wire or spilling them, it can be handy to pack them into a more compact single buffer first and then unpack them into multiple buffers at the other end.
Describe the solution you'd like
Recently this functionality was added at the C++ layer ( #7096 ). It would be good to have bindings to this for Python and use this in relevant serialize/deserialize methods.
Describe alternatives you've considered
We could do this packing elsewhere like in Distributed ( dask/distributed#3732 ). Though this would then not use the C++ implementation here. It also wouldn't solve this for other Python use cases
Additional context
We've discussed adding this to a config potentially ( #5311 ). Not sure if this is still needed given the newer C++ implementation
Is your feature request related to a problem? Please describe.
When shipping DataFrames over the wire or spilling them, it can be handy to pack them into a more compact single buffer first and then unpack them into multiple buffers at the other end.
Describe the solution you'd like
Recently this functionality was added at the C++ layer ( #7096 ). It would be good to have bindings to this for Python and use this in relevant
serialize/deserializemethods.Describe alternatives you've considered
We could do this packing elsewhere like in Distributed ( dask/distributed#3732 ). Though this would then not use the C++ implementation here. It also wouldn't solve this for other Python use cases
Additional context
We've discussed adding this to a config potentially ( #5311 ). Not sure if this is still needed given the newer C++ implementation