We are currently using Arrow mainly for converting Protobuf state into queryable Arrow data. I think it would be interesting to see how we could use Arrow for streaming operations and how this can be exposed in the API (e..g, Dataframes?).
As of now, the only thing we have is ArrowWindow that exposes (Arc<Schema>, Vec<RecordBatch>) per Window trigger.
Relevant references:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/trill-vldb2015.pdf
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables/
We are currently using Arrow mainly for converting Protobuf state into queryable Arrow data. I think it would be interesting to see how we could use Arrow for streaming operations and how this can be exposed in the API (e..g, Dataframes?).
As of now, the only thing we have is
ArrowWindowthat exposes(Arc<Schema>, Vec<RecordBatch>)per Window trigger.Relevant references:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/trill-vldb2015.pdf
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables/