V0.7.0/arrow beam pipelines#240
Merged
Merged
Conversation
1. First-class streaming — ExArrow.Stream.from_parquet/1, from_ipc/1, from_flight/2, from_flight_sql/2, from_adbc/1,2 with source tracking and telemetry - closed #224 2. Batch operations — ExArrow.Batch with schema/1, select/2, drop/2, rename/2, take/2, filter/2 - closed #225 3. Flow integration — ExArrow.Flow.from_batches/1, map_batches/2, each_batch/2 - closed #226 4. GenStage integration — ParquetProducer, FlightProducer, ADBCProducer with demand-driven backpressure - closed #227 5. Broadway integration — BatchBuilder, ParquetSink, FlightSink - closed #228 6. Telemetry — ExArrow.Telemetry with all 6 required events, measurements, and metadata - closed #229 7. Pipeline sinks — ExArrow.Sink.Parquet, Flight, DataFrame, Nx - closed #230 8. Pipeline DSL — ExArrow.Pipeline with map_batches/2, each_batch/2, write_parquet/2, write_flight/3, write_dataframe/1 - closed #231 9. Benchmarks — v070_stream_flow_pipeline_bench.exs and v070_record_batch_vs_maps_bench.exs (1K/100K/1M rows) - closed #232 10. Educational guides — guides/06 through guides/10 - closed #233 Architectural decisions: - RecordBatch as the unit: all streams, pipelines, and sinks operate on ExArrow.RecordBatch handles, never row maps - Optional dependencies: flow, gen_stage, broadway, telemetry are all optional — modules degrade gracefully when absent - Telemetry-first: every stream constructor and batch emission fires telemetry; the :telemetry dep is optional - Lazy pipelines: ExArrow.Pipeline is lazy (Elixir Stream under the hood); nothing runs until a sink consumes - GenStage producers validate and open streams in start_link/3 so errors return cleanly as {:error, reason}
Version bumps (0.6.3 → 0.7.0) - mix.exs — version + new optional deps (already done in prior step) - CHANGELOG.md — v0.7.0 entry added (already done) - All 5 livebooks (00_quickstart through 04_adbc_integration) — Hex version ~> 0.6.3 → ~> 0.7.0 - livebook/README.md — version reference - README.md — installation snippets and Mix.install examples README updates - Tagline updated to mention "Arrow-native pipelines" - v0.7.0 banner at the top - Table of contents updated with new sections (What's changed, Batch operations, Streaming pipelines, Shipped v0.7.0) - New "What's changed in v0.7.0" section: new modules table, optional deps, telemetry events, new guides, new benchmarks, stats - Quick start expanded with v0.7.0 stream constructor + pipeline examples - New "Batch operations" section (ExArrow.Batch) - New "Streaming pipelines" section (ExArrow.Pipeline, sinks, Flow/GenStage/Broadway) - "Where ExArrow fits" mermaid diagram updated to show pipelines/Flow/Broadway/GenStage - "What it brings" expanded with the v0.7.0 pipeline bullet - Documentation list updated with the 5 new guides - Benchmarks table updated with the 2 new v0.7.0 suites - Roadmap "Shipped (v0.7.0)" section added; "OTel/telemetry" removed from longer-term (shipped) docs/overview.md - New "What's changed in v0.7.0" section at the top - Optional integrations table extended with Telemetry, Flow, GenStage, Broadway Guide cross-references - docs/parquet_guide.md and docs/ipc_guide.md — added v0.7.0 callout pointing to the new ExArrow.Stream.from_* constructors - docs/benchmarks.md — added the two v0.7.0 benchmark suite descriptions Livebook content - livebook/00_quickstart.livemd — new "9. Streaming pipelines (v0.7.0)" section with stream constructors, batch ops, pipeline DSL, and telemetry examples - livebook/README.md — table updated to mention v0.7.0 features in the quickstart
High Severity (Must Fix)
- F-001 Batch.drop/2 — Added validation that rejects unknown column names with {:error, "unknown column(s) for drop: ..."}. Added 2 regression tests (G-001). closed #234
- F-002 Guide 10 — Fixed broken map_batches(&ExArrow.Batch.select(&1, ...)) shorthand to properly unwrap {:ok, batch}. Added regression test (G-002) that verifies the pipeline rejects a fun returning {:ok, batch}. closed #235
- F-003 Guide 06 — Replaced crashing Stream.iterate/2 example with a correct recursive loop pattern. closed #236
- F-004 Guide 07 — Fixed ExArrow.Batch.column_names/1 → ExArrow.RecordBatch.column_names/1. closed #237
- F-005 flow.ex — Fixed @SPEC from_batches → @SPEC map_batches on the map_batches/2 function.
- F-006 gen_stage.ex — Fixed dispatch/2 done branch to pass state through instead of constructing a fresh %State{done: true} that drops :source and :demand. Added regression test (G-004).
Medium Severity (Should Fix) closed #238
- F-007 stream.ex — Moved telemetry execute calls inside the with success branch for all 5 constructors so failed opens don't emit events. Added regression test (G-003).
- F-008 pipeline.ex — Added > #### Memory admonitions to write_parquet/2, write_flight/3, and write_dataframe/1 documenting the materialisation behavior.
- F-009 broadway.ex — Added @broadway_available check and broadway_available?/0 function; changed @SPEC types from Broadway.Message.t() to term() so the module is self-consistent without the dep.
Low Severity closed #239
- F-010/D-002 — Extracted shared ipc_stream/1, ipc_binary/1, s64_batch/2 into test/support/test_fixtures.ex; updated 5 test files to import ExArrow.TestFixtures.
- F-011 — Added guides/05_arrow_pipelines_overview.md to close the numbering gap.
- F-012 — Removed ExArrow.Broadway from groups_for_modules in mix.exs (kept submodules).
- F-013 — Removed :bytes from the documented measurements table in telemetry.ex and from ParquetSink moduledoc.
- F-014/D-001 — Removed notify_or_ignore/1 dead code from gen_stage.ex.
- F-015/A-001/A-002 — Removed milestone labels and em-dash banner separators from stream.ex, batch.ex, pipeline.ex, and test files.
- F-016/A-003 — Deleted livebook/README copy.md.
- D-003 — Merged normalise/1 and normalise_flight/1 into shared ExArrow.Sink.Helpers.normalise_source/2.
- G-005 — Added test for Sink.Parquet.write/2 with {schema, :not_a_list} and unsupported source type.
- G-006 — Added test for GenStage producer :stream option with source: nil verifying the :parquet fallback label in telemetry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.7.0 is a substantial and well-structured release that adds first-class Arrow
streaming, batch operations, pipeline DSL, and integrations with Flow, GenStage,
Broadway, and Telemetry — roughly 5,000 lines of new code across 39 files. The
core abstractions (
ExArrow.Stream,ExArrow.Batch,ExArrow.Pipeline,ExArrow.Telemetry) are sound and the test suite is wide-coverage.