fix(deps): update ballista stack (major) by renovate[bot] · Pull Request #113 · duyet/ballista

renovate · 2025-11-16T17:44:08Z

ℹ️ Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Type	Update	Change
ballista (source)	dependencies	major	`49.0` → `54.0`
datafusion (source)	dependencies	major	`49.0` → `55.0`

Release Notes

apache/datafusion-ballista (ballista)

`v54.1.0`

Compare Source

Apache DataFusion Ballista 54.1.0 Changelog

This release consists of 10 commits from 3 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

fix: [branch-54] disable the SortMergeJoinExec broadcast conversion by default #2130 (andygrove)
fix: [branch-54] map a task's partition through UnionExec when restricting file scans (backport of #2070) #2131 (andygrove)
fix: [branch-54] keep EmptyExec partition count intact across stage serialization (backport of #2062) #2132 (andygrove)
fix: [branch-54] send the shutdown notification before dropping the notifier (backport of #2107) #2237 (sandugood)
fix: [branch-54] bind the gRPC listener before registering with the scheduler (backport of #2225) #2236 (andygrove)
fix: [branch-54] keep GROUP BY-less aggregates when propagating empty stages (backport of #2194) #2235 (andygrove)
fix: [branch-54] spill sort shuffle writer when the memory pool rejects a reservation grow (backport of #2061) #2238 (andygrove)
fix: [branch-54] enable the executor client pool by default (backport of #2069) #2234 (andygrove)
fix: [branch-54] see through Shared wrapper when classifying task IO errors (backport of #2119) #2239 (goingforstudying-ctrl)

Other:

chore: [branch-54] upgrade DataFusion to 54.1.0 and bump Ballista version to 54.1.0 #2127 (andygrove)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

     8	Andy Grove
     1	sandugood
     1	goingforstudying-ctrl

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.

`v54.0.0`

Compare Source

Apache DataFusion Ballista 54.0.0 Changelog

This release consists of 171 commits from 11 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

fix(python): clear latent lint debt and gate wheel jobs on lint/tests #1749 (andygrove)
fix(python): exclude mimalloc from Python wheel to avoid macOS segfault #1769 (andygrove)
fix: handle task execution errors and panics in executor #1793 (milenkovicm)
fix: downgrade time version to fix build issue #1862 (milenkovicm)
fix: use DataFrame.name instead of hardcoded string in metaclass method wrapping #1866 (djanand)
fix: honor client datafusion.* session config overrides in scheduler planning #1902 (andygrove)
fix: Adaptive query planning fails with remote object stores: "No suitable object store found" #1962 (milenkovicm)
fix: satisfy clippy 1.97 useless_borrows_in_formatting #1978 (andygrove)

Performance related:

perf: avoid redundant TaskStatus clones in scheduler update_task_status #1983 (andygrove)
perf: prune irrelevant partitions task #1911 (coderfender)

Implemented enhancements:

feat: task_manager delegats physical plan creation to execution graph #1726 (milenkovicm)
feat: ShuffleReader shows upstream_stage #1767 (milenkovicm)
feat: runtime stage's metrics rendering #1778 (sandugood)
feat: add /api/job/{job_id}/config endpoint to retrieve session configuration #1792 (killzoner)
feat(aqe): delay join decision & introduce broadcast join in AQE #1752 (milenkovicm)
feat: Add web rendering of the TUI application #1742 (martin-g)
feat(aqe): chaos monkey testing for ballista execution #1802 (milenkovicm)
feat(aqe): empty join handling #1723 (sandugood)
feat(tui): Show shimmer animation for "Queued" and "Running" jobs #1808 (martin-g)
feat(cli): use local time in tracing subscriber logging for ballista-cli #1824 (jiengup)
feat(scheduler): Use command line options for CORS origins/methods instead of env vars #1818 (martin-g)
feat(tui): Add configuration popup for jobs #1825 (jiengup)
feat(TUI): enable various plan rendering formats #1828 (sandugood)
feat(tui): Add theming support #1850 (martin-g)
feat: NewType pattern for job_id #1852 (JarroVGIT)
feat(tui): Show job failed status below the Jobs table #1871 (martin-g)
feat: broadcast small build sides via non-zero CollectLeft thresholds #1900 (andygrove)
feat: broadcast small build side of SortMergeJoinExec in the static planner #1904 (andygrove)
feat: Allow submitting a pre-built physical plan to the scheduler #1924 (andygrove)
feat: cancel running stages and tasks on job failure or cancellation #1903 (killzoner)
feat: support running TPC-H benchmarks against S3-compatible object storage #1942 (andygrove)
feat: bound shuffle fetch with a reduce-side in-flight governor #1951 (andygrove)
feat: improve session state creation for AQE planner #1963 (milenkovicm)
feat: delete intermediate shuffle files on job success #1982 (andygrove)
feat: Share read-side runtime state across a session's tasks on the executor #1995 (andygrove)

Documentation updates:

docs(release): document PyPI publish process for python client #1610 (andygrove)
docs(release): cover TestPyPI verification and fix stale wheel-download notes #1760 (andygrove)
refactor: make use of async_trait::async_trait consistent across project #1848 (JarroVGIT)
chore(docs): Update docs website to bring inline with Datafusion style #1857 (JarroVGIT)
Improve changelog generator and split CHANGELOG.md into per-version files #1768 (andygrove)
docs: add versioned upgrade guide (54.0.0) #1939 (andygrove)
docs: document TPC-H plan-stability golden files and regeneration #1979 (andygrove)
docs: update companies using ballista #1992 (coderfender)
docs: link AQE limitations to tracking issues and add AQE contributor guide section #1990 (andygrove)

Other:

chore(deps): bump idna from 3.11 to 3.15 in /python #1737 (dependabot[bot])
chore(ci): Execute the CI workflows only when this is a push to main or a PR against any branch #1738 (martin-g)
minor: move aqe executions plans to a directory #1744 (milenkovicm)
ci: pin all third-party actions to ASF-allowlisted SHAs #1748 (andygrove)
ci: pin Windows wheel builder to windows-2022 #1751 (andygrove)
chore: bump python crate to 53.0.0 and use path deps for ballista crates #1750 (andygrove)
ci: produce real aarch64 wheel and upload sdist artifact #1753 (andygrove)
ci: only run Python Release Build on RC tags and matching PRs #1754 (andygrove)
ci: gate TPC-H workflow on rust-only path allow list #1755 (andygrove)
ci: gate Docker workflow on rust-only path allow list #1756 (andygrove)
ci: gate Rust workflow on workspace path allow list and rename to Ballista Rust #1757 (andygrove)
ci: use maturin sdist subcommand to avoid building a wheel #1759 (andygrove)
ci: gate CodeQL workflow on .github path changes #1758 (andygrove)
minor: Ballista showplan helper script #1761 (milenkovicm)
chore(deps): bump requests from 2.32.5 to 2.33.0 in /python #1781 (dependabot[bot])
chore(ci): Execute the "Dependencies" workflow on push to main and for PRs #1782 (martin-g)
chore(deps): bump astral-tokio-tar from 0.6.1 to 0.6.2 #1794 (dependabot[bot])
chore(deps): bump pygments from 2.19.2 to 2.20.0 in /python #1800 (dependabot[bot])
minor: code cleanup and log level changes #1805 (milenkovicm)
minor(aqe): backport few datafusion physical rules identified as not idempotent #1806 (milenkovicm)
minor(aqe): add plan_id to dynamic join #1809 (milenkovicm)
chore(docker): Use yellow_tripdata_2022-01.parquet from nightlies.apache.org #1817 (martin-g)
chore(ci): Use MacOS Github hosted runners for the small CI checks #1810 (martin-g)
chore(ci): Fix the Rust installer for MacOS in dependencies.yml #1833 (martin-g)
chore(ci): Drop Windows CI runs #1801 (martin-g)
chore(ci): Fix the name of a CI job dependency #1834 (martin-g)
chore(github): Enable "Update branch" button for PRs #1839 (martin-g)
chore(ci): Download macos aarch64 build of protoc #1835 (martin-g)
chore(tui): Upgrade Ratatui to 0.30.1 / Ratzilla to 0.3.1 #1840 (martin-g)
chore(ci): Run cargo check and cargo doc CI jobs on MacOS #1846 (martin-g)
chore: re enable approx distinct test #1844 (coderfender)
bug: fix empty projection serde ambiguity bug #1863 (coderfender)
chore(tui): Use lighter foreground colors for the dark theme. For better contrast #1872 (martin-g)
minor: remove redundant scheduler info loggers #1879 (milenkovicm)
chore(ci): Deploy WebTUI to nightlies.a.o #1888 (martin-g)
ci: run TPC-H SF10 under AQE off + on, and build it faster #1913 (andygrove)
chore(deps): upgrade to DataFusion 54 #1906 (andygrove)
ci: verify TPC-H query results against single-process DataFusion #1917 (andygrove)
chore(deps): upgrade pyballista to DataFusion 54 #1919 (djanand)
Fix/python cargo lock drift #1899 (djanand)
refactor(scheduler): keep submit_job non-breaking, add submit_physical_plan #1941 (andygrove)
Add TPC-H distributed plan-stability test suite #1966 (andygrove)
Display input stage id on UnresolvedShuffleExec #1969 (andygrove)
chore(deps): bump pyarrow from 23.0.0 to 23.0.1 in /python #1961 (dependabot[bot])
chore: add script and CI check to sync vendored DataFusion protos #1991 (andygrove)
feat(tui + rest): surface failed tasks #1949 (killzoner)
Add shuffle-read fetch metrics and surface per-operator metrics in plan display #1968 (andygrove)
Log task execution duration in the executor "Finished task" message #1999 (andygrove)
chore: bump Ballista version to 54.0.0 #2004 (andygrove)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    83	dependabot[bot]
    38	Andy Grove
    18	Martin Grigorov
    14	Marko Milenković
     4	Bhargava Vadlamani
     3	Jarro van Ginkel
     3	alexander domenti
     3	jgrim
     2	Gunther Xing
     2	Jogesh A Dinavahi
     1	Jogesh Dinavahi

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.

`v53.0.0`

Compare Source

Apache DataFusion Ballista 53.0.0 Changelog

This release consists of 174 commits from 16 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

fix: remove unwrap from executor and improve task error handling #1540 (eachsaj)
fix: handle None task slot in update_task_info after executor lost #1523 (milenkovicm)
fix: [Python] serialize optimized logical plan to fix subquery support #1586 (andygrove)
fix: allow S3 access without explicit credentials #1584 (andygrove)
fix: route collect/show/to_pandas through Ballista cluster #1585 (andygrove)
fix: propagate session config from Python client to Ballista cluster #1592 (andygrove)
fix(python): ignore ballista-namespaced cluster_config keys locally #1613 (andygrove)
fix: df.write_ fix as it was broken after update #1625 (milenkovicm)
fix(rest): remove unwrap and return 404 if executor does not exist #1628 (milenkovicm)
fix(metrics): avoid stage metrics inflation by tracking partition snapshots #1652 (danielhumanmod)
fix: compilation issue after merge #1658 (milenkovicm)
fix: rest api calculates stage running time correctly #1675 (milenkovicm)
fix: REST API does not show running jobs #1703 (gittihub-jpg)
fix: no executor warning, correct prometheus feature name in TUI #1698 (killzoner)

Implemented enhancements:

[REST]: Cancelling a completed/failed job should return "false" #1494 (martin-g)
feat: Add plain "status" field to the JobResponse #1497 (martin-g)
feat: Expose Logical and Physical plan details in the REST API #1498 (milenkovicm)
feat: (remote) shuffle reader cleanup #1503 (milenkovicm)
feat: enable scheduler rest api by default #1506 (milenkovicm)
feat: [REST] Return the job's start and end times #1511 (martin-g)
feat: remove hash policy task distribution as not used #1526 (eachsaj)
feat: remove full path from partition locations #1527 (milenkovicm)
feat: add config to ExecutorEngine::create_query_stage_exec #1542 (milenkovicm)
feat: Improve REST API adding task related info to job stages #1543 (milenkovicm)
feat: Ballista Text User Interface #1436 (martin-g)
feat: jupyter notebook support #1513 (sandugood)
feat: ExecutionEngine::create_query_stage_exec accepts partition_id #1556 (milenkovicm)
feat: make ballista client retry policy configurable #1577 (eachsaj)
feat: Executor system & process metrics reporting #1547 (sandugood)
feat: Scheduler config update #1597 (sandugood)
feat: Support EXPLAIN ANALYZE in Ballista #1567 (danielhumanmod)
feat: Add standalone shuffle writer benchmark that shuffles real Parquet input #1600 (andygrove)
feat: defer sort-shuffle materialization with interleave_record_batch #1598 (andygrove)
feat(executor): bound executor memory via --memory-pool-size #1624 (andygrove)
feat(bench): show TPC-H query timings in seconds and add total time #1641 (andygrove)
feat(aqe): Support sort-based shuffle writer in AQE #1640 (danielhumanmod)
feat: rest api supports plan tree rendering #1650 (sandugood)
feat: Cache ballista clients on executor #1578 (milenkovicm)
feat(aqe): Lazy stage evaluation in AQE #1649 (milenkovicm)
feat: move shuffle writer disk I/O off tokio worker threads #1537 (hcrosse)
feat(scheduler): broadcast-style hash join for small-side joins #1647 (andygrove)
feat: default to sort-merge join #1651 (andygrove)
feat: TUI shows running job information #1717 (milenkovicm)
feat(aqe): CoalescePartitionsRule — shuffle-partition coalescing on resolved stats #1684 (metegenez)
feat: TUI make task popup scrollable #1725 (milenkovicm)
feat(tui): Use separate areas for the table and its associated scrollbar #1729 (martin-g)

Documentation updates:

docs: document hash-based and sort-based shuffle implementations #1595 (andygrove)
docs: improve Python documentation structure #1579 (andygrove)
docs: add Python client build-from-source section to contributor guide #1620 (andygrove)
chore: remove NYC Taxi benchmark #1644 (andygrove)
docs: document experimental Adaptive Query Execution in tuning guide #1645 (andygrove)
build(bench): rework docker-compose TPC-H stack #1646 (andygrove)
docs: add Ballista TUI documentation #1593 (goingforstudying-ctrl)
[TUI] Show executor's details in a popup #1670 (martin-g)
[TUI] Add a config setting for rendering job stage's plan as a tree #1704 (martin-g)
[TUI] Add support for horizontal scrolling to the job/stage plan popups #1711 (martin-g)
[TUI] Add screenshots of the TUI application in README/cli.md #1714 (martin-g)

Other:

CI: Add CodeQL workflow for GitHub Actions security scanning #1484 (kevinjqliu)
ci: Harden labeler workflow, remove unnecessary checkout from pull_request_target job #1487 (kevinjqliu)
chore(deps): bump tokio from 1.49.0 to 1.50.0 #1490 (dependabot[bot])
chore(deps): bump actions/setup-node from 6.2.0 to 6.3.0 #1489 (dependabot[bot])
ci: add take and stale #1488 (kevinjqliu)
chore(deps): bump uuid from 1.21.0 to 1.22.0 #1492 (dependabot[bot])
chore(deps): bump github/codeql-action from 4.32.5 to 4.32.6 #1491 (dependabot[bot])
chore(deps): bump libc from 0.2.182 to 0.2.183 #1495 (dependabot[bot])
chore(deps): bump graphviz-rust from 0.9.6 to 0.9.7 #1496 (dependabot[bot])
chore(deps): bump quinn-proto from 0.11.13 to 0.11.14 #1499 (dependabot[bot])
chore(deps): bump quinn-proto from 0.11.13 to 0.11.14 in /python #1500 (dependabot[bot])
chore(deps): bump tempfile from 3.26.0 to 3.27.0 #1501 (dependabot[bot])
chore(deps): cargo update deps #1502 (milenkovicm)
chore(deps): bump once_cell from 1.21.3 to 1.21.4 #1505 (dependabot[bot])
chore(deps): bump clap from 4.5.60 to 4.6.0 #1504 (dependabot[bot])
minor: task scheduling policy config cleanup #1507 (milenkovicm)
minor: address task policy config comments #1508 (milenkovicm)
chore(deps): bump tracing-subscriber from 0.3.22 to 0.3.23 #1510 (dependabot[bot])
minor: [REST] add datafusion version info #1512 (milenkovicm)
chore(deps): bump lz4_flex from 0.12.0 to 0.12.1 in /python #1514 (dependabot[bot])
chore(deps): bump github/codeql-action from 4.32.6 to 4.33.0 #1515 (dependabot[bot])
chore(deps): bump rustls-webpki from 0.103.9 to 0.103.10 in /python #1517 (dependabot[bot])
chore(deps): bump lz4_flex from 0.12.0 to 0.12.1 #1518 (dependabot[bot])
chore(deps): bump github/codeql-action from 4.33.0 to 4.34.1 #1519 (dependabot[bot])
chore(deps): bump rustls-webpki from 0.103.9 to 0.103.10 #1521 (dependabot[bot])
ci: pin third-party actions to Apache-approved SHAs #1516 (kevinjqliu)
chore(deps): bump env_logger from 0.11.9 to 0.11.10 #1522 (dependabot[bot])
chore(deps): update to datafusion v.53 #1486 (milenkovicm)
chore(deps): bump insta from 1.46.3 to 1.47.0 #1524 (dependabot[bot])
chore(deps): bump uuid from 1.22.0 to 1.23.0 #1525 (dependabot[bot])
chore: update datafusion proto #1528 (milenkovicm)
minor: cleanup scheduler clap configuration #1529 (milenkovicm)
chore(deps): bump insta from 1.47.0 to 1.47.1 #1532 (dependabot[bot])
chore(deps): bump ctor from 0.6.3 to 0.8.0 #1534 (dependabot[bot])
chore(deps): bump md-5 from 0.10.6 to 0.11.0 #1533 (dependabot[bot])
chore(deps): bump github/codeql-action from 4.34.1 to 4.35.1 #1530 (dependabot[bot])
chore(deps): bump insta from 1.47.1 to 1.47.2 #1535 (dependabot[bot])
chore(deps): bump libc from 0.2.183 to 0.2.184 #1538 (dependabot[bot])
chore(deps): bump tokio from 1.50.0 to 1.51.0 #1544 (dependabot[bot])
chore(deps): bump tui-big-text from 0.8.3 to 0.8.4 #1546 (dependabot[bot])
chore(deps): bump tokio from 1.51.0 to 1.51.1 #1545 (dependabot[bot])
chore(deps): bump ctor from 0.8.0 to 0.9.1 #1548 (dependabot[bot])
chore(deps): bump ctor from 0.9.1 to 0.10.0 #1549 (dependabot[bot])
chore(deps): bump rustls from 0.23.37 to 0.23.38 #1550 (dependabot[bot])
chore(deps): bump libc from 0.2.184 to 0.2.185 #1553 (dependabot[bot])
chore(deps): bump rand from 0.9.2 to 0.9.4 in /python #1552 (dependabot[bot])
chore(deps): bump tokio from 1.51.1 to 1.52.0 #1555 (dependabot[bot])
chore(deps): bump axum from 0.8.8 to 0.8.9 #1554 (dependabot[bot])
chore(deps): bump github/codeql-action from 4.35.1 to 4.35.2 #1557 (dependabot[bot])
chore: Fix Clippy issues with Rust 1.95.0 #1558 (martin-g)
chore: update deps #1561 (milenkovicm)
chore(deps): bump actions/setup-node from 6.3.0 to 6.4.0 #1562 (dependabot[bot])
chore(deps): bump mimalloc from 0.1.48 to 0.1.49 #1566 (dependabot[bot])
chore(deps): bump aws-config from 1.8.15 to 1.8.16 #1564 (dependabot[bot])
chore(deps): bump rand from 0.9.4 to 0.10.1 #1565 (dependabot[bot])
chore(deps): bump ctor from 0.10.0 to 0.10.1 #1571 (dependabot[bot])
chore(deps): bump mimalloc from 0.1.49 to 0.1.50 #1570 (dependabot[bot])
chore(deps): bump rustls from 0.23.38 to 0.23.39 #1569 (dependabot[bot])
chore(deps): bump libc from 0.2.185 to 0.2.186 #1572 (dependabot[bot])
[TUI] Change the key binding for job's plans #1573 (martin-g)
chore(deps): bump rustls-webpki from 0.103.12 to 0.103.13 #1576 (dependabot[bot])
chore(deps): bump rustls-webpki from 0.103.10 to 0.103.13 in /python #1575 (dependabot[bot])
minor: make config naming convention consistent #1580 (milenkovicm)
chore: Remove unnecessary optimizer rules due to datafusion upgrade to v53 #1594 (sandugood)
chore(deps): bump reqwest from 0.13.2 to 0.13.3 #1605 (dependabot[bot])
ci: drop Intel macOS Python wheel build #1612 (andygrove)
chore(deps): bump ctor from 0.10.1 to 0.11.1 #1618 (dependabot[bot])
chore(deps): bump rustls from 0.23.39 to 0.23.40 #1619 (dependabot[bot])
chore: update python deps to ballista and datafusion 52 #1590 (andygrove)
feat(sort-shuffle): byte-copy spill files and enable block-IO transport #1615 (andygrove)
Use BallistaSessionContext instead of BallistaBuilder in tpch.py #1621 (martin-g)
feat(sort-shuffle): enable sort-based shuffle by default #1623 (andygrove)
perf(sort-shuffle): fix performance regression caused by datafusion upgrade #1626 (andygrove)
chore(deps): bump ctor from 0.11.1 to 0.12.0 #1639 (dependabot[bot])
fix(sort-shuffle): bound writer memory with per-task spill threshold #1636 (andygrove)
chore(deps): bump graphviz-rust from 0.9.7 to 0.9.8 #1655 (dependabot[bot])
chore(deps): bump github/codeql-action from 4.35.2 to 4.35.3 #1653 (dependabot[bot])
chore(deps): bump tokio from 1.52.1 to 1.52.2 #1657 (dependabot[bot])
chore(deps): bump actions/labeler from 6.0.1 to 6.1.0 #1659 (dependabot[bot])
[TUI] Show job's stages and their tasks #1574 (martin-g)
chore(deps): bump tonic-build from 0.14.5 to 0.14.6 #1662 (dependabot[bot])
chore(deps): bump tonic from 0.14.5 to 0.14.6 #1661 (dependabot[bot])
chore(deps): bump astral-tokio-tar from 0.6.0 to 0.6.1 #1664 (dependabot[bot])
chore(deps): bump ctor from 0.12.0 to 1.0.2 #1663 (dependabot[bot])
chore(deps): bump github/codeql-action from 4.35.3 to 4.35.4 #1665 (dependabot[bot])
chore(deps): bump ctor from 1.0.2 to 1.0.3 #1668 (dependabot[bot])
chore(deps): bump tonic-prost from 0.14.5 to 0.14.6 #1667 (dependabot[bot])
chore(deps): bump tonic-prost-build from 0.14.5 to 0.14.6 #1666 (dependabot[bot])
[TUI] Configurable tick interval #1669 (martin-g)
minor: add pending stage indicator for AdaptiveDatafusionExec #1672 (milenkovicm)
chore(deps): bump ctor from 1.0.3 to 1.0.4 #1680 (dependabot[bot])
chore(deps): bump tokio from 1.52.2 to 1.52.3 #1678 (dependabot[bot])
minor: change parameter ordering in AdaptivePlanner::try_new_with_optimizers #1687 (milenkovicm)
chore(deps): bump ctor from 1.0.4 to 1.0.5 #1694 (dependabot[bot])
Fix REST API panic on job list/detail when end_time < start_time #1693 (abhinavgautam01)
Right align all numeric columns in the TUI tables #1695 (martin-g)
fix(join-selection): guard CollectLeft swap when right has multiple partitions #1691 (andygrove)
Minor improvements for #1675 #1686 (martin-g)
[CLI/TUI] Use only tracing crate for logging in CLI and TUI #1697 (martin-g)
chore(deps): bump urllib3 from 2.6.3 to 2.7.0 in /python #1699 (dependabot[bot])
chore(deps): bump pyjwt from 2.10.1 to 2.12.0 in /python #1700 (dependabot[bot])
chore(deps): bump pytest from 9.0.2 to 9.0.3 in /python #1702 (dependabot[bot])
minor: change log level for few statements #1706 (milenkovicm)
chore(deps): bump config from 0.15.22 to 0.15.23 #1709 (dependabot[bot])
Saturate scheduler job elapsed time #1708 (MukundaKatta)
[TUI] Executor's id is not a numeric column. It should be center aligned #1713 (martin-g)
Make use of Swatinem/rust-cache to make the CI workflows faster #1705 (martin-g)
Merge Executor's brief and extended properties #1716 (martin-g)
[INFRA] Set up default rulesets for default and release branches #1715 (asf-gitbox-commits)
chore(deps): bump github/codeql-action from 4.35.4 to 4.35.5 #1719 (dependabot[bot])
chore(deps): bump ctor from 1.0.5 to 1.0.6 #1720 (dependabot[bot])
chore(deps): bump dashmap from 6.1.0 to 6.2.1 #1721 (dependabot[bot])
ci: add TPC-H SF10 workflow #1688 (andygrove)
minor: [TUI] Extract a helper method for the table/scrollbar area splitter #1730 (martin-g)
minor: [TUI] Sort the metrics before rendering them #1731 (martin-g)
chore: Do not run -cli tests twice #1732 (martin-g)
chore: generate changelog for ballista 53 #1735 (milenkovicm)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    74	dependabot[bot]
    29	Marko Milenković
    25	Andy Grove
    23	Martin Grigorov
     5	alexander domenti
     4	Kevin Liu
     3	Daniel Tu
     3	Saj
     1	Abhinav Gautam
     1	Harrison Crosse
     1	Mukunda Rao Katta
     1	The Apache Software Foundation
     1	gittihub-jpg
     1	goingforstudying-ctrl
     1	jgrim
     1	mete

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.

`v52.0.0`

Compare Source

Apache DataFusion Ballista 52.0.0 Changelog

Performance related:

perf: optimize shuffle writer with buffered I/O and fix file size bug #1386 (andygrove)

Implemented enhancements:

feat: add config option for skipping arrow ipc read validation #1374 (killzoner)
feat: improve tpch benchmark CLI #1391 (andygrove)
feat: Add sort-based shuffle implementation #1389 (andygrove)
feat: New ballista python interface #1338 (milenkovicm)
feat: Add batch coalescing ability to shuffle reader exec #1380 (danielhumanmod)
feat: Add arrow flight proxy to scheduler #1351 (sebbegg)
feat: Creating SubstraitSchedulerClient and standalone Substrait examples #1376 (mattcuento)
feat: Cluster RPC customisations to support TLS and custom headers #1400 (phillipleblanc)
feat: add -c config override flag to tpch benchmark #1435 (andygrove)
feat: Extract execution_graph to a trait #1361 (milenkovicm)
feat: Add spark-compat mode to integrate datafusion-spark features au… #1416 (mattcuento)
feat: add Dataframe.cache() factory (no planner handling) #1420 (killzoner)
feat: Adaptive query execution (AQE) planner fundamentals #1372 (milenkovicm)
feat: Make push scheduling policy default as it has lower latency #1461 (milenkovicm)
feat: job scheduling with push based job status updates #1478 (milenkovicm)

Fixed bugs:

fix: compile issue after unsuccessful merge #1402 (milenkovicm)
fix: prost build keda and TLS RPC example #1429 (killzoner)
fix: remove scheduler_config_spec.toml as it is unused #1462 (milenkovicm)
fix: Don't use maxrows as a "fetched rows" but calculate it from the batches #1480 (martin-g)

Documentation updates:

docs: fix outdated content in documentation #1385 (andygrove)
docs: use tpchgen-rs for TPC-H data generation #1390 (andygrove)
docs: add Jupyter notebook support documentation #1399 (andygrove)
chore: Document ballista features in README.md #1418 (mattcuento)

Merged pull requests:

feat: add config option for skipping arrow ipc read validation #1374 (killzoner)
docs: fix outdated content in documentation #1385 (andygrove)
restrict python CI to python directory #1383 (Huy1Ng)
perf: optimize shuffle writer with buffered I/O and fix file size bug #1386 (andygrove)
docs: use tpchgen-rs for TPC-H data generation #1390 (andygrove)
feat: improve tpch benchmark CLI #1391 (andygrove)
doc: Add Ballista extensions example to the docs. #1382 (LouisBurke)
feat: Add sort-based shuffle implementation #1389 (andygrove)
feat: New ballista python interface #1338 (milenkovicm)
doc: add more details for protobuf extension #1393 (LouisBurke)
feat: Add batch coalescing ability to shuffle reader exec #1380 (danielhumanmod)
docs: add Jupyter notebook support documentation #1399 (andygrove)
feat: Add arrow flight proxy to scheduler #1351 (sebbegg)
chore: update datafusion to 52 #1394 (killzoner)
feat: Creating SubstraitSchedulerClient and standalone Substrait examples #1376 (mattcuento)
fix: compile issue after unsuccessful merge #1402 (milenkovicm)
feat: Cluster RPC customisations to support TLS and custom headers #1400 (phillipleblanc)
chore: Document ballista features in README.md #1418 (mattcuento)
fix: prost build keda and TLS RPC example #1429 (killzoner)
Improve sort-based shuffle: single spill file per partition and batch coalescing #1431 (andygrove)
feat: add -c config override flag to tpch benchmark #1435 (andygrove)
feat: Extract execution_graph to a trait #1361 (milenkovicm)
chore: add confirmation before tarball is released [#1445](https://redirect.github.com/apache/datafusion-b

❗ Important

✂ PR body was truncated to here.

Configuration

📅 Schedule: (UTC)

Branch creation
- At any time (no schedule defined)
Automerge
- At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

coderabbitai · 2026-01-20T00:37:00Z

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 0402fea3-c34b-4213-b15d-bc5e4d2e5c55

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

socket-security · 2026-06-08T15:55:44Z

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

socket-security · 2026-07-15T22:28:08Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	ballista@49.0.0 ⏵ 54.1.0	^-18
	datafusion@49.0.2 ⏵ 55.1.0	^-2
	tempfile@3.23.0 ⏵ 3.27.0	^-1

View full report

renovate Bot force-pushed the renovate/major-ballista-stack branch from 7913594 to 65412c7 Compare November 19, 2025 10:34

renovate Bot changed the title ~~fix(deps): update rust crate datafusion to v50~~ fix(deps): update rust crate datafusion to v51 Nov 19, 2025

renovate Bot force-pushed the renovate/major-ballista-stack branch from 65412c7 to e038316 Compare November 22, 2025 02:43

renovate Bot changed the title ~~fix(deps): update rust crate datafusion to v51~~ fix(deps): update ballista stack (major) Nov 22, 2025

renovate Bot force-pushed the renovate/major-ballista-stack branch from e038316 to 8b1ac8d Compare December 10, 2025 12:32

renovate Bot force-pushed the renovate/major-ballista-stack branch 2 times, most recently from db62ad5 to 83124c3 Compare January 3, 2026 17:54

renovate Bot force-pushed the renovate/major-ballista-stack branch from 83124c3 to b733b79 Compare January 12, 2026 18:04

renovate Bot force-pushed the renovate/major-ballista-stack branch from b733b79 to 987c022 Compare January 20, 2026 00:36

renovate Bot force-pushed the renovate/major-ballista-stack branch from 987c022 to 0639e9c Compare February 12, 2026 11:43

renovate Bot force-pushed the renovate/major-ballista-stack branch from 0639e9c to 077ad40 Compare February 25, 2026 11:03

renovate Bot force-pushed the renovate/major-ballista-stack branch from 077ad40 to 69c761b Compare March 7, 2026 21:38

renovate Bot changed the title ~~fix(deps): update ballista stack (major)~~ fix(deps): update ballista stack to v52 (major) Mar 7, 2026

renovate Bot force-pushed the renovate/major-ballista-stack branch from 69c761b to d7c966d Compare March 12, 2026 10:12

renovate Bot force-pushed the renovate/major-ballista-stack branch 2 times, most recently from 20b7f4f to f11c6f5 Compare March 24, 2026 01:46

renovate Bot changed the title ~~fix(deps): update ballista stack to v52 (major)~~ fix(deps): update ballista stack (major) Mar 24, 2026

renovate Bot force-pushed the renovate/major-ballista-stack branch from f11c6f5 to 679f40b Compare April 3, 2026 17:57

renovate Bot force-pushed the renovate/major-ballista-stack branch from 679f40b to ace5a0d Compare April 16, 2026 21:51

renovate Bot force-pushed the renovate/major-ballista-stack branch from ace5a0d to 15fd198 Compare May 18, 2026 09:28

renovate Bot force-pushed the renovate/major-ballista-stack branch from 15fd198 to 52cf943 Compare May 25, 2026 16:31

renovate Bot changed the title ~~fix(deps): update ballista stack (major)~~ fix(deps): update ballista stack to v53 May 25, 2026

renovate Bot force-pushed the renovate/major-ballista-stack branch from 52cf943 to b7d2125 Compare June 8, 2026 15:55

renovate Bot changed the title ~~fix(deps): update ballista stack to v53~~ fix(deps): update ballista stack (major) Jun 8, 2026

renovate Bot force-pushed the renovate/major-ballista-stack branch from b7d2125 to c4d660e Compare July 15, 2026 22:27

renovate Bot changed the title ~~fix(deps): update ballista stack (major)~~ fix(deps): update ballista stack to v54 Jul 15, 2026

renovate Bot force-pushed the renovate/major-ballista-stack branch from c4d660e to 58add45 Compare July 21, 2026 00:52

renovate Bot force-pushed the renovate/major-ballista-stack branch from 58add45 to 79e71fc Compare July 21, 2026 19:03

renovate Bot force-pushed the renovate/major-ballista-stack branch 2 times, most recently from d134340 to e3e2cd2 Compare August 11, 2026 21:36

renovate Bot force-pushed the renovate/major-ballista-stack branch from e3e2cd2 to 6393005 Compare August 18, 2026 21:38

renovate Bot changed the title ~~fix(deps): update ballista stack to v54~~ fix(deps): update ballista stack (major) Aug 18, 2026

renovate Bot force-pushed the renovate/major-ballista-stack branch 2 times, most recently from 8b38713 to 1554909 Compare September 2, 2026 16:57

renovate Bot force-pushed the renovate/major-ballista-stack branch 2 times, most recently from 1e7d6bb to 40e69e9 Compare September 11, 2026 19:52

renovate Bot force-pushed the renovate/major-ballista-stack branch from 40e69e9 to 8714fab Compare September 15, 2026 11:01

fix(deps): update ballista stack

60eda1b

renovate Bot force-pushed the renovate/major-ballista-stack branch from 8714fab to 60eda1b Compare September 16, 2026 10:39

Navigation Menu

fix(deps): update ballista stack (major) - #113

fix(deps): update ballista stack (major)#113
renovate[bot] wants to merge 1 commit into
masterfrom
renovate/major-ballista-stack

renovate Bot commented Nov 16, 2025 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jan 20, 2026 •

edited

Loading

Review skipped

Uh oh!

socket-security Bot commented Jun 8, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

Conversation

renovate Bot commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Notes

v54.1.0

Apache DataFusion Ballista 54.1.0 Changelog

Credits

v54.0.0

Apache DataFusion Ballista 54.0.0 Changelog

Credits

v53.0.0

Apache DataFusion Ballista 53.0.0 Changelog

Credits

v52.0.0

Apache DataFusion Ballista 52.0.0 Changelog

Configuration

Uh oh!

coderabbitai Bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

socket-security Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

renovate Bot commented Nov 16, 2025 •

edited

Loading

`v54.1.0`

`v54.0.0`

`v53.0.0`

`v52.0.0`

coderabbitai Bot commented Jan 20, 2026 •

edited

Loading

socket-security Bot commented Jun 8, 2026 •

edited

Loading

socket-security Bot commented Jul 15, 2026 •

edited

Loading