Rollup optimized by wenjiecn · Pull Request #1 · wenjiecn/bolt

wenjiecn · 2026-01-22T07:36:14Z

What problem does this PR solve?

Issue Number: close #xxx

Type of Change

🐛 Bug fix (non-breaking change which fixes an issue)
✨ New feature (non-breaking change which adds functionality)
🚀 Performance improvement (optimization)
⚠️ Breaking change (fix or feature that would cause existing functionality to change)
🔨 Refactoring (no logic changes)
🔧 Build/CI or Infrastructure changes
📝 Documentation only

Description

Describe your changes in detail.
For complex logic, explain the "Why" and "How".

Performance Impact

No Impact: This change does not affect the critical path (e.g., build system, doc, error handling).

Positive Impact: I have run benchmarks.

Click to view Benchmark Results

Paste your google-benchmark or TPC-H results here.
Before: 10.5s
After:   8.2s  (+20%)

Negative Impact: Explained below (e.g., trade-off for correctness).

Release Note

Please describe the changes in this PR

Release Note:

Release Note:
- Fixed a crash in `substr` when input is null.
- optimized `group by` performance by 20%.

Checklist (For Author)

I have added/updated unit tests (ctest).
I have verified the code with local build (Release/Debug).
I have run clang-format / linters.
(Optional) I have run Sanitizers (ASAN/TSAN) locally for complex C++ changes.
No need to test or manual test.

Breaking Changes

No

Yes (Description: ...)

Click to view Breaking Changes

Breaking Changes:
- Description of the breaking change.
- Possible solutions or workarounds.
- Any other relevant information.

Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com>

…l.py, and update docs/config (bytedance#169) **1. Scripts Refactoring (`scripts/run-clang-tidy.py` & `scripts/util.py`):** * **Merged Dependencies:** Integrated functionality from `scripts/util.py` directly into `run-clang-tidy.py` to make it a standalone utility, and deleted `scripts/util.py`. * **Parallel Execution:** Added `-j/--jobs` support using `ThreadPoolExecutor` to run checks in parallel, significantly speeding up analysis. * **New Features:** * Added recursive directory scanning via `-d/--directory`. * Added file exclusion via `--exclude` (defaults to ignoring tests/benchmarks). * Added support for custom config files (`--config-file`) and binary paths (`--clang-tidy-binary`). * **Logic Improvements:** Enhanced build path detection and batch processing logic. **2. Documentation (`CONTRIBUTING.md`):** * **Revamped Guide:** Completely rewrote the "Static Analysis" section to include Prerequisites, Basic/Advanced Usage, Focused Checks, and Auto-Fixing. * **New References:** Added a comprehensive "Script Options Reference" table and CI/CD integration details. **3. Configuration (`.clang-tidy`):** * **Relaxed Rules:** * Disabled `WarningsAsErrors` (changed from `"*"` to `""`) to prevent blocking builds on non-critical warnings. * Allowed `#pragma once` by adding `-portability-avoid-pragma-once`. * Removed `abseil-*` checks. * Removed `readability-function-cognitive-complexity.IgnoreMacros`.

…brary (bytedance#173)

…(Velox commit: 7d33a6a, author: Ke Jia) (bytedance#177)

bytedance#175)

…bytedance#181)

…ing spill (bytedance#185)

This commit introduces the decryption feature for Parquet reader. To use the decryption feature, you need to implement the KMS class for key retrieval. The following files are partially adapted from Arrow: bolt/dwio/parquet/encryption/EncryptionInternal.h bolt/dwio/parquet/encryption/EncryptionInternal.cpp bolt/dwio/parquet/encryption/InternalFileDecryptor.h bolt/dwio/parquet/encryption/InternalFileDecryptor.cpp bolt/dwio/parquet/encryption/EncryptionType.h bolt/dwio/parquet/thrift/ThriftInternal.h

…ded a pipeline to run scripts/setup-dev-en.sh (bytedance#184)

Signed-off-by: fangzhuhe <fzhedu@gmail.com> Signed-off-by: fangzhuhe <fangzhuhe@bytedance.com>

)

…up. (bytedance#213)

…ance#201) Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

…tion (bytedance#236)

…pruning fallback when LogicalType or ConvertedType is absent (bytedance#242)

…ce#243)

…bolt internal repeat VectorFunction returning ARRARY(VARCHAR) when running with Gluten (bytedance#249)

Co-authored-by: Lu Niu <lniu@pinterest.com>

…tedance#268) Co-authored-by: zhli1142015 <zhli@microsoft.com>

Co-authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>

bytedance#269) Co-authored-by: zhli1142015 <zhli@microsoft.com>

wenjiecn and others added 30 commits January 15, 2026 09:15

rollup_optimized

54e974a

Update bolt/exec/LocalPlanner.cpp

b6e31b3

Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com>

fix clang-format

9825d64

del the sve & arm macro

9964953

Merge branch 'main' into rollup_optimized

c1e5212

fix clang-format

15043bd

[fix][shuffle] make shuffle ut ends with _test to avoid install as li…

f6b43a5

…brary (bytedance#173)

chore(ci): Use pre-commit checks in CI (bytedance#120)

ba88dc4

Fix missing of 'bytedance::bolt::filesystems::registeredFilesystems' …

9a625d5

…(Velox commit: 7d33a6a, author: Ke Jia) (bytedance#177)

chore: make error message 'hasBeenProcessedRowCountStr can't be empty… (

f2dcfa9

bytedance#175)

Enable UT under memory/sparksql (bytedance#182)

768dbc4

[Fix][Gluten][Mem] Add missing logic in MemoryManager::createRootPool (…

68a4598

…bytedance#181)

fix(shuffle): avoid shuffle writer OOM caused by minMemLimit_ prevent…

1f988e5

…ing spill (bytedance#185)

Add full dynamic column map reader support (bytedance#179)

42852a2

Update community section with Slack channel link (bytedance#186)

f848047

Rewrite dynamic column map reader to be almost zero copy (bytedance#176)

3582e77

[Chore][Setup] Fix the issue with incorrect conda installation and ad…

25baae8

…ded a pipeline to run scripts/setup-dev-en.sh (bytedance#184)

refactor: Introduce ConnectorNames.h in connectors (bytedance#156)

e3b0111

fix: try catch execptions in folly::guard (bytedance#140)

4bf8415

Signed-off-by: fangzhuhe <fzhedu@gmail.com> Signed-off-by: fangzhuhe <fangzhuhe@bytedance.com>

fix(shuffle): Evict shuffle composite RowVector after inited (bytedan…

4b1f30c

…ce#192)

[Fix][Gluten][Mem] Reject allocating mem quota when cv timeout (byted…

6269939

…ance#193)

fix storage adapters export (bytedance#196)

6e3ae61

[refactor] Modularize cmake targets (bytedance#189)

f6659b1

add sve support for isAscii (bytedance#132)

c76169e

Fix UAF in scan preloading and add OOM protection (bytedance#167)

c8d8555

[fix]:Correct Typos in Code and Documentation (bytedance#199)

4b17e73

fix: force the linker to import the symbols needed by JIT (bytedance#207

2cb65cc

)

[ci]: Generate compile db before run pre-commit clang-tidy hook (byte…

4e4a6a1

…dance#205)

kexianda and others added 30 commits February 12, 2026 00:02

[Chore] export link flags for JIT in conanfile (bytedance#220)

e3db266

ci: Unify behavior for run-clang-tidy in CI and local (bytedance#217)

b9480fb

Add merge_group target to CI workflows (bytedance#216)

63e8e86

feat: dynamic arm cpu detection (bytedance#202)

1dc14e2

feat(shuffle): Add codec in bolt to support checksum during shuffle (b…

0ab4ce7

…ytedance#180)

fix s3 dependencies in CMake (bytedance#230)

65dd3c8

Add a Flush API to the Parquet writer to flush the buffered row gro…

e304347

…up. (bytedance#213)

[feat]: use sonic json in json functions (bytedance#215)

460066b

Refactor AsyncThreadCtx to use shared_ptr and state management (byted…

d81f3d6

…ance#201) Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

ci: Compile s3 filesystem to cover clang-tidy for s3 (bytedance#233)

4c03d8b

fix result mismatch caused by missing column filter (bytedance#232)

38a975f

fix: reset cursor to avoid failure of LocalPartitionTest.earlyCancela…

a708dd2

…tion (bytedance#236)

fix: init rows with nulls in Paimon PartialUpdate (bytedance#234)

a2caa7c

fix(parquet): propagate ConvertedType to ScanSpec and enable VARCHAR …

be68938

…pruning fallback when LogicalType or ConvertedType is absent (bytedance#242)

Adding paimon reader benchmark (bytedance#244)

e083913

Run benchmarks with RelWithDebInfo config instead of Release (bytedan…

e81d70e

…ce#243)

Parallelizing pre-checkin x{CPU_NUM} speedup (bytedance#246)

7bb60e5

[fix] rewrite Spark repeat function as a VectorFunction to overwrite …

2c9bd94

…bolt internal repeat VectorFunction returning ARRARY(VARCHAR) when running with Gluten (bytedance#249)

[build]: fix Mac clang-17 compiler warnings (bytedance#254)

09cea0b

feat: Add Spark trunc function (bytedance#240)

2f7909b

Fix build and unit test failures on macOS (bytedance#266)

833beea

fix: define missing variable on ARM (bytedance#239)

c109ba0

Merge branch 'main' into rollup_optimized

337fc85

update code for review

268cec6

add UT for rollup optimized

22994bc

[fix][cp] Support ENUM type in parquet reader (bytedance#253)

57b5ac3

Co-authored-by: Lu Niu <lniu@pinterest.com>

[perf][cp] Improve Spark comparison functions performance by SIMD (by…

6f0b3ac

…tedance#268) Co-authored-by: zhli1142015 <zhli@microsoft.com>

[chore][cp] fix typo Orimitive to Primitive (bytedance#256)

ef662b3

Co-authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>

[perf][cp] Support fast path for Spark comparison function in join fi… (

d748fb5

bytedance#269) Co-authored-by: zhli1142015 <zhli@microsoft.com>

Merge branch 'main' into rollup_optimized

ed1cb76

Navigation Menu

Repository navigation

Rollup optimized - #1

Rollup optimized#1
wenjiecn wants to merge 65 commits into
mainfrom
rollup_optimized

wenjiecn commented Jan 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

Conversation

wenjiecn commented Jan 22, 2026

What problem does this PR solve?

Type of Change

Description

Performance Impact

Release Note

Checklist (For Author)

Breaking Changes

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants