Skip to content

Add support for reading Hudi tables using FileGroupReader#1

Merged
codope merged 2 commits into
onehouseinc:hudi-1.0.xfrom
voonhous:ctty/hudi-trino-connector
Apr 15, 2025
Merged

Add support for reading Hudi tables using FileGroupReader#1
codope merged 2 commits into
onehouseinc:hudi-1.0.xfrom
voonhous:ctty/hudi-trino-connector

Conversation

@voonhous
Copy link
Copy Markdown

@voonhous voonhous commented Mar 19, 2025

Description

Additional context and related issues

Added support for reading Hudi tables using FileGroupReader

  • Added synthesised column handlers
  • Added $path synthesized column handle test
  • Added a Hudi MOR test table with multiple fg per partition
  • Added partition pruning tests
  • Upgrade hudi 1.0.1
  • Fix compilation post upgrade
  • Removed unused functions in HudiPageSourceProvider
  • Migrate to getNextSourcePage API
  • Fix compilation errors after rebasing to 427

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

@voonhous voonhous changed the title Added support for reading Hudi tables using FileGroupReader Add support for reading Hudi tables using FileGroupReader Mar 19, 2025
@voonhous voonhous changed the title Add support for reading Hudi tables using FileGroupReader WIP: Add support for reading Hudi tables using FileGroupReader Mar 19, 2025
@voonhous voonhous force-pushed the ctty/hudi-trino-connector branch from af8c162 to 77f6627 Compare March 20, 2025 02:38
Copy link
Copy Markdown

@codope codope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@voonhous Changes look good and I am ok with your commits. However, I had a few comments on Ethan's changes - codope#26
We will have to rebase this PR after those comments are addressed. However, this is good to go for benchmarking.

@voonhous
Copy link
Copy Markdown
Author

voonhous commented Mar 20, 2025

No problem, once those lands, will rebase again! If that's the case, let me do the rebase properly again.

@voonhous voonhous force-pushed the ctty/hudi-trino-connector branch from 77f6627 to 79f8766 Compare March 20, 2025 06:25
- Added synthesised column handlers
- Added $path synthesized column handle test
- Added a Hudi MOR test table with multiple fg per partition
- Added partition pruning tests
- Upgrade hudi 1.0.1
- Fix compilation post upgrade
- Removed unused functions in HudiPageSourceProvider
- Migrate to getNextSourcePage API
- Fix compilation errors after rebasing to 427

Co-authored-by: Shawn Chang <yxchang@amazon.com>
@voonhous voonhous force-pushed the ctty/hudi-trino-connector branch from 79f8766 to f3f686d Compare March 20, 2025 06:30
@voonhous voonhous changed the title WIP: Add support for reading Hudi tables using FileGroupReader Add support for reading Hudi tables using FileGroupReader Mar 21, 2025
@codope codope merged commit f3f686d into onehouseinc:hudi-1.0.x Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants