[OSS PR #18314] [HUDI-8371][CHERRYPICK] Fix column stats index with MDT for a few scenarios#40
Conversation
apache#9616) * [HUDI-6819] Fix logic for throwing exception in HoodieBackedTableMetadataWriter.
1. This commit allows users to disable metadata using write configs cleanly. 2. Valid instants consideration while reading from MDT is solid now. We are going to treat any special instant time (that has additional suffix compared to DT's commit time) as valid. Especially with MDT partition initialization, the suffix is dynamic, and so we can't really find exact match. So, might have to go with total instant time length and treat all special instant times as valid ones. In the LogRecordReader, we will first ignore any uncommitted instants. And then if it's completed in MDT timeline, we check w/ the instantRange. So it should be fine to return true for any special instant times.
…val (apache#9521) The purpose of 8849 change is to fix the ordering of rollbacks such that, the completion of rollback instant happens first followed by commits file removal from the timeline. For eg, if t5.c.inflight is partially failed, and t6.rb.requested is triggered to rollback. towards the completion, t6.rb is moved to completed state. and later all t5 commit files are removed from the timeline. This could lead to dangling commit files (t5) if the process crashes just after moving the t6 rollback to completion. So, 8849 also introduced polling completed rollbacks and ensure we don't trigger another rollback for t5. But we missed that we already landed 5148 which was addressing a similar issue. As per 5148, we first need to delete the commit files from timeline (t5) and then transition the rollback to completion (t6.rb). So, even if there is a crash, if we re-attempt t6.rb.requested, it will get to completion w/o any issues (even if t5 is not in the timeline at all). Hence reverting some of the core changes added as part of 8849. But there are some tests added and so not reverting the entire patch. --------- Co-authored-by: Jonathan Vexler <=> Co-authored-by: sivabalan <n.siva.b@gmail.com>
…llback metadata (apache#9653) [HUDI-6833] Add field for tracking log files from failed commit in rollback metadata
- Closing write clients properly in tests --------- Co-authored-by: sivabalan <n.siva.b@gmail.com>
- We face frequent flakiness around 2 modules (hudi-hadoop-mr and hudi-java-client). so, moving them out to github actions from azure CI. - Added explicit timeouts for few of deltastreamer continuous tests so that those fail instead of timing out. --------- Co-authored-by: sivabalan <n.siva.b@gmail.com>
…pache#9611) - We attempted a fix to avoid reading spurious log blocks on the reader side with apache#9545. When I tested the patch end to end, found some gaps. Specifically, the attempt Id we had with taskContextSupplier was not referring to task's attempt number. So, fixing it in this patch. Tested end to test by simulating spark retries and spurious log blocks. Reader is able to detect them and ignore multiple copies of log blocks.
…ySyncTool (apache#9650) Co-authored-by: jp0317 <zjpzlz@gmail.com>
…apache#9482) Adds schema evolution support to the BigQuerySyncTool by converting the Hudi schema into the BigQuery Schema format when creating and updating the table.
…entsHoodieIncrSource (apache#9538) Apply filtering before we start checkpoint batching. This change list will bring GCS job similar to S3 job. --------- Co-authored-by: Lokesh Lingarajan <lokeshlingarajan@Lokeshs-MacBook-Pro.local> Co-authored-by: sivabalan <n.siva.b@gmail.com>
… archival time window (apache#9666) When time travel query overlaps with cleaner or archival window, we should explicitly fail the query. If not, we might end up serving partial/wrong results or empty rows.
--------- Co-authored-by: Danny Chan <yuzhao.cyz@gmail.com>
When users explicitly defines primaryKey and preCombineField when CREATE TABLE, subsequent INSERT INTO will deduce the operation as UPSERT. --------- Co-authored-by: sivabalan <n.siva.b@gmail.com>
… full read of initial commit (apache#9473) This will happen in new onboarding as the old code will initialize prev=start = firstcommit-time, incremental read following this will always get entries > prev, which case we will skip part of first commit in processing. --------- Co-authored-by: Lokesh Lingarajan <lokeshlingarajan@Lokeshs-MacBook-Pro.local> Co-authored-by: sivabalan <n.siva.b@gmail.com>
This commits fix the Hive sync config by creating new HiveConf object every time when initializing HiveSyncConfig and adding hadoopConf as resource. We have to load Hadoop conf otherwise properties like `--conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory` won't be able to be passed via Spark Hudi job. Co-authored-by: Shawn Chang <yxchang@amazon.com>
Before this PR, the auto-tuning logic for dedup parallelism dictates the write parallelism so that the user-configured `hoodie.upsert.shuffle.parallelism` is ignored. This commit reverts apache#6802 to fix the issue.
* [HUDI-6993] Support Flink 1.18 (apache#9949) * Address build failures in older Flink Versions * Remove unnecessary dependency on flink-connector-hive * Fix Flink 1.18 Validate-bundles --------- Signed-off-by: Prabhu Joseph <joprabhu@amazon.com> Co-authored-by: Prabhu Joseph <joprabhu@amazon.com> Co-authored-by: root <root@ip-172-31-45-97.us-west-2.compute.internal> * [HUDI-7173] Fix hudi-on-flink read issues involving schema evolution and decimal types (apache#10247) * change back to 0.14.2-SNAPSHOT * [HUDI-8067] Use exec to run the IT (apache#11751) Co-authored-by: Jonathan Vexler <=> --------- Signed-off-by: Prabhu Joseph <joprabhu@amazon.com> Co-authored-by: Prabhu Joseph <prabhujose.gates@gmail.com> Co-authored-by: Prabhu Joseph <joprabhu@amazon.com> Co-authored-by: root <root@ip-172-31-45-97.us-west-2.compute.internal> Co-authored-by: voonhous <voonhousu@gmail.com> Co-authored-by: Jon Vexler <jbvexler@gmail.com>
We are looking to fix two problems by adding per log file marker. a. i. MOR data table rollbacks missed to sync original log files from failed commit to MDT. a.ii. Along these lines, if rollback instant is retried multiple times, any log files added from failed rollback attempts should also be synced to MDT. b. If there are spurious log files created even w/ successful commits, we need to ensure these spurious log files are also synced to MDT. So, to fix all of the above, we are adding per log file marker. Any log file added or appended to will create markers. We don't really need to distinguish between create and append and so we will go w/ APPEND IoType for markers. Co-authored-by: guanziyue <30882822+guanziyue@users.noreply.github.com> Co-authored-by: Lokesh Jain <ljain@apache.org> Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com> Co-authored-by: guanziyue <30882822+guanziyue@users.noreply.github.com> Co-authored-by: Lokesh Jain <ljain@Lokeshs-MacBook-Pro.local> Co-authored-by: Lokesh Jain <ljain@192.168.1.11>
…nged except version id (apache#17873) Co-authored-by: Shuo Cheng <njucshuo@gmail.com> Co-authored-by: Lokesh Jain <ljain@Lokeshs-MacBook-Pro.local> Co-authored-by: Lokesh Jain <ljain@192.168.1.11>
…8218) Adding support to recreate tables in hms based meta sync, when certain conditions are met.
* [HUDI-9088] Fix unnecessary scanning of target table in MERGE INTO on Spark (apache#12934) * [HUDI-8212] Add extra config of billing project ID for BigQuery sync (apache#11988) Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com> * [HUDI-9288] Fixing HoodieFileGroup api related to uncommitted slices (apache#13125) * [HUDI-6868] Support extracting passwords from credential store for Hive Sync (apache#10577) Co-authored-by: Danny Chan <yuzhao.cyz@gmail.com> * [MINOR] [BRANCH-0.x] Added condition to check default value to fix extracting password from credential store (apache#11247) * [HUDI-9039] Run do init table transaction only when required (apache#12847) Co-authored-by: Leon Lin <lliangyu@amazon.com> * [HUDI-9681] Remove mkdir in partition listing and add try catch to listStatus of partition (apache#13739) * [HUDI-9770] During hive/glue sync, ensure drop partition events are generated when partition is present in the metastore (apache#13794) * [HUDI-7478] Fix max delta commits guard check w/ MDT (apache#10820) Co-authored-by: Vova Kolmakov <vlvkolmakov@nsk.beeline.ru> * [HUDI-8161] Make spark-sql command 'desc' independent from schema evolution config (apache#11871) Co-authored-by: Vova Kolmakov <kolmakov.vladimir@huawei-partners.com> * Fix CI failures * Add some preliminary change * Fix a CI test * Fix validation bundle failures * Try update docker_java17_test.sh --------- Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com> Co-authored-by: Aditya Goenka <63430370+ad1happy2go@users.noreply.github.com> Co-authored-by: Tim Brown <tim@onehouse.ai> Co-authored-by: Danny Chan <yuzhao.cyz@gmail.com> Co-authored-by: Leon Lin <52585348+lliangyu-lin@users.noreply.github.com> Co-authored-by: Leon Lin <lliangyu@amazon.com> Co-authored-by: Roushan Kumar <roushankumar.nit@gmail.com> Co-authored-by: wombatu-kun <wombatukun@gmail.com> Co-authored-by: Vova Kolmakov <vlvkolmakov@nsk.beeline.ru> Co-authored-by: Vova Kolmakov <kolmakov.vladimir@huawei-partners.com>
…8216) --------- Co-authored-by: Aditya Goenka <ad1happy2go@gmail.com> Co-authored-by: Lin Liu <linliu.code@gmail.com>
--------- Co-authored-by: Lin Liu <linliu.code@gmail.com>
…geMetadata Convert HoodieRecord list to IndexedRecord before calling collectColumnRangeMetadata, matching the 3-arg signature in 0.14.x (master's version accepted HoodieRecord + Schema). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace Collector wildcard pattern with forEach+map in collectColumnRangeMetadata (HoodieTableMetadataUtil) and readRangeFromParquetMetadata (ParquetUtils) to fix Java 8 type inference failures - Replace FileSlice.hasLogFiles() with getLogFiles().findAny().isPresent() since hasLogFiles() doesn't exist in 0.14.x Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collect flatMap result to List before grouping to avoid raw type inference issue where Java 8 loses generic type parameter through the flatMap. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
These changes improve lock provider implementations including: Removing table name from lock keys to fix table rename issues New DynamoDB lock provider with hash-based partition keys New ZooKeeper lock provider with auto-derived partition keys Also fixes bugs related to serialization, NPE and DynamoDB protocol endpoint configuration issue Improvements: [HUDI-8005] Add new DynamoDB lock provider with hash-based partition keys [HUDI-8090] Add new ZooKeeper lock provider with auto-derived partition keys [[MINOR] Use ConfigUtils to handle defaults for all Zookeeper lock provider configs apache#13039] Use ConfigUtils to handle defaults for ZooKeeper lock provider configs [refactor: Remove Builder from DynamoDbBasedLockConfig apache#17780] Refactor DynamoDbBasedLockConfig to remove misapplied builder pattern --------- Co-authored-by: Davis-Zhang-Onehouse <169106455+Davis-Zhang-Onehouse@users.noreply.github.com> Co-authored-by: Lokesh Jain <ljain@Lokeshs-MacBook-Pro.local> Co-authored-by: Lokesh Jain <ljain@192.168.1.11> Co-authored-by: Nicolas Paris <nicolas.paris@adevinta.com> Co-authored-by: Vova Kolmakov <wombatukun@gmail.com> Co-authored-by: Vova Kolmakov <kolmakov.vladimir@huawei-partners.com> Co-authored-by: Tim Brown <tim@onehouse.ai> Co-authored-by: Lokesh Jain <ljain@192.168.1.7> Co-authored-by: voonhous <voonhousu@gmail.com> Co-authored-by: Lin Liu <linliu.code@gmail.com>
…d and add a guard for complex key generator (apache#18223) Fix the record key encoding with a single record key field and add a guard for complex key generator Cherry-picks fix for record key encoding regression in complex key generator: - Reverts breaking change in key encoding format for single record key field - Adds write config hoodie.write.complex.keygen.new.encoding to control encoding behavior - Adds validation during table initialization to prevent duplicates - Disables RLI and secondary index on affected tables (version 8 and below) for correctness - Affects tables upgraded to 0.14.1 using complex key generator with single field --------- Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com> Co-authored-by: danny0405 <yuzhao.cyz@gmail.com> Co-authored-by: Rahil Chertara <rchertara@gmail.com> Co-authored-by: Rahil C <32500120+rahil-c@users.noreply.github.com> Co-authored-by: Lokesh Jain <ljain@192.168.0.234> Co-authored-by: Lokesh Jain <ljain@Lokeshs-MacBook-Pro.local> Co-authored-by: Lokesh Jain <ljain@192.168.1.11> Co-authored-by: Pavithran Ravichandiran <paviravichennai@gmail.com> Co-authored-by: Lokesh Jain <ljain@192.168.1.57> Co-authored-by: Lokesh Jain <ljain@192.168.1.8> Co-authored-by: Lokesh Jain <ljain@apache.org>
[HUDI-9159] Add support for StorageBasedLockProvider Cherry-picks the following PRs into release-0.14.2: HUDI-9156: Add heartbeat manager for StorageBasedLockProvider HUDI-9158: Add storage based lock provider abstract implementation and constructor support HUDI-9160: Add GCS implementation of StorageBasedLockProvider HUDI-9159: S3 implementation of StorageLock for StorageBasedLockProvider HUDI-9122: Add RFC updates and tweak to S3 storage lock client HUDI-9158: Rename config from hoodie.write.lock.storage.heartbeat.poll.secs to hoodie.write.lock.storage.renew.interval.secs HUDI-9742: Add lock release metrics HUDI-9752: Add metrics for storage based lock provider HUDI-9782: Add audit logging scaffolding and implementation for storage based lock provider HUDI-9168: Refactor hudi-client-common to not import hudi-aws --------- Co-authored-by: Alex R <alex@onehouse.ai> Co-authored-by: Alex R <arheebikes@gmail.com> Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com> Co-authored-by: sivabalan <n.siva.b@gmail.com> Co-authored-by: Lokesh Jain <ljain@192.168.1.11> Co-authored-by: Jon Vexler <jbvexler@gmail.com> Co-authored-by: Lokesh Jain <ljain@Lokeshs-MacBook-Pro.local> Co-authored-by: Lokesh Jain <ljain@Lokeshs-MBP.lan> Co-authored-by: Lokesh Jain <ljain@192.168.0.234> Co-authored-by: Lokesh Jain <ljain@192.168.1.7> Co-authored-by: Danny Chan <yuzhao.cyz@gmail.com> Co-authored-by: Lokesh Jain <ljain@192.168.1.57> Co-authored-by: Lin Liu <linliu.code@gmail.com> Co-authored-by: Lokesh Jain <ljain@192.168.1.8>
…apache#18476) PR cherry-picks apache#18475 This PR adds comprehensive test coverage for clustering operations when there's a pending ingestion in a different partition, and fixes an issue with row writer clustering that was incorrectly using timestamp-based reads. The read code path for clustering row writer already filters based on the explicit file paths we are setting in params. So, removing the TIMESTAMP_AS_OF in the query. --------- Co-authored-by: Lokesh Jain <ljain@Lokeshs-MacBook-Pro.local>
…lstats_initializationfix-oss-cp
|
Important Review skippedToo many files! This PR contains 300 files, which is 150 over the limit of 150. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (300)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirror of apache#18314 for automated bot review.
Original author: @vamsikarnika
Base branch: release-0.14.2-prep