Describe the problem you faced
For datasets having very infrequent cleans, ECTR could go into archived timeline. with 1.x, we do have lsm timeline and hence should be able to add support to fetch the ECTR from lsm timeline if we can't find them in active timeline.
Ref patch that we added for 0.x to get around this:
#18380
To Reproduce
Expected behavior
If ECTR falls behind active timeilne, clean planner should seamlessly fetch it from lsm timeline and proceed w/ planning instead of doing full table planning.
Environment Description
- Hudi version:
- Spark version:
- Flink version:
- Hive version:
- Hadoop version:
- Storage (HDFS/S3/GCS..):
- Running on Docker? (yes/no):
Additional context
No response
Stacktrace
Describe the problem you faced
For datasets having very infrequent cleans, ECTR could go into archived timeline. with 1.x, we do have lsm timeline and hence should be able to add support to fetch the ECTR from lsm timeline if we can't find them in active timeline.
Ref patch that we added for 0.x to get around this:
#18380
To Reproduce
Expected behavior
If ECTR falls behind active timeilne, clean planner should seamlessly fetch it from lsm timeline and proceed w/ planning instead of doing full table planning.
Environment Description
Additional context
No response
Stacktrace