Skip to content

Adding support to fetch ECTR from lsm timeline #18536

@nsivabalan

Description

@nsivabalan

Describe the problem you faced

For datasets having very infrequent cleans, ECTR could go into archived timeline. with 1.x, we do have lsm timeline and hence should be able to add support to fetch the ECTR from lsm timeline if we can't find them in active timeline.

Ref patch that we added for 0.x to get around this:
#18380

To Reproduce

Expected behavior

If ECTR falls behind active timeilne, clean planner should seamlessly fetch it from lsm timeline and proceed w/ planning instead of doing full table planning.

Environment Description

  • Hudi version:
  • Spark version:
  • Flink version:
  • Hive version:
  • Hadoop version:
  • Storage (HDFS/S3/GCS..):
  • Running on Docker? (yes/no):

Additional context

No response

Stacktrace

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions