Skip to content
This repository was archived by the owner on Jan 5, 2024. It is now read-only.
This repository was archived by the owner on Jan 5, 2024. It is now read-only.

Unable to use hudi sink #22

@ankitdimania-eligible

Description

@ankitdimania-eligible

I'm trying to use Hudi sink but the write to Hudi-client is not working. Has anyone seen this issue before:

2019-10-08 20:29:46 INFO  HoodieTableConfig:69 - Loading dataset properties from /tmp/hoodie/payment_reports_18/.hoodie/hoodie.properties
2019-10-08 20:29:46 INFO  HoodieTableMetaClient:94 - Finished Loading Table of type COPY_ON_WRITE from /tmp/hoodie/payment_reports_18
2019-10-08 20:29:46 INFO  HoodieTableMetaClient:96 - Loading Active commit timeline for /tmp/hoodie/payment_reports_18
2019-10-08 20:29:46 INFO  HoodieActiveTimeline:77 - Loaded instants java.util.stream.ReferencePipeline$Head@1a850eab
2019-10-08 20:29:46 INFO  HoodieActiveTimeline:212 - Marking instant complete [==>20191008202942__commit__INFLIGHT]
2019-10-08 20:29:46 INFO  HoodieActiveTimeline:377 - Created a new file in meta path: /tmp/hoodie/payment_reports_18/.hoodie/20191008202942.inflight
2019-10-08 20:29:46 ERROR JobDag:188 - Failed in JobDag
com.uber.hoodie.exception.HoodieIOException: Could not rename /tmp/hoodie/payment_reports_18/.hoodie/20191008202942.inflight to /tmp/hoodie/payment_reports_18/.hoodie/20191008202942.commit
        at com.uber.hoodie.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:330)
        at com.uber.hoodie.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:215)
        at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:518)
        at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:491)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink$HoodieWriteClientWrapper.commit(HoodieSink.java:489)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.commit(HoodieSink.java:290)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.commit(HoodieSink.java:267)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.write(HoodieSink.java:184)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.write(HoodieSink.java:161)
        at com.uber.marmaray.common.job.SingleSinkSubDag.executeNode(SingleSinkSubDag.java:51)
        at com.uber.marmaray.common.job.JobSubDag.execute(JobSubDag.java:149)
        at com.uber.marmaray.common.job.JobDag.execute(JobDag.java:171)
        at com.uber.marmaray.common.job.JobManager.lambda$null$0(JobManager.java:207)
        at com.uber.marmaray.common.job.ThreadPoolService$ThreadPoolServiceCallable.call(ThreadPoolService.java:415)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions