Skip to content

Fixes to get full bootstrap working for non-hive-style partitioned tables and unit tests#28

Open
umehrot2 wants to merge 1 commit into
bvaradar:hudi_bootstrapfrom
umehrot2:umehrot2_hudi_bootstrap
Open

Fixes to get full bootstrap working for non-hive-style partitioned tables and unit tests#28
umehrot2 wants to merge 1 commit into
bvaradar:hudi_bootstrapfrom
umehrot2:umehrot2_hudi_bootstrap

Conversation

@umehrot2
Copy link
Copy Markdown

  • Fix in path generation to get it working for S3 paths
  • Schema fixes we discussed offline to maintain consistency

Tips

What is the purpose of the pull request

(For example: This pull request adds quick-start document.)

Brief change log

(for example:)

  • Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@codecov-commenter
Copy link
Copy Markdown

Codecov Report

Merging #28 into hudi_bootstrap will increase coverage by 0.00%.
The diff coverage is 0.44%.

Impacted file tree graph

@@                Coverage Diff                @@
##             hudi_bootstrap     #28    +/-   ##
=================================================
  Coverage              0.57%   0.57%            
  Complexity                2       2            
=================================================
  Files                   375     380     +5     
  Lines                 16482   16730   +248     
  Branches               1631    1674    +43     
=================================================
+ Hits                     94      96     +2     
- Misses                16385   16631   +246     
  Partials                  3       3            
Impacted Files Coverage Δ Complexity Δ
...java/org/apache/hudi/client/HoodieWriteClient.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...rg/apache/hudi/client/bootstrap/BootstrapMode.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...lient/bootstrap/BootstrapSourceSchemaProvider.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...i/client/bootstrap/FullBootstrapInputProvider.java 0.00% <ø> (ø) 0.00 <0.00> (ø)
...lient/bootstrap/MetadataBootstrapKeyGenerator.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
...ap/translator/IdentityBootstrapPathTranslator.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
...ator/MetadataBootstrapPartitionPathTranslator.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
...java/org/apache/hudi/config/HoodieWriteConfig.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
.../main/java/org/apache/hudi/keygen/KeyGenUtils.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
...main/java/org/apache/hudi/keygen/KeyGenerator.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65a4ac8...0b71fbf. Read the comment docs.

Comment thread hudi-spark/src/test/scala/TestDataSource.scala Outdated
Comment thread hudi-spark/src/test/scala/TestDataSource.scala Outdated
@umehrot2 umehrot2 force-pushed the umehrot2_hudi_bootstrap branch 2 times, most recently from 71cc484 to 520e30f Compare May 23, 2020 01:36
@umehrot2 umehrot2 force-pushed the umehrot2_hudi_bootstrap branch from 520e30f to 1d27c91 Compare May 23, 2020 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants