Fix Debezium logical converter for date to cover historical dates#469
Open
Navid Hassanzadeh (navidhsz) wants to merge 2 commits into
Open
Fix Debezium logical converter for date to cover historical dates#469Navid Hassanzadeh (navidhsz) wants to merge 2 commits into
Navid Hassanzadeh (navidhsz) wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR fixes a bug in the Debezium date conversion logic that caused historical dates (particularly those before 1582) to be incorrectly converted when handling Postgres Debezium date fields. The issue stemmed from using java.util.Date for conversion, which doesn't handle dates before the Gregorian calendar adoption correctly. The fix switches to using Java's LocalDate API for accurate date arithmetic across all calendar periods.
Key changes:
- Replaced
java.util.Date-based date conversion withLocalDate-based conversion inDateConverter - Added comprehensive parameterized tests covering historical dates from year 100 through the Gregorian calendar transition period
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| DebeziumLogicalConverters.java | Replaced legacy date conversion using java.util.Date with LocalDate API to correctly handle historical dates before 1582 |
| DebeziumLogicalConvertersTest.java | Added parameterized tests covering edge cases including leap years and the Julian-to-Gregorian calendar transition period |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…convert/logicaltype/DebeziumLogicalConverters.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
By default, Postgres Debezium uses adaptive mode for handling date fields. In this mode, any date type is encoded as an
Int32representing the number of days since the Unix epoch. Dates before the epoch are represented as negative numbers. When Debezium sends an integer-encoded date, the BigQuery connector converts it to a string representation. However, this conversion can fail for certain historical dates before1582, when the calendar changed from Julian to Gregorian. For example, the date1500-03-10is encoded as-171596, but BigQuery converts it to1500-02-29, which is not a valid date, causing BigQuery to reject the record and send it to the DLQ (if configured). Another example is1582-10-01, which is encoded as-141441by Debezium but shown as1582-09-21by the BigQuery connector.