kafka-consumer(ticdc): tolerate replayed resolved and DDL events#12596
Conversation
|
Skipping CI for Draft Pull Request. |
ti-chi-bot
Bot
added
do-not-merge/work-in-progress
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request improves the Kafka consumer's resilience to replayed events by relaxing watermark fallback checks and introducing logical DDL deduplication. A critical feedback point highlights that the current deduplication logic for split DDLs (e.g., from RENAME TABLES) is insufficient when replaying sequences, as it only compares against the single most recent event. The reviewer suggests using CommitTs and Seq ordering to correctly identify and ignore replayed DDLs.
ti-chi-bot
Bot
removed
the
do-not-merge/work-in-progress
|
/retest |
|
/retest |
3 similar comments
|
/retest |
|
/retest |
|
/retest |
ti-chi-bot
Bot
added
needs-1-more-lgtm
|
/test pull-cdc-integration-storage-test |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: 3AceShowHand, lidezhu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
lgtm
and removed
needs-1-more-lgtm
[LGTM Timeline notifier]Timeline:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more. @@ Coverage Diff @@
## master #12596 +/- ##
===========================================
Coverage ? 53.7381%
===========================================
Files ? 1011
Lines ? 141168
Branches ? 0
===========================================
Hits ? 75861
Misses ? 59568
Partials ? 5739 🚀 New features to boost your workflow:
|
|
/retest |
4 similar comments
|
/retest |
|
/retest |
|
/retest |
|
/retest |
wlwilliamx
added
needs-cherry-pick-release-7.5
|
In response to a cherrypick label: new pull request created to branch |
|
In response to a cherrypick label: new pull request created to branch |
|
In response to a cherrypick label: new pull request created to branch |
4 participants
What problem does this PR solve?
Issue Number: close #12595
What is changed and how it works?
cmd/kafka-consumeras duplicate delivery instead of a fatal errorCheck List
Tests
Questions
Will it cause performance regression or break compatibility?
No. This only makes the standalone Kafka consumer tolerate duplicate MQ delivery in line with TiCDC's at-least-once behavior.
Do you need to update user documentation, design documentation or monitoring documentation?
No.
Release note