DM: Extend MySQL and MariaDB support for newer versions by dveeden · Pull Request #12628 · pingcap/tiflow

dveeden · 2026-05-07T09:40:53Z

Updated go-mysql

What problem does this PR solve?

Issue Number: closes #12629

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Support for newer versions of MySQL and MariaDB was extended

dveeden · 2026-05-07T09:41:31Z

/cc @OliverS929 @D3Hunter

gemini-code-assist

Code Review

This pull request updates the project to Go 1.25.9 and performs a significant upgrade of various dependencies, most notably go-mysql and tidb. The core changes involve refactoring GTID handling to align with the updated go-mysql library, including the transition from UUIDSet to MysqlGTIDSet and structural changes to MariaDB GTID sets. Feedback from the review highlights a potential side-effect in GTID incrementing that should be addressed by working on a copy of the data. Additionally, there are recommendations for improving code maintainability through the use of helper functions for repeated logic, removing redundant assignments, and simplifying loops where variables are unused.

gemini-code-assist · 2026-05-07T09:43:36Z

 		mariaGTID := singleGTID.(*gmysql.MariadbGTID)
 		mariaGTID.SequenceNumber++
 		gtidSet := new(gmysql.MariadbGTIDSet)
-		gtidSet.Sets = map[uint32]map[uint32]*gmysql.MariadbGTID{
-			mariaGTID.DomainID: {
-				mariaGTID.ServerID: mariaGTID,
-			},
+		gtidSet.Sets = map[uint32]*gmysql.MariadbGTID{
+			mariaGTID.DomainID: mariaGTID,
 		}
 		clone = gtidSet


In the MariaDB flavor case, mariaGTID is a pointer to an element within the original gSet. Modifying mariaGTID.SequenceNumber directly causes a side effect on the input gSet. To ensure GTIDIncrease is pure and doesn't modify its input, you should work on a copy of the GTID struct.

Suggested change

mariaGTID := singleGTID.(*gmysql.MariadbGTID)

mariaGTID.SequenceNumber++

gtidSet := new(gmysql.MariadbGTIDSet)

gtidSet.Sets = map[uint32]map[uint32]*gmysql.MariadbGTID{

mariaGTID.DomainID: {

mariaGTID.ServerID: mariaGTID,

},

gtidSet.Sets = map[uint32]*gmysql.MariadbGTID{

mariaGTID.DomainID: mariaGTID,

}

clone = gtidSet

mariaGTID := *singleGTID.(*gmysql.MariadbGTID)

mariaGTID.SequenceNumber++

gtidSet := new(gmysql.MariadbGTIDSet)

gtidSet.Sets = map[uint32]*gmysql.MariadbGTID{

mariaGTID.DomainID: &mariaGTID,

}

clone = gtidSet

gemini-code-assist · 2026-05-07T09:43:36Z

+		for u, tags := range *mSet {
+			sid = u
+			for _, intervals := range tags {
+				interval = intervals[0]
+			}
+		}


The logic for extracting the single SID and Interval from a MysqlGTIDSet is repeated here and in newGenerator (in generator.go). Since DM requires these sets to have exactly one entry (as enforced by verifySingleGTID), consider moving this extraction logic into a helper function to improve code reuse and maintainability.

gemini-code-assist · 2026-05-07T09:43:36Z

+			for tag, intervals := range tags {
+				intervals[0].Start++
+				intervals[0].Stop++
+				(*mSet)[u][tag] = intervals


This assignment is redundant because intervals is a slice, which is a reference type in Go. Modifying its elements on lines 167-168 already updates the data within the map. Removing this line would clarify that the modification is happening in-place on the cloned set.

gemini-code-assist · 2026-05-07T09:43:36Z

+		var sid uuid.UUID
+		var tags map[gmysql.Tag]gmysql.IntervalSlice
+		for sid, tags = range *mysqlGTIDs {
 		}
-		var uuidSet *gmysql.UUIDSet
-		for _, uuidSet = range mysqlGTIDs.Sets {
+		_ = sid


The variable sid is declared and assigned but not used, necessitating a blank assignment _ = sid to satisfy the compiler. This can be simplified by using a blank identifier directly in the range loop.

var tags map[gmysql.Tag]gmysql.IntervalSlice for _, tags = range *mysqlGTIDs { }

Updated go-mysql

dveeden · 2026-05-08T08:52:00Z

/retest

#4578) ### What Update TiFlow latest CI images to the newer Go 1.25 image tag: - `ghcr.io/pingcap-qe/ci/jenkins:v2026.4.12-10-gc29110c-go1.25` for jenkins job - `ghcr.io/pingcap-qe/ci/base:v2026.4.12-10-gc29110c-go1.25` for the Prow unit test job ### Why TiFlow PR pingcap/tiflow#12628 updates `go.mod` to `go 1.25.9`. Some CI jobs were still running with older Go 1.25 images, causing errors like: ```text compile: version "go1.25.9" does not match go tool version "go1.25.8" ``` ### Test result I have run replay tests for all failed tasks. The Go version mismatch issue is now fully resolved. The remaining failures are either integration test failures inherent to the TiFlow PR itself, or runtime issues on individual Jenkins/K8s agents. These are not directly related to the current upgrade of the Go image. --------- Signed-off-by: lyb <yebin.li@pingcap.com>

wuhuizuo · 2026-05-09T02:22:19Z

/test pull-verify

dveeden · 2026-05-11T06:19:44Z

/retest

ti-chi-bot · 2026-05-11T08:16:00Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lance6716 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dveeden · 2026-05-11T10:49:50Z

/retest

dveeden · 2026-05-11T11:17:05Z

/retest

go-mysql v1.15.0 (PR pingcap#1080) decodes upstream heartbeat events as a dedicated *replication.HeartbeatEvent type instead of GenericEvent. DM's switches on *replication.GenericEvent no longer match real heartbeats, so the validator never persists checkpoints, the syncer never flushes on idle, and the relay no longer marks heartbeats as ignored. This caused validator_basic to hang with non-zero pending rows. Accept both types since DM still injects its own heartbeats locally as GenericEvent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dveeden · 2026-05-11T15:52:04Z

/retest

The test asserted finished_bytes=59674 verbatim, but the tidb-pkg upgrade in this branch ships a slightly different lightning that reports 59754 bytes for the same dumped data. Replace the literal byte count with a backreference assertion that finished_bytes equals total_bytes (and is non-empty), so the test no longer breaks on benign size shifts from lightning. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dveeden · 2026-05-11T17:03:42Z

/retest pull-verify

ti-chi-bot · 2026-05-11T17:03:47Z

@dveeden: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test pull-build

/test pull-cdc-integration-kafka-test

/test pull-cdc-integration-mysql-test

/test pull-cdc-integration-pulsar-test

/test pull-cdc-integration-storage-test

/test pull-check

/test pull-dm-compatibility-test

/test pull-dm-integration-test

/test pull-error-log-review

/test pull-syncdiff-integration-test

/test pull-unit-test-cdc

/test pull-verify

/test wip-pull-unit-test-dm

/test wip-pull-unit-test-engine

The following commands are available to trigger optional jobs:

/test pull-dm-integration-test-next-gen

Use /test all to run the following jobs that were automatically triggered:

pingcap/tiflow/ghpr_verify

pingcap/tiflow/pull_cdc_integration_kafka_test

pingcap/tiflow/pull_cdc_integration_pulsar_test

pingcap/tiflow/pull_cdc_integration_storage_test

pingcap/tiflow/pull_cdc_integration_test

pingcap/tiflow/pull_dm_compatibility_test

pingcap/tiflow/pull_dm_integration_test

pingcap/tiflow/pull_dm_integration_test_next_gen

pingcap/tiflow/pull_syncdiff_integration_test

pull-build

pull-check

pull-error-log-review

pull-unit-test-cdc

Details

In response to this:

/retest pull-verify

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dveeden · 2026-05-11T17:04:05Z

/test pull-verify

dveeden · 2026-05-11T19:42:58Z

/retest

dveeden · 2026-05-12T05:39:36Z

/retest

go-sql-driver/mysql v1.8 (bumped from v1.7.1 in this branch) returns typed Go values (int64, float64, ...) when Scan'd into *interface{}, where v1.7 always returned []byte. ColumnsHolder.Values was scanned through *interface{} and the canal-json / open-protocol handle-key-only decoders type-assert .([]uint8) on every value, which now panics on any non-string column (caught by the canal_json_handle_key_only integration test). Scan each column into *[]byte via a private rawValues slot instead, so ints/floats/bools round-trip through database/sql's convertAssign and land as their textual representation - matching the pre-v1.8 contract that downstream decoders rely on. Use *[]byte rather than *sql.RawBytes because rows.Close (deferred) invalidates RawBytes before the caller reads the holder; *[]byte gets a fresh Go-owned slice. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ti-chi-bot · 2026-05-12T08:17:11Z

@dveeden: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-dm-integration-test-next-gen	`bbdf2ed`	link	false	`/test pull-dm-integration-test-next-gen`
pull-dm-integration-test	`bbdf2ed`	link	true	`/test pull-dm-integration-test`
pull-unit-test-cdc	`bbdf2ed`	link	true	`/test pull-unit-test-cdc`
pull-cdc-integration-kafka-test	`bbdf2ed`	link	true	`/test pull-cdc-integration-kafka-test`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot Bot added do-not-merge/needs-triage-completed release-note Denotes a PR that will be considered when it comes time to generate release notes. area/dm Issues or PRs related to DM. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 7, 2026

ti-chi-bot Bot requested review from D3Hunter and OliverS929 May 7, 2026 09:41

ti-chi-bot Bot removed the do-not-merge/needs-triage-completed label May 7, 2026

gemini-code-assist Bot reviewed May 7, 2026

View reviewed changes

dveeden force-pushed the tagged_gtid branch from 4b77445 to 4d018fa Compare May 7, 2026 11:21

DM: Support for Taggged GTIDs

92c0406

Updated go-mysql

dveeden force-pushed the tagged_gtid branch 2 times, most recently from e147389 to 59316e0 Compare May 7, 2026 12:10

Use Go 1.25.9

a6afe34

dveeden force-pushed the tagged_gtid branch from 59316e0 to a6afe34 Compare May 7, 2026 12:52

ti-chi-bot Bot added the do-not-merge/needs-linked-issue label May 7, 2026

wuhuizuo changed the title ~~DM: Extend MySQL and MariaDB support for newer versions~~ DM: Extend MySQL and MariaDB support for newer versions May 8, 2026

ti-chi-bot Bot removed the do-not-merge/needs-linked-issue label May 8, 2026

wuhuizuo changed the title ~~DM: Extend MySQL and MariaDB support for newer versions~~ DM: Extend MySQL and MariaDB support for newer versions May 8, 2026

lybcodes mentioned this pull request May 8, 2026

chore(pingcap/tiflow): update latest Go 1.25 CI images for tiflow jobs PingCAP-QE/ci#4578

Merged

dveeden added 3 commits May 11, 2026 09:24

Update errors_release.txt

00886e2

Updating TestMySQLLogger

e08e6ca

Updating TestJoinTables

62df4f2

ti-chi-bot Bot added the area/engine Issues or PRs related to Dataflow Engine. label May 11, 2026

Update TestAddGSetWithPurged

066ac74

Conversation

dveeden commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Uh oh!

dveeden commented May 7, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

dveeden commented May 8, 2026

Uh oh!

wuhuizuo commented May 9, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

ti-chi-bot Bot commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

ti-chi-bot Bot commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 12, 2026

Uh oh!

ti-chi-bot Bot commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dveeden commented May 7, 2026 •

edited

Loading