CI: Add basic integration test by dveeden · Pull Request #12577 · pingcap/tiflow

dveeden · 2026-03-27T11:45:54Z

What problem does this PR solve?

Issue Number: close #12614

What is changed and how it works?

Check List

Tests

Integration test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

gemini-code-assist

Code Review

This pull request introduces a new basic integration test for Data Migration (DM), including SQL setup scripts, source/task configurations, and expected results. The feedback suggests using single quotes for SQL string literals to follow standards and replacing the deprecated block-allow-list configuration with the modern filters syntax in the task YAML.

gemini-code-assist · 2026-03-27T11:47:01Z

+INSERT INTO t1 VALUES (1, "test 1");
+INSERT INTO t1 VALUES (2, "test 2"), (3, "test 3");
+UPDATE t1 SET name="test 2 updated" WHERE id=2;


For consistency and adherence to SQL standards, it's better to use single quotes for string literals instead of double quotes.

INSERT INTO t1 VALUES (1, 'test 1'); INSERT INTO t1 VALUES (2, 'test 2'), (3, 'test 3'); UPDATE t1 SET name='test 2 updated' WHERE id=2;

gemini-code-assist · 2026-03-27T11:47:01Z

+    block-allow-list: "test"
+
+block-allow-list:
+  test:
+    do-dbs: ["test*"]


The block-allow-list configuration is deprecated. For new tests, please use the current filters syntax for better maintainability and to align with current best practices.

filter-rules: ["test"] filters: test: schema-pattern: "test*"

dveeden · 2026-04-19T19:37:07Z

/retest

dveeden · 2026-04-20T08:22:13Z

/retest

dveeden · 2026-04-20T08:45:36Z

/retest

dveeden · 2026-04-20T16:49:47Z

/retest

OliverS929 · 2026-04-21T07:14:37Z

+          ./bin/dmctl query-status mysql-to-tidb
+      - name: install mysql-tester
+        run: |
+          go install github.com/pingcap/mysql-tester/src@latest


Can we pin mysql-tester instead of installing @latest at runtime? Otherwise an unrelated upstream mysql-tester change can break this workflow without any TiFlow change. We can do manual updates if needed.

Yes, I'll do that. However as there are no releases nor tags for mysql-tester this is a bit more painful than it should.

Got it, thanks. I think my main concern was just avoiding @latest; pinning to a commit SHA is totally fine as well. It looks like the MariaDB job is pinned now, while the MySQL job is still floating. Non-blocking from my side.

dveeden · 2026-04-21T13:24:45Z

/retest

dveeden · 2026-04-21T13:41:03Z

/retest

dveeden · 2026-04-28T07:20:06Z

/retest

wuhuizuo · 2026-04-28T07:26:58Z

/cc gmhdbjd D3Hunter

wuhuizuo · 2026-04-28T07:32:48Z

+      - 'dm/**'
+      - 'cmd/dm*/*'


maybe it should be triggered when go.mod file is changed.

wuhuizuo · 2026-04-28T07:34:49Z

+          ./bin/dmctl query-status mariadb-to-tidb
+      - name: install mysql-tester
+        run: |
+          go install github.com/pingcap/mysql-tester/src@f2d90ea9522d30c9a8e8d70cc31c7f016ca2801f


can we manager the version in go.mod file?

Yes, we could do this by using the tool functionality, see also: https://go.dev/doc/go1.24#tools

Do you think this is better?

diff --git a/go.mod b/go.mod index aceb438e4..bba975342 100644 --- a/go.mod +++ b/go.mod @@ -180,6 +180,7 @@ require ( github.com/cncf/xds/go v0.0.0-20251022180443-0feb69152e9f // indirect github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce // indirect github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 // indirect + github.com/defined2014/mysql v0.0.0-20231121061906-fcfacaa39f49 // indirect github.com/envoyproxy/go-control-plane/envoy v1.35.0 // indirect github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect @@ -216,13 +217,14 @@ require ( github.com/oasdiff/yaml3 v0.0.0-20250309153720-d2182401db90 // indirect github.com/perimeterx/marshmallow v1.1.5 // indirect github.com/pingcap/metering_sdk v0.0.0-20251110022152-dac449ac5389 // indirect + github.com/pingcap/mysql-tester v0.0.0-20260121034350-f2d90ea9522d // indirect github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect github.com/qri-io/jsonpointer v0.1.1 // indirect github.com/qri-io/jsonschema v0.2.1 // indirect github.com/robfig/cron/v3 v3.0.1 // indirect github.com/segmentio/asm v1.2.0 // indirect github.com/segmentio/fasthash v1.0.3 // indirect - github.com/sergi/go-diff v1.3.1 // indirect + github.com/sergi/go-diff v1.4.0 // indirect github.com/spf13/afero v1.15.0 // indirect github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect github.com/tidwall/btree v1.7.0 // indirect @@ -462,3 +464,5 @@ replace golang.org/x/text => golang.org/x/text v0.28.0 // tls10server=1 godebug tlsrsakex=1 + +tool github.com/pingcap/mysql-tester/src

And then:

cd dm/tests/integration_basic go tool github.com/pingcap/mysql-tester/src

OliverS929

Other than these two small issues, the rest looks good to me.

OliverS929 · 2026-04-28T07:48:24Z

+  pull_request:
+    paths:
+      - 'dm/**'
+      - 'cmd/dm*/*'


Could we widen the pull_request.paths filter a bit? Changes in shared DM-related code or files like pkg/**, go.mod, go.sum, or this workflow itself can still affect make dm, but this workflow would not run for them.

I thought dm only used dm/pkg/ and not pkg/, but that's not the case. Maybe at some point the two pkg directories should be merged...

I've update the paths.

OliverS929 · 2026-04-28T07:51:28Z

+          ./bin/dmctl query-status mysql-to-tidb
+      - name: install mysql-tester
+        run: |
+          go install github.com/pingcap/mysql-tester/src@latest


Got it, thanks. I think my main concern was just avoiding @latest; pinning to a commit SHA is totally fine as well. It looks like the MariaDB job is pinned now, while the MySQL job is still floating. Non-blocking from my side.

dveeden · 2026-04-28T09:25:07Z

/retest

dveeden · 2026-04-28T09:40:22Z

/retest

ti-chi-bot · 2026-04-28T09:49:31Z

@dveeden: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-dm-integration-test-next-gen	`53d6577`	link	false	`/test pull-dm-integration-test-next-gen`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot · 2026-04-29T04:04:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: GMHDBJD, OliverS929

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [GMHDBJD,OliverS929]
~~dm/OWNERS~~ [GMHDBJD]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2026-04-29T04:04:35Z

[LGTM Timeline notifier]

Timeline:

2026-04-28 07:52:22.73977498 +0000 UTC m=+2670747.945135037: ☑️ agreed by OliverS929.
2026-04-29 04:04:34.320209095 +0000 UTC m=+2743479.525569142: ☑️ agreed by GMHDBJD.

ti-chi-bot Bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. area/dm Issues or PRs related to DM. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 27, 2026

gemini-code-assist Bot reviewed Mar 27, 2026

View reviewed changes

dveeden force-pushed the ci_tests branch 10 times, most recently from 916dd1a to 507fdaa Compare March 27, 2026 12:49

ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 27, 2026

dveeden force-pushed the ci_tests branch from 507fdaa to 0b72bc0 Compare March 27, 2026 12:51

ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 27, 2026

dveeden marked this pull request as draft March 27, 2026 12:52

ti-chi-bot Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 27, 2026

dveeden marked this pull request as ready for review March 27, 2026 13:20

ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 27, 2026

dveeden force-pushed the ci_tests branch 6 times, most recently from 9ef81ba to 6fe00a2 Compare March 27, 2026 14:02

OliverS929 reviewed Apr 21, 2026

View reviewed changes

dveeden added 3 commits April 21, 2026 14:45

Update based on review

aba7f40

Merge remote-tracking branch 'upstream/master' into ci_tests

c340eaa

Add readiness check

0e1c542

Update to MySQL 9.7

d154664

ti-chi-bot Bot requested review from D3Hunter and GMHDBJD April 28, 2026 07:27

wuhuizuo reviewed Apr 28, 2026

View reviewed changes

OliverS929 approved these changes Apr 28, 2026

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Apr 28, 2026

Update based on review

53d6577

GMHDBJD approved these changes Apr 29, 2026

View reviewed changes

ti-chi-bot Bot added lgtm approved and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 29, 2026

ti-chi-bot Bot merged commit 825344b into pingcap:master Apr 29, 2026
34 of 35 checks passed

Conversation

dveeden commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

dveeden commented Apr 19, 2026

Uh oh!

dveeden commented Apr 20, 2026

Uh oh!

dveeden commented Apr 20, 2026

Uh oh!

dveeden commented Apr 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dveeden commented Apr 21, 2026

Uh oh!

dveeden commented Apr 21, 2026

Uh oh!

dveeden commented Apr 28, 2026

Uh oh!

wuhuizuo commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OliverS929 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dveeden commented Apr 28, 2026

Uh oh!

dveeden commented Apr 28, 2026

Uh oh!

ti-chi-bot Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ti-chi-bot Bot commented Apr 29, 2026

Uh oh!

ti-chi-bot Bot commented Apr 29, 2026

[LGTM Timeline notifier]

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

dveeden commented Mar 27, 2026 •

edited

Loading

ti-chi-bot Bot commented Apr 28, 2026 •

edited

Loading