v0.13.0

Highlights

• A critical bug was discovered in MySQL, where recently committed changes are not always visible to new transactions. This was reproduced outside of Spirit, and an example script is available in #820. While we are still investigating the full scope of this issue, pragmatically it affects versions of MySQL that we will continue to need to support. Our workaround has been to:
  • No longer support MINIMAL row-based replication (new in this release)
  • Rely on the replication stream instead of DELETE/REPLACE statements to capture changes (default change, and earlier behavior is no longer supported in this release)
  • KeyAboveWatermark remains enabled, and is disabled internally prior to the checksum starting (unchanged)
  • Work by @morgo in #826 #819 #821 #823 #824 #808 #764 #750
• MySQL 9.7 is now supported, and tested in CI by @morgo in #698
• Throttling has been enhanced:
  • Spirit now supports multiple replica DSNs, and throttles on the slowest one, by @morgo in #744
  • A throttler based on Aurora commit latency has been added, by @morgo in #827
• The checksum process now yields automatically every 24hrs (configurable). This helps support InnoDB purge in cleaning up older versions. By @morgo in #703
• The --strict configuration option for schema changes is no longer recommended. The documentation has been updated to reflect this; no code changes to the option itself are planned at this time. By @morgo in #740
• Add support for GEOMETRY/SPATIAL indexes by @morgo in #687 #825
• Add a has_timestamp linter, which flags tables whose schema includes a TIMESTAMP column, by @morgo in #682
• Spirit will no longer resume from a checkpoint if it is considered too old. This helps reduce cases where starting from scratch is faster than replaying through a week of binary logs. By @morgo in #697
• Add support for schema changes which rename columns. By @morgo in #702 #709 #705
• Spirit can now manage tables with names up to MySQL's full 64-character identifier limit (previously capped at 56). Auxiliary table names are deterministically truncated when needed, and resume refuses to pick up a checkpoint that belongs to a different table after a truncation collision. By @morgo in #713 #830

Upgrade Notes

• Checkpoint compatibility: Complete any in-progress schema changes before upgrading, as the checkpoint format is not guaranteed to be stable between versions. This release in particular adds an original_table_name column to the checkpoint table to support 64-character table names.

Additional Changes

• A bug was discovered where Spirit could not correctly run on tables which contained columns using reserved words. Fix by @morgo in #829
• A bug was discovered where the threads configuration was not being propagated in buffered schema changes. Fix by @JashLal in #707
• A bug was discovered where buffered schema changes did not correctly handle empty binary values. Fix by @JashLal in #710
• MoveTable operations now support N:M reshard operations (API only). By @morgo in #680 #692 #693 #691 #681
• The test suite has been improved to make better use of helper functions, and fast fail on errors. By @morgo in #803 #802 #717 #727 #728 #729 #730 #731 #732 #733 #796 #793 #800 #798 #794 #801 #795 #797 #804 #799 #815 #813 #814 #817 #741
• Improvements to test stability by @morgo and @aparajon in #696 #762 #763 #768 #770 #774 #776 #806 #816 #775 #708 #745 #749 #759 #754 #755
• Improvements to linting and statement parsing API by @morgo and @aparajon in #689 #684 #753 #734 #694 #695 #685
• Improvements to test coverage by @morgo in #711 #683
• Documentation improvements by @morgo and @Kiran01bm in #699 #739 #832
• ci: skip workflows on documentation-only changes by @Kiran01bm in #679 #833

Full Changelog: v0.12.0...v0.13.0

v0.12.0

Highlights

• Introduced a new fmt subcommand by @morgo in #677 #678

Additional Changes

• The golang and golang-ci versions have been bumped, go 1.26 is now required. Fixed by @morgo in #672
• Remove noisy log messages in move operations by @morgo in #673
• Support atomic schema changes in buffered copies by @morgo in #674
• Minor API changes in statement parsing by @morgo in #675

Full Changelog: v0.11.3...v0.12.0

v0.11.3

Highlights

• Fixed an issue where authenticating with the MySQL native password plugin was disallowed @deanbaker in #665

Additional Changes

• Enhance load schema with filtering by @morgo in #663
• Add PlanChanges and DeclarativeToImperative to API by @morgo in #667

Full Changelog: v0.11.2...v0.11.3

v0.11.2

This release contains bug fixes over the previous release. v0.11.2 is now the recommended version for use in production.

Note: the 0.11.1 release was removed, after an issue with inheriting privileges from roles was discovered.

Highlights

• Spirit now supports privileges that are granted by the RDS specific rds_superuser_role role by @morgo in #659 #662
• Spirit now checks for unsafe runtime use of minimal RBR events, as well as static configuration checks on startup y @morgo in #651

Additional Changes

• Add statement classification helpers to pkg/statement by @aparajon in #652
• Add table.LoadSchemaFromDB by @morgo in #656
• Fixed an issue where constraints were not correctly diff'ed by @morgo in #657
• Fixed an issue where SET/ENUM data types were not correctly validated by @morgo in #660
• Various improvements to the test suite and code quality by @morgo and @Kiran01bm in #650 #653 #655 #661

Full Changelog: v0.11.0...v0.11.1

v0.11.0

Highlights

• Incompatible change: the command spirit is now spirit migrate. Additional subcommands include lint, diff and move. These features are now considered stable and ready to use by @morgo in #596 #610 #606 #609 #598 #604 #617 #625 #635 #649 #646
• Incompatible change: the --force-kill flag is now enabled by default. To disable it, use --skip-force-kill by @morgo in #605 #615 #643

Additional Changes

• spirit --version now reports the binary version by @fhe-ap and @Kiran01bm in #634 #616
• The composite chunker now supports Key above/Key below watermark optimizations by @Kiran01bm and @morgo in #590 #626
• Additional checks are now performed for set/enum datatypes by @morgo in #619
• Passwords with special characters were not handled correctly by @deanbaker in #593
• Dumping a checkpoint could panic when called too early in the migration process. Fix by @aparajon in #641
• The progress API now reports per-table progress in multi-table migrations by @aparajon in #640 #644
• The replication client now better validates binary log positions on resume. Fix by by @aparajon in #642
• Added mysql 9.6 support by @morgo in #603
• Logging refinements in the use of force-killing statements by @morgo in #637
• Improvements to documentation by @morgo and @aparajon in #595 #621
• Various improvements to the test suite and code quality by @morgo and @Kiran01bm in #639 #591 #599 #602 #622 #632 #594 #600 #614 #631 #633 #647

New Contributors

• @aparajon made their first contribution in #621
• @fhe-ap made their first contribution in #634
• @deanbaker made their first contribution in #593

Full Changelog: v0.10.2...v0.11.0

v0.10.2

This release contains bug fixes over the previous release. v0.10.2 is now the recommended version for use in production.

Highlights

• The --force-kill option did not take into account NULL processlist values, which can occur on replicas and systems where the event scheduler is running. Fixed by @fadyHemaya in #573

• The auto_increment property of a table was not correctly maintained when applying schema changes to empty tables. Fixed by @Kiran01bm in #583

• The BlockWait() function, which was used to ensure the binary log was advancing, could sometimes introduce performance issues be excessive use of FLUSH BINARY LOGS. Fixed by @eduardo-landin-mestre in #581

• Documentation has been added to many spirit packages, supporting its use as an API as well as a command line tool by @morgo in #566 #567 #570 #569 #576 #579 #584 #588 #589

Additional Changes

• Various bug fixes for the move command. We expect it to change from experimental to generally available shortly. Fixed by @morgo in #560 #562 #580
• Various test-stability and code quality fixes by @morgo in #564 #568 #571 #578 #575 #586

New Contributors

• @fadyHemaya made their first contribution in #573
• @eduardo-landin-mestre made their first contribution in #581

Full Changelog: v0.10.1...v0.10.2

v0.10.1

This release contains a number of bug fixes over the previous pre-release. v0.10.1 is now the recommended version for use in production.

Highlights

• Fixes an issue where binary log rotation events are not handled correctly in the replication applier. This may lead to issues where migrations that resume from checkpoint either skip events or fail by @morgo in #553 #545
• When the replication syncer connection fails, it did not always reconnect in a way that is recoverable by @morgo in #558
• Add support for loading credentials from file provided in command line by @brirams in #422
• The stability of MoveTables and buffered copying is improved. It remains experimental for now by @morgo in #549 #543 #544 #550 #557 #551 #547 #540 #533 #538 #539

Additional Changes

• Improved test infrastructure and GitHub Actions workflows by @Kiran01bm in #530
• Change LinterSeverity to int, add default Severity to all Violations by @kolbe in #532
• Fix handling of negative string values in NewDatum for unsigned types by @tyd in #541
• Bump required Go version to 1.25 and golang-ci linter to v2.6.2 by @morgo in #555, #531

Upgrade Notes

• Checkpoint compatibility: Complete any in-progress schema changes before upgrading, as the checkpoint format is not guaranteed to be stable between versions
• Go version: Requires Go 1.25 or later for building from source

Full Changelog: v0.10.0...v0.10.1

v0.10.0-prerelease

This release contains a large number of changes over the previous, and introduces the concept of experimental features to Spirit.

With experimental features:

• We will try to introduce them in ways where the code is well isolated, and you will always have to opt-in to using them.
• We will always document why they are still considered experimental.

Highlights

• Support for --force-kill to kill connections preventing lock acquisition by @kolbe, @morgo in #428 #458 #438 #506 #503
• Support for atomic multi-table changes by @morgo in #469 #431 #434 #437 #505 #482 #516
• The --force-inplace option has been removed, as we instead rely on parsing and explicitly allowing known safe inplace operations by @Kiran01bm, @kolbe in #470 #429
• The --checksum option has been removed and is now always enabled by @morgo in #489
• When flushing replication changes, Spirit now waits for the copier to advance past that point. This helps reduce potential contention scenarios at the cost of checkpoints sometimes being delayed. We call this optimization keyBelowLowWatermark by @morgo in #490

Additional Changes

• Improved TLS support by @Kiran01bm #443 #510
• Improved handling for cases where a table is externally modified, or can't write checkpoint handling by @morgo in #483
• The sentinel table option is now consistently named _spirit_sentinel #447 by @morgo
• Refactor alter visibility to the checks sub-system by @Kiran01bm in #484
• Fix RDS hostname detection to prevent potential subdomain spoofing by @Kiran01bm in #509
• move checksum retries into checksum package by @morgo in #507
• make tests parallel by @morgo in #514
• panic: could not convert datum to uint64 by @tyd in #518
• Have metadatalocks support multiple locks in one connection by @morgo in #525
• feat: Add maxRecreateAttempts logic by @tyd in #527
• Add additional info to ErrNotSupportedStatement error by @kolbe in #528
• Experimental support for buffered changes based on the DBLog algorithm by @morgo in #460 #453 #452 #459 #473
• Experimental support for native linting by @kolbe #486 #492 #501 #511 #529 #500
• Experimental support for a move command by @morgo #467 #496 #515 #523

New Contributors

• @tyd made their first contribution in #518

Full Changelog: v0.9.0...v0.10.0

v0.9.2

This addresses a critical issue where after several retries (handling errors such as deadlocks or lock waits), the replication applier will drop changes that could not be applied.

In many cases, this corruption would have been caught by the checksum, but regardless, all users are encouraged to upgrade immediately.

See issue #472

v0.9.1

This addresses a critical issue where updating rows such that the value of a PRIMARY KEY column changed were not tracked correctly, potentially leading to changes being lost in the new table.

In many cases, this corruption would have been caught by the checksum, but regardless, all users are encouraged to upgrade immediately.

See issue #417