Repository navigation
Fix resume and revert data loss: register the DML listener before streaming - #1794
Open
CharlieW-22 wants to merge 1 commit into
Open
CharlieW-22 wants to merge 1 commit into
CharlieW-22 wants to merge 1 commit into
Conversation
On --resume and --revert, initiateStreaming() starts the binlog reader at a checkpoint before addDMLEventsListener() registers the listener for the original table. Row events read in between were neither decoded nor queued, so changes made while gh-ost was stopped (resume) or after the cut-over (revert) never reached rows already in the target table, and the cut-over made the stale table live. When resuming or reverting, register the listener inside initiateStreaming(), before the reader starts. Fresh migrations keep the current order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1793
Description
On
--resumeand--revert, gh-ost starts the binlog reader at a checkpoint before it registers the DML listener for the original table. Row events for that table read in between are never decoded or queued. So changes made while gh-ost was stopped (resume), or after the cut-over (revert), never reach rows that are already in the target table, and gh-ost cuts over and exits 0 with stale rows. The details and a standalone reproduction are in #1793.This PR registers the listener inside
initiateStreaming(), beforeInitDBConnections()starts the reader, when resuming or reverting. It also removes the later registration on those two paths:Migrate()skips it when resuming, andRevert()no longer calls it.Fresh migrations are unchanged. They keep the current order on purpose: there, the streamer has to deliver the
GhostTableMigratedchangelog event before anything drainsapplyEventsQueue, so an early DML listener could fill the queue and hold that event back.Things I checked:
applyEventsQueuejust pauses the reader untilexecuteWriteFuncs()(orexecuteDMLWriteFuncs()) drains it. I tried this locally with 3,000 changes waiting. The queue filled up during the hook, and both the resume and the revert finished with every change applied.--revertcan't be combined with--resume, andMigrate()never runs in revert mode, so every path adds the listener exactly once.migrator.gochange merges cleanly with Checkpoint only fully applied binlog transactions #1790, which changes whataddDMLEventsListener()registers but not when it's called. Fix: --resume without --execute drops the ghost/changelog tables #1770 adds tests in the same place inmigrator_test.go, so whichever of the two lands second will need a trivial rebase.Tests
Two regression tests in
MigratorTestSuite, both in GTID mode:TestResumeAppliesDMLWrittenWhileStoppedcheckpoints a migration while the cut-over is postponed and aborts it. It updates every third row while gh-ost is stopped, then resumes and cuts over. It then checks that the migrated table matches the old table row for row.TestRevertAppliesDMLWrittenAfterCutOvermigrates, updates every third row after the cut-over, and reverts. It then checks that the reverted table matches the migrated one row for row.Both resume or revert with a
gh-ost-on-validatedhook that runssleep 1. The hook runs inside the gap, so the race is lost every time: onmaster, both tests failed in every run, with all 34 updated rows stale. With this change, both passed in every run (6 of 6). Together they add about 20 seconds to the suite, mostly the resume test waiting for its first checkpoint (10 seconds, the interval the other checkpoint tests use).To run just these two (Docker needed, for the MySQL test container):
I also ran these locally:
script/cibuildpasses, with Go 1.25.12 and the MySQL 8.0.42 test container.script/docker-gh-ost-replica-testspasses all 90 tests onmysql:8.0.41, includinggtid-resume, which also passes onmysql:8.4.3andmariadb:11.8.8.GOMAXPROCS=1and exact-row-count cases. The official v1.1.11 binary lost updates in all 4 of its control runs on the same days.script/cibuildreturns with no formatting errors, build errors or unit test errors.