Repository navigation
raft: test followers killed during InstallSnapshot - #1
Open
coolstack-coder wants to merge 1 commit into
Open
coolstack-coder wants to merge 1 commit into
coolstack-coder wants to merge 1 commit into
Conversation
Kill a lagging follower at three points of a snapshot install (torn temp file, after the snapshot is saved, after the log is rewritten), let the leader compact past that snapshot, then restart the follower from disk. Recovery holds at all three points. Also fsync the directory after the atomic rename in writeAtomic, so a power loss cannot keep a compacted log without its snapshot.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On LinkedIn, Ram Mehta suggested a harder snapshot test: kill a follower partway through installing a snapshot, then make it catch up from a snapshot that has already been compacted past the index it needs. This PR adds that test.
What the test does
TestFollowerKilledDuringSnapshotInstallruns a 3-node cluster on realFileStorageand repeats the scenario at three crash points:mid_snapshot_write: the snapshot temp file is half written and the old snapshot is untouched.
after_snapshot_save: the new snapshot is on disk but the log has not been rewritten yet.
after_log_rewrite: snapshot and log are both on disk, but the FSM has not been restored yet.
For each one: the follower falls 60 writes behind, the leader starts sending it a snapshot, and the follower is killed at the crash point. While it is down, the leader takes 60 more writes and compacts past the snapshot the follower was installing. The follower then restarts from disk while still cut off, and the test checks what survived: a fully saved snapshot must be kept and a torn one ignored. Finally it reconnects and must converge on exactly the leader's state through a fresh snapshot.
A
crashingStoragewrapper simulates the SIGKILL by dropping every write after the crash point, and the in-memory transport drops the reply of a node that died mid-RPC.Result
Recovery held at all three crash points (20 runs each with
-race).To check that the test can actually fail, I temporarily made recovery ignore the saved snapshot. Both
after_snapshot_saveandafter_log_rewritethen failed withsaved snapshot 60 was lost on restart.Fix found while writing it
writeAtomicwrote and fsynced the temp file, then renamed it, but never fsynced the directory. A rename is only durable once its directory is synced, so after a power loss the rewritten log could survive while the snapshot it starts from does not. The node would then find a gap and drop its log. This PR syncs the directory after the rename whenever fsync is enabled.This cannot happen on a plain SIGKILL, since the page cache survives, and the new test does not simulate power loss. So this part is covered by the existing storage tests passing with fsync on, not by a new failing test.
Harness changes
The test cluster now takes a storage opener, so the same harness can run on memory or on disk. Existing tests still use
MemoryStorageand behave the same.How to run
The full suite (
go test -race ./...) passes.