Skip to content

raft: test followers killed during InstallSnapshot - #1

Open
coolstack-coder wants to merge 1 commit into
mainfrom
raft-snapshot-install-crash-test
Open

coolstack-coder wants to merge 1 commit into
mainfrom
raft-snapshot-install-crash-test

Conversation

@coolstack-coder

Copy link
Copy Markdown
Owner

Why

On LinkedIn, Ram Mehta suggested a harder snapshot test: kill a follower partway through installing a snapshot, then make it catch up from a snapshot that has already been compacted past the index it needs. This PR adds that test.

What the test does

TestFollowerKilledDuringSnapshotInstall runs a 3-node cluster on real FileStorage and repeats the scenario at three crash points:

mid_snapshot_write: the snapshot temp file is half written and the old snapshot is untouched.
after_snapshot_save: the new snapshot is on disk but the log has not been rewritten yet.
after_log_rewrite: snapshot and log are both on disk, but the FSM has not been restored yet.

For each one: the follower falls 60 writes behind, the leader starts sending it a snapshot, and the follower is killed at the crash point. While it is down, the leader takes 60 more writes and compacts past the snapshot the follower was installing. The follower then restarts from disk while still cut off, and the test checks what survived: a fully saved snapshot must be kept and a torn one ignored. Finally it reconnects and must converge on exactly the leader's state through a fresh snapshot.

A crashingStorage wrapper simulates the SIGKILL by dropping every write after the crash point, and the in-memory transport drops the reply of a node that died mid-RPC.

Result

Recovery held at all three crash points (20 runs each with -race).

To check that the test can actually fail, I temporarily made recovery ignore the saved snapshot. Both after_snapshot_save and after_log_rewrite then failed with saved snapshot 60 was lost on restart.

Fix found while writing it

writeAtomic wrote and fsynced the temp file, then renamed it, but never fsynced the directory. A rename is only durable once its directory is synced, so after a power loss the rewritten log could survive while the snapshot it starts from does not. The node would then find a gap and drop its log. This PR syncs the directory after the rename whenever fsync is enabled.

This cannot happen on a plain SIGKILL, since the page cache survives, and the new test does not simulate power loss. So this part is covered by the existing storage tests passing with fsync on, not by a new failing test.

Harness changes

The test cluster now takes a storage opener, so the same harness can run on memory or on disk. Existing tests still use MemoryStorage and behave the same.

How to run

go test -race -run TestFollowerKilledDuringSnapshotInstall -count=20 ./internal/raft/

The full suite (go test -race ./...) passes.

Kill a lagging follower at three points of a snapshot install (torn temp
file, after the snapshot is saved, after the log is rewritten), let the
leader compact past that snapshot, then restart the follower from disk.
Recovery holds at all three points.

Also fsync the directory after the atomic rename in writeAtomic, so a
power loss cannot keep a compacted log without its snapshot.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant