Multipart snapshot read retries infinitely when validation fails due to <= in fallback query

[kollolsb]

Describe the bug

When a multipart snapshot is partially written to the persistor, the read-time validation retry enters an infinite loop. The affected node becomes permanently unrecoverable — it can never wake up, process messages, or overwrite the corrupt data.

To Reproduce

1. Run Quine with a remote persistor (Cassandra/Keyspaces) with many nodes in memory (10k+)
2. Trigger a process restart (ECS task restart, OOM kill, SIGKILL after SIGTERM timeout)
3. Quine's shutdown path tells all in-memory nodes to persist snapshots simultaneously, overwhelming the persistor's write capacity
4. Some multipart snapshot writes partially succeed (e.g., part 1 persisted but part 0 lost due to WriteTimeoutException)
5. After restart, any reference to an affected node enters an infinite read loop

Expected behavior

When a corrupt multipart snapshot is detected (validation fails), Quine should fall back to the next older valid snapshot, or return None if no older snapshot exists. The node should either recover from an older snapshot (+ journal if enabled) or start empty — not loop forever.

Context

We run Quine as an ECS service connected to AWS Keyspaces. When the ECS task restarts, Quine's shutdown path tells all in-memory nodes to sleep simultaneously (~10k+ nodes). This overwhelms Keyspaces' write capacity, causing WriteTimeoutException for a subset of nodes. Since multipart snapshot writes are non-atomic (each part is a separate CQL INSERT), a timeout can leave a partially-written snapshot — e.g., part 1 persisted but part 0 lost.

After the task restarts, any attempt to wake an affected node enters an infinite read loop that eventually throttles the Keyspaces partition, producing ReadTimeoutException. The shard retries wake-up 3000 times, each restarting the loop.

Relevant config:

• quine.store.type=keyspaces
• quine.persistence.journal-enabled=false
• Soft node limit: 50k
• Write timeout: 30s

Observed Logs

Shutdown write failures:

2026-06-11 08:38:51,786 INFO [pekko://graph-service/user/shard-1] com.thatdot.quine.graph.GraphShardActor - Shard #1 has 10000 node(s) awake.

com.thatdot.quine.persistor.WrappedPersistorException: Error calling PersistSnapshot(QuineId(224503EDB78C386381A5B4D103DE9D2A),EventTime(1780002062216|00003|001),58409)
Caused by: com.datastax.oss.driver.api.core.servererrors.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency LOCAL_QUORUM (2 replica were required but only 0 acknowledged the write)

After restart — corrupt partial snapshot detected, retry loops on same timestamp:

2026-06-11 08:40:58,765 WARN [NotFromActor] com.thatdot.quine.persistor.cassandra.aws.KeyspacesPersistor - Snapshot part has unexpected index: 1 (expected: 0)
2026-06-11 08:40:58,765 WARN [NotFromActor] com.thatdot.quine.persistor.cassandra.aws.KeyspacesPersistor - Snapshot part has unexpected count: 2 (expected: 1)
2026-06-11 08:40:58,789 WARN [NotFromActor] com.thatdot.quine.persistor.cassandra.aws.KeyspacesPersistor - Failed reading multipart snapshot for id: QuineId(98FD01658D8E3007B2D388A8CC0277EA) upToTime: EventTime(1781166924085|00000|001); retrying with time: EventTime(1781166924085|00000|001)

Loop eventually throttles the partition:

Caused by: com.datastax.oss.driver.api.core.servererrors.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded)

Shard retries wake-up 3000 times:

2026-06-11 08:40:58,825 INFO [pekko://graph-service/user/shard-1] com.thatdot.quine.graph.GraphShardActor - 2999 retries remaining waking up 98fd0165-8d8e-3007-b2d3-88a8cc0277ea. Retrying.

Root Cause

Snapshots.scala:47-50 defines the fallback query with <= (inclusive):

private val getLatestTimeBefore: SimpleStatement =
  getLatestTime
    .where(timestampColumn.is.lte)   // <= instead of <
    .build()

When MultipartSnapshotPersistenceAgent.getLatestSnapshot (PersistenceAgent.scala:292) retries with the corrupt snapshot's timestamp, the <= T query returns T itself (table has DESC clustering order, LIMIT 1). This re-reads the same corrupt parts, fails validation, and recurses with the same timestamp indefinitely.

Impact

• Affected nodes are permanently bricked (cannot wake up)
• The infinite loop prevents journal recovery even if journaling is enabled (StaticNodeSupport.scala:116 is never reached because getSnapshot() never completes)
• Each bricked node generates thousands of wasted persistor reads before the shard gives up
• Multiple bricked nodes compound the read load

Suggested Fix

Change Snapshots.scala:49 from lte to lt:

private val getLatestTimeBefore: SimpleStatement =
  getLatestTime
    .where(timestampColumn.is.lt)   // strictly less than
    .build()

This makes the retry find the next older snapshot or None (no older snapshot exists), allowing natural termination. With journaling enabled, None falls through to full journal replay. Without journaling, the node starts empty (data loss but not bricked).

Additional context

• The MultipartSnapshotPersistenceAgent trait explicitly documents that writes are non-atomic (PersistenceAgent.scala:253-255), making this a known-possible failure mode with no correct recovery path
• The shutdown write storm is the primary trigger: GraphShardActor.scala:622-627 calls sleepActor on all nodes in a tight loop with no backpressure, creating thousands of concurrent persistor writes

[cmeade]

Thank you kollolsb! That does look like a bug! We'll discuss our options for a combination of fixing the retry and reducing the write storm on shutdown.

[fredsensibill]

Hi! Wanted to check in on this. We've hit this issue 4 times now and it's been crashing our stack each time. Is there any update, or anything we can help with to move it along?

[cmeade]

An initial investigation led us to conclude that a holistic solution will be slightly different than the suggestion here. We created a ticket with some adjustments that's prioritized but still making its way through the queue.