Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#37

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-39f2
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#37
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-39f2

Conversation

@cursor

@cursor cursor Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

After v0.6.0 added HLC monotonic enforcement, restore_from_file() and gossip.receive() still assigned fresh HLC timestamps instead of preserving/restoring them from snapshots or gossip payloads.

Concrete trigger:

  1. Node writes remember("k", "v1", hlc=(5000, 10, "nodeA")) and snapshots to disk.
  2. After crash/restart, restore_from_file() assigns a fresh HLC (wall, 0, uuid) instead of (5000, 10, "nodeA").
  3. A causal successor write remember("k", "v2", hlc=(5000, 11, "nodeA")) raises TimestampRegressionsilent data loss.
  4. Gossip without HLC can overwrite fresher local data; gossip with stale HLC is not rejected.

Root cause

restore_from_file() constructed MemoryNode without the hlc field (defaulting to _hlc.now()), and gossip.receive() called remember() without passing hlc/ts_ns. bulk_write() had the same gap.

Fix

  • Add _parse_hlc() for snapshot/wire normalization with legacy ts_ns fallback
  • Restore HLC and advance global clock (_hlc.update()) on snapshot restore
  • Pass HLC through bulk_write and gossip.receive() with stale-update rejection
  • Hold lock during restore_from_file() and gc_expired() to prevent races

Validation

  • pytest tests/test_enterprise_backup.py tests/test_gossip.py tests/test_rust_brain.py tests/test_rust_brain_concurrency.py — 37 passed
  • New tests: test_restore_preserves_hlc_for_replication, gossip HLC acceptance/rejection tests
Open in Web View Automation 

After v0.6.0 added HLC monotonic enforcement, restore_from_file() and
gossip.receive() still assigned fresh HLC timestamps. Post-restore causal
writes were rejected (TimestampRegression) causing silent data loss; gossip
without HLC could overwrite fresher data.

- Add _parse_hlc() for snapshot/wire normalization with legacy fallback
- Restore HLC and advance global clock on snapshot restore
- Pass HLC through bulk_write and gossip receive with stale rejection
- Hold lock during restore_from_file and gc_expired to prevent races

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant