Skip to content

Conversation

@Besroy
Copy link
Contributor

@Besroy Besroy commented Dec 12, 2025

Add manual snapshot creation API for log truncation control with 3 capabilities:

  1. Update truncation upper limit: Allows truncating logs up to a specified threshold when the in-memory truncation_upper_limit is zero (e.g., after restart).
  2. Trigger snapshot via NuRaft API: Leverages NuRaft's snapshot creation AP: Triggers snapshot creation immediately based on current commit idx. NuRaft handles race conditions internally so we don't need to do more (e.g., rejecting concurrent snapshot requests and using distance checks and flags to prevent concurrency issues)
  3. support truncate logs upto compact_lsn: if the compaction in raft didn't truncate logs as much as expected, it will trigger compact again to truncate logs. Please set wait_for_commit=true to make sure truncated logs have been committed.

@codecov-commenter
Copy link

codecov-commenter commented Dec 12, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 20.68966% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.44%. Comparing base (1a0cef8) to head (ee8a6c1).
⚠️ Report is 302 commits behind head on master.

Files with missing lines Patch % Lines
src/lib/replication/repl_dev/raft_repl_dev.cpp 28.57% 13 Missing and 2 partials ⚠️
src/lib/replication/service/raft_repl_service.cpp 0.00% 6 Missing ⚠️
src/lib/replication/repl_dev/solo_repl_dev.h 0.00% 1 Missing ⚠️
src/lib/replication/service/generic_repl_svc.cpp 0.00% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #840      +/-   ##
==========================================
- Coverage   56.51%   49.44%   -7.08%     
==========================================
  Files         108      110       +2     
  Lines       10300    11460    +1160     
  Branches     1402     5432    +4030     
==========================================
- Hits         5821     5666     -155     
+ Misses       3894     2096    -1798     
- Partials      585     3698    +3113     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Besroy
Copy link
Contributor Author

Besroy commented Dec 12, 2025

No need, found the root cause is we didn't save receiving_snapshot_ in srv_state

@Besroy Besroy closed this Dec 12, 2025
@Besroy Besroy reopened this Jan 4, 2026
@Besroy Besroy force-pushed the schedule_snapshot branch 3 times, most recently from a1d18dd to 4d2e80e Compare January 4, 2026 08:58
@Besroy Besroy force-pushed the schedule_snapshot branch 2 times, most recently from 9b8f510 to 0adfb81 Compare January 5, 2026 02:57
// Note that manual compact may cause concurrency issue with the automatic log compaction in raft server. Currently,
// it doesn't matter because this API is used in op scenario when members syncup and no new write are handled. If we
// want to use it in more phase, we need to add lock to avoid concurrency issue.
if (compact_lsn > 0 && m_compact_lsn.load() < compact_lsn) {
Copy link
Contributor Author

@Besroy Besroy Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we just want to compact logs to make sure new member catch up by BR, and compact during create_snasphot has been triggered, we don't need compact again here. If we want to truncate logs upto compact_lsn, we need it here.

@Besroy Besroy force-pushed the schedule_snapshot branch from 0adfb81 to 3d66023 Compare January 5, 2026 03:30
@Besroy Besroy force-pushed the schedule_snapshot branch from 3d66023 to ee8a6c1 Compare January 5, 2026 04:07
@Besroy Besroy requested a review from yuwmao January 5, 2026 04:55
@xiaoxichen xiaoxichen merged commit 4d66465 into eBay:master Jan 5, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants