Skip to content

commitlog: Improve committed_meta#4338

Open
kim wants to merge 3 commits intomasterfrom
kim/commitlog/committed-meta
Open

commitlog: Improve committed_meta#4338
kim wants to merge 3 commits intomasterfrom
kim/commitlog/committed-meta

Conversation

@kim
Copy link
Contributor

@kim kim commented Feb 18, 2026

  • Extends commit::Metadata to include the checksum
  • Extends segment::Metadata to include Some(commit::Metadata)
    containing the last commit in the segment (if there is one)
  • Changes committed_meta to:
    • ignore empty segments at the end of the log
    • try harder to provide useful metadata, even if only a prefix of the
      latest segment is readable

This is allows to eliminate remaining Commitlog::open calls with the
purpose of querying the latest commit (offset). Commitlog::open
creates an empty segment if the tail of the log is corrupt, which is a
non-obvious side-effect that can be confusing when debugging.

It also allows to eliminate uses where the commits_from iterator is
used to find the latest full commit. The Commits iterator requires the
caller to handle the case of a corrupted commit at the end of the log,
by advancing the iterator once more after it has yielded an error in
order to check that it is exhausted, and then deciding whether to ignore
the error. This is easy to forget.

committed_meta now just does the right thing, preserving information
about tail corruption for when that's useful.

- Extends `commit::Metadata` to include the checksum
- Extends `segment::Metadata` to include `Some(commit::Metadata)`
  containing the last commit in the segment (if there is one)
- Changes `committed_meta` to:
  - ignore empty segments at the end of the log
  - try harder to provide useful metadata, even if only a prefix of the
    latest segment is readable

This is allows to eliminate remaining `Commitlog::open` calls with the
purpose of querying the latest commit (offset). `Commitlog::open`
creates an empty segment if the tail of the log is corrupt, which is a
non-obvious side-effect that can be confusing when debugging.

It also allows to eliminate uses where the `commits_from` iterator is
used to find the latest full commit. The `Commits` iterator requires the
caller to handle the case of a corrupted commit at the end of the log,
by advancing the iterator once more after it has yielded an error in
order to check that it is exhausted, and then deciding whether to ignore
the error. This is easy to forget.

`committed_meta` now just does the right thing, preserving information
about tail corruption for when that's useful.
@kim kim requested a review from Shubham8287 February 18, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments