feat: introduce CDC chunk dedup and fuse performance optimization - #2041
Open
imeoer wants to merge 6 commits into
Open
feat: introduce CDC chunk dedup and fuse performance optimization#2041imeoer wants to merge 6 commits into
imeoer wants to merge 6 commits into
Conversation
Squash-merge of the cdc-eval branch (PR dragonflyoss#2032 CDC chunk dedup plus fuse metadata/data path optimizations) rebased onto upstream/v3 after the blob footer redesign, record->entry rename, and typed algorithm enums: - blob meta: BlobMetadataCdcEntry table (56B entries), CHUNK_CDC flag, logical_block_count header field, owned + mmap storage, validation - builder: FastCDC 4K/16K/64K split, blake3 dedup, unique-stream append, block-group tail padding, cdc_dedup build stats, no-xattr compat bit - storage: CDC read path with coalesced entry fills, decoded-group LRU, whole-group publish sweep, all-ready mmap fast path, write_at_to; CDC branch for the diskless remote cache - fuse: readdirplus/no_open(dir)/parallel-dirops negotiation, dirent binary-search lookup, lazy dir handles, xattr ENOSYS shortcut Signed-off-by: imeoer <imeoer@gmail.com>
TestBench gains a v2-fuse mode (enabled by NYDUS_V2_NYDUSD and NYDUS_V2_IMAGE_BIN) that serves the same corpus as a RAFS v6 zstd image through the v2 nydusd, and the image build honors NYDUSFS_PERF_BLOCK_GROUP_SIZE. Refresh the README table with the five-column results measured after the fuse read/metadata path optimizations. Signed-off-by: imeoer <imeoer@gmail.com>
imeoer
requested review from
BruceAko,
anjia0532,
bergwolf,
chlins,
gaius-qi and
hhhhsdxxxx
and removed request for
a team
August 24, 2026 07:32
imeoer
marked this pull request as draft
August 24, 2026 07:49
Cutting: FastCDC now runs over each file's bytes as one stream (a carry buffer holds bytes past the last stable cut across chunk feeds), so cut points depend on file content alone. Pieces span chunk boundaries; elided zero runs and file ends finalize the pending bytes. This removes the forced cut at every chunk boundary and decouples dedup quality from --chunk-size entirely. Blob meta (CDC layout reworked, ~2x smaller, O(1) lookups): - hot entry shrinks 56 -> 12 bytes (logical u40 | unique u40 | size-1 u16), capping the CDC address spaces at 1 TiB - per-entry blake3 digests truncate to 16 bytes and move to a cold tail segment the runtime read path never pages in - a logical block index (4 bytes per 64 KiB window, derived from the entries, verified on load) replaces the per-read binary search with a bounded O(1) walk - segments are 8-byte aligned; offsets derive from existing header fields, the CRC covers everything Also honor --cache-dir when mounting a self-contained full blob: the upstream open_blob refactor pinned it to the diskless remote cache, which decoded a block group per read (4k random reads dropped from ~18k to ~1.2k IOPS). Signed-off-by: imeoer <imeoer@gmail.com>
Re-measured after the file-granular CDC and compact blob meta change; the fuse column now runs the 12-byte entry table with the O(1) logical block index. Signed-off-by: imeoer <imeoer@gmail.com>
none skips content-defined dedup entirely: chunk bytes stream straight into block groups (block padding stored physically since the group stream IS the logical space), and the blob meta degenerates to header plus block group table only (a few KiB regardless of image size), the same chunkless shape redirect blobs use. The bench harness gains an optional fuse-nocdc column behind NYDUS_BENCH_NOCDC_FUSE=1. Signed-off-by: imeoer <imeoer@gmail.com>
The blob metadata mapping tables are chunking-algorithm agnostic: an entry is a pure logical-to-unique byte range mapping, and how the builder picked the piece boundaries (FastCDC today, other content-defined or fixed schemes tomorrow) is a build-time policy the on-disk format does not encode. Rename CHUNK_CDC, BlobMetadataCdcEntry and the cdc_* accessor/helper family to dedup accordingly; the FastCDC algorithm code in the builder keeps its cdc names. No format change. Signed-off-by: imeoer <imeoer@gmail.com>
imeoer
marked this pull request as ready for review
August 25, 2026 06:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Please briefly describe the changes your pull request makes.
Related Issues
Please link to the relevant issue. For example:
Fix #123orRelated #456.Change Details
Please describe your changes in detail:
Test Results
If you have any relevant screenshots or videos that can help illustrate your changes, please add them here.
Change Type
Please select the type of change your pull request relates to:
Self-Checklist
Before submitting a pull request, please ensure you have completed the following: