v3: CDC (content-defined chunking) chunk dedup for blob data

[Copilot]

Problem

Fixed-size chunking dedups poorly: a small insertion shifts every later byte, so near-identical content (e.g. two versions of the node binary) shares almost nothing. This adds opt-in content-defined chunking so duplicate data is stored once regardless of offset, within and across files/layers — with no changes to the EROFS-visible format.

Measured benefits (rootfs bytes, CDC 4K/16K/64K vs fixed chunking)

Inter-image (node:22.14.0–22.17.0-slim):

Scheme	Global dedup
fixed 1 MiB (today)	33.7%
fixed 4 KiB blocks	35.8%
CDC 4K/16K/64K	48.9%

• node binary across versions: fixed-4K 2.0% (offset shift) vs CDC 24–26%
• Upgrade increment 22.16→22.17: 61 MiB unique (CDC) vs 117 MiB (fixed); post-zstd 22 vs 66 MiB — ~3× smaller pull

Intra-image (cross-layer):

Image	fixed 1 MiB	CDC
node:22	8.0%	14.4%
golang	2.7%	6.9%
python	3.5%	10.1%
pytorch	7.0%	15.7% (~1 GiB, mostly CUDA fatbins)

These are lower bounds from clean upstream images; business images (frequent small updates, vendored duplicates) should dedup higher.

Design

Two address spaces, CDC entirely below the EROFS layer:

• Logical space (unchanged): inode chunk indexes, cache data file, read/fetch/probe APIs, zero-chunk elision. Size = new logical_block_count header field × 4 KiB.
• Unique space (what groups describe): each fixed chunk's bytes are split at FastCDC v2020 cut points (min 4K / avg 16K / max 64K), deduped by BLAKE3, and only unseen pieces enter the group stream — group compression composes with dedup, simply compressing the post-dedup bytes.

The chunk table stores 56-byte records (digest, logical_byte_offset, unique_byte_offset, size), sorted by logical offset; uncovered logical ranges read as zeros. Runtime: binary-search records overlapping a read → map each cold record's unique range to group(s) via the existing >> group_block_bits division → decode (memoized, single-flight, cross-process locked) → copy to the logical offset in the cache file. Readiness is per record in a .chunk.map sidecar.

Changes

• nydus-format: BlobMetadataCdcChunk record, CHUNK_CDC incompat flag, logical_block_count header field (bytes 56..64, zero for non-CDC → non-CDC output stays byte-identical), validation, mmap access, CRC/serialization branches.
• nydus (build): BlobWriter::with_cdc() — FastCDC split + blake3 dedup map + byte-granular unique-stream append (tail group block-padded at finish); --cdc CLI flag; cdc_dedup stats in build output; fastcdc = "3" dep.
• nydus-storage (runtime): CDC branches in LocalBlobCache — logical-sized cache file, per-record readiness map, ensure_byte_range fills records in unique-offset order with a group-decode memo, prefetch_all decodes each group ~once, ready_ranges counts logical gaps as ready zeros.
• nydus optimize: rejects CDC source blobs (Unsupported) — its group re-slicing assumes logical == group space.
• Tests: format round-trip/validation (owned + mmap + bias), builder dedup on duplicate + offset-shifted content, storage unit tests (dedup records, holes, ready ranges, prefetch backend-read counting), e2e build+read-back through NydusCore.
• Docs: --cdc option and CDC blob meta layout in docs/nydus.md.

$ nydus build --type dir-nydus --cdc --blob-dir ./blobs ./rootfs
...
    cdc_dedup: logical 412340224 bytes, unique 289112064 bytes, saved 123228160 bytes (29.9%)

Future work

• Cross-image build-time dedup via chunk-dict (records referencing other blobs through the device table)
• Unique-space cache file option (keep dedup savings on disk at runtime)
• CDC support in nydus optimize

[imeoer]

dup with #2041