v3: CDC (content-defined chunking) chunk dedup for blob data

docs/nydus.md

@@ -190,6 +190,18 @@ Current implementation notes:
 	the group is stored plain and its blob_meta group record has
 	`compressed_size == uncompressed_block_count * 4096`.
 - `--compressor none` writes every group plain.
+- Content-defined chunking (CDC) deduplication is always on. EROFS inode
+	chunk indexes and `--chunk-size` still address the dense logical
+	uncompressed space, but beneath them file data is split at FastCDC v2020
+	cut points (min 4 KiB / avg 16 KiB / max 64 KiB) and deduplicated by
+	BLAKE3 digest: only never-seen-before pieces enter the group data stream,
+	so shared content is stored once even when files embed it at different
+	offsets. The blob meta chunk table holds 56-byte CDC records
+	`(digest, logical_byte_offset, unique_byte_offset, size)`, the header
+	carries the `CHUNK_CDC` incompat flag plus the logical block count, and
+	the build summary prints a `cdc_dedup` line with logical/unique byte
+	counts and the dedup percentage. CDC blobs are read through the same
+	group cache and are fully supported by `nydus optimize`.
 - `--exclude <path>` omits paths inside the source tree from the blob and the
 	resulting filesystem tree entirely. It accepts absolute or
 	current-working-directory-relative paths and may be repeated.
@@ -1054,8 +1066,9 @@ At the same time:
 
 Whenever build emits a full blob, it writes one blob meta region before the
 footer. Blob meta is the canonical catalog for the external data blob. A blob
-meta chunk is a content-addressed record (BLAKE3 digest + absolute block range)
-used for inspection and future deduplication; chunks are independent of groups.
+meta chunk is a content-addressed CDC record (BLAKE3 digest + logical byte
+offset + unique byte offset + size) that maps the dense logical address space
+onto the deduplicated unique data stream; chunks are independent of groups.
 A blob meta group is the compression unit and cache population unit. EROFS inode
 chunk indexes point into the logical uncompressed external-device address space;
 blob meta maps a block offset to its group by a single division and the cache
@@ -1081,13 +1094,13 @@ embedded blob meta region
 | group_block_bits (u8 + pad)   |
 | reserved tail (compat area)   |
 +-------------------------------+
-| chunk records                 |
-| 48 bytes each                 |
+| CDC chunk records             |
+| 56 bytes each                 |
 |                               |
 | digest (BLAKE3)               |
-| uncompressed_block_offset     |
-| uncompressed_block_count      |
-| reserved                      |
+| logical_byte_offset           |
+| unique_byte_offset            |
+| size + reserved               |
 +-------------------------------+
 | group records                 |
 | 40 bytes each                 |
@@ -1129,7 +1142,7 @@ Header details:
 	verifies this crc32c before mmaping a cached blob meta file for chunk lookup.
 - `chunks_offset` is fixed at the header size. `groups_offset` follows the dense
 	chunk table.
-- `chunk_count` is the number of chunk records.
+- `chunk_count` is the number of CDC chunk records.
 - `group_count` is the number of compressed group records.
 - `chunk_block_bits` is log2 of the EROFS chunk size in 4 KiB blocks:
 	`chunk_size = 4096 << chunk_block_bits`, so the default 1 MiB chunk stores
@@ -1155,22 +1168,34 @@ Header details:
 
 Chunk details:
 
-- Chunks are decoupled from groups: a chunk may straddle a group boundary, and a
-	group may contain parts of several chunks. The chunk table is a digest index,
-	not a per-group map.
-- `digest` is the BLAKE3 hash of the chunk's decoded, block-aligned bytes — the
-	deduplication key.
-- `uncompressed_block_offset` is the chunk's absolute 4 KiB block offset in the
-	dense decoded address space (chunks are stored back-to-back).
-- `uncompressed_block_count` is the chunk span in 4 KiB blocks. Only the chunk's
-	final block carries zero padding; full chunks are already block-aligned, so the
-	dense layout packs real blocks instead of large zero runs.
+- CDC records are decoupled from groups: a group may contain many records'
+	bytes, and the chunk table is a byte-granular mapping from the logical space
+	to the unique stream, not a per-group map. The builder does guarantee one
+	group invariant: no record's unique bytes ever straddle a group boundary —
+	when a fresh piece would cross one, the unique stream is zero padded up to
+	the boundary first, so every group is self-contained (a decoded group alone
+	satisfies every record it holds). Readers must still tolerate straddling
+	records from foreign writers by decoding every group the record's unique
+	range touches.
+- `digest` is the BLAKE3 hash of the piece's bytes — the deduplication key.
+- `logical_byte_offset` is the piece's byte position in the dense logical
+	uncompressed address space that EROFS chunk indexes point into.
+- `unique_byte_offset` is the byte position of the piece's (single) copy in the
+	deduplicated unique data stream that the groups compress. Many records may
+	share one unique range — that sharing is the deduplication.
+- `size` is the piece length in bytes (FastCDC pieces, 4 KiB min / 16 KiB avg /
+	64 KiB max). Records are sorted by `logical_byte_offset` and never overlap;
+	logical gaps read back as zeros.
 
 Group details:
 
 - Groups are formed by packing whole decoded blocks up to `--compress-size`
 	regardless of chunk boundaries, then compressing the batch as one unit. So
-	every group but the last is exactly `1 << group_block_bits` blocks.
+	every group but the last is exactly `1 << group_block_bits` blocks. The
+	builder zero-pads the unique stream to the group boundary when a fresh CDC
+	piece would straddle it (see Chunk details above), so the padding bytes are
+	stored inside the group — they compress to almost nothing and are never
+	referenced by any record.
 - `uncompressed_block_offset` is the decoded cache 4 KiB block offset for the
 	group. Groups are dense and contiguous in the decoded address space.
 - `compressed_byte_offset` is the encoded payload's byte offset within the data
@@ -1199,6 +1224,77 @@ does not bias `uncompressed_block_offset`. Only the data region as a whole is
 padded to a 4 KiB boundary (so the embedded bootstrap that follows starts on a
 block); groups themselves are not individually padded.
 
+### CDC (content-defined chunking) blob meta
+
+Every data blob is built with content-defined chunking: the header sets the
+`CHUNK_CDC` incompat flag (`1 << 2`) and the chunk table holds the 56-byte CDC
+records described above (redirect/ondemand blobs are groups-only, with an
+empty chunk table and the flag clear):
+
+```text
+CDC chunk record (56 bytes)
+
+u8  digest[32]           BLAKE3 of the piece's bytes (the dedup key)
+u64 logical_byte_offset  byte position in the logical uncompressed space
+u64 unique_byte_offset   byte position in the deduplicated unique stream
+u32 size                 piece length in bytes (4 KiB..64 KiB FastCDC pieces)
+u32 reserved
+```
+
+Two address spaces are involved:
+
+- The **logical** space is unchanged: EROFS inode chunk indexes still point at
+	fixed power-of-two chunks in a dense uncompressed address space, and the
+	cache data file still mirrors it, so the kernel-visible format and the
+	read/`fetch`/`probe` APIs are untouched. Its size is
+	`logical_block_count * 4096`, from a u64 header field at offset 56 (that
+	field must be zero for groups-only blobs without the flag, such as
+	ondemand blobs).
+- The **unique** space is what the group records describe: each fixed chunk's
+	real bytes are split at FastCDC v2020 cut points (min 4 KiB / avg 16 KiB /
+	max 64 KiB) and only never-seen-before pieces (by BLAKE3 digest) are
+	appended, byte-granular, to the group stream, which is then grouped and
+	compressed exactly as before. Many CDC records may reference the same
+	unique bytes — that sharing is the deduplication. The builder zero-pads the
+	stream up to the next group boundary whenever a fresh piece would straddle
+	it, so every record's unique bytes live in exactly one group ("groups are
+	self-contained"); a deduped record inherits that invariant from the first
+	occurrence it points at, and the padding bytes are dead stream bytes no
+	record references.
+
+Records are sorted by `logical_byte_offset` and never overlap; logical ranges
+not covered by any record (tail-block padding, elided all-zero chunks) read
+back as zeros.
+
+Runtime design for CDC blobs:
+
+- **Cache layout.** The cache data file mirrors the logical space
+	(`logical_block_count * 4096` bytes); readiness is tracked per CDC record —
+	not per group — in a `.chunk.map` sidecar (same on-disk format as the
+	`.group.map` sidecar, one bit per chunk record). Non-CDC (groups-only)
+	blobs keep the per-group `.group.map`.
+- **On-demand reads.** A read binary-searches the records overlapping the
+	logical range, sorts the cold ones by unique offset, maps each record's
+	unique range to its group with the `>> group_block_bits` division, decodes
+	the group (memoized within the call, single-flight within the process,
+	cross-process claimed per record), and copies the record's bytes to its
+	logical offset in the cache file before marking the record ready. Records
+	from a foreign writer that straddle a group boundary are still handled by
+	decoding every group the unique range touches.
+- **Prefetch.** `prefetch_all` walks the records in unique-offset order so
+	each group is fetched and decoded roughly once, fanning every decoded
+	group's bytes out to all the records it contains.
+- **Redirect (ondemand-blob) fill.** `nydus optimize` operates at group
+	granularity on the unique stream, so it supports CDC blobs directly: traced
+	unique-stream groups are re-encoded into the ondemand blob, and the phase-0
+	redirect fill CRC-checks each decoded group and fans its bytes out to the
+	CDC records it fully contains, at their logical offsets in the source
+	cache. Because the builder guarantees records never straddle groups, a
+	filled group leaves no partially-warm records behind — a traced workload
+	replayed after prefetch is served entirely from cache with zero on-demand
+	backend reads. A fill that finds all its records already ready counts as a
+	cache hit instead of a redirect fill.
+
 ### Blocks, chunks and groups
 
 The three units live in two address spaces: blocks, chunks and groups are
@@ -1230,10 +1326,11 @@ blob cache at runtime: the core read paths satisfy them with zeros
 directly, and native EROFS mounts decode the null address in-kernel the same
 way.
 
-The per-file chunks are then packed densely, back-to-back, into the decoded
-external-device address space that EROFS chunk indexes point into; each
-chunk's BLAKE3 digest and absolute block range are recorded in the blob meta
-chunk table:
+The per-file chunks are then packed densely, back-to-back, into the logical
+external-device address space that EROFS chunk indexes point into; beneath
+them, each chunk's bytes are split at FastCDC cut points and recorded as CDC
+records in the blob meta chunk table (duplicate pieces point at the same
+unique bytes):
 
 ```text
 blkaddr    0        256    448      704      960
@@ -1279,8 +1376,8 @@ encoded data region of the full blob:
 
 Hash and validation summary:
 
-- **BLAKE3 per chunk** (blob meta chunk table) — the deduplication key over
-	the chunk's decoded, block-aligned bytes.
+- **BLAKE3 per CDC piece** (blob meta chunk table) — the deduplication key
+	over the piece's bytes.
 - **CRC32C per group** (blob meta group record) — validated after every fetch
 	and decode, on both the on-demand and prefetch paths.
 - **SHA256 over the data region** — written into the bootstrap device slot as
@@ -1368,12 +1465,15 @@ The build pipeline now follows this sequence:
 	space. Chunks are packed densely: each chunk advances by its real
 	block-aligned size, so only a chunk's final block carries zero padding (no
 	full-chunk zero runs).
-3. Record one blob_meta chunk entry per chunk (BLAKE3 digest + absolute block
-	range) and feed the decoded data stream into a block-oriented group builder
-	that flushes a compression group whenever it fills to `--compress-size`,
-	regardless of chunk boundaries. A chunk may therefore span two groups.
-4. Compute BLAKE3 digest over each uncompressed chunk and CRC32C over each
-	uncompressed group.
+3. Split each chunk's bytes at FastCDC cut points, deduplicate the pieces by
+	BLAKE3 digest, and record one blob_meta CDC chunk entry per piece (digest +
+	logical byte offset + unique byte offset + size). Only never-seen-before
+	pieces enter the unique data stream, which feeds a block-oriented group
+	builder that flushes a compression group whenever it fills to
+	`--compress-size`. When a fresh piece would straddle a group boundary the
+	unique stream is zero padded up to the boundary first, so no record ever
+	spans two groups.
+4. Compute CRC32C over each uncompressed group of the unique stream.
 5. Compress each group according to the blob_meta header compressor and append
 	the encoded bytes directly to the data region. Encoded groups are packed
 	back-to-back with no inter-group padding. For zstd, groups that do not shrink
@@ -1455,18 +1555,27 @@ blob digest:
 - `<full_blob_digest>.blob.data` stores decoded uncompressed data.
 - `<full_blob_digest>.blob.meta` stores the verified blob meta copy cached from
 	the local backend.
-- `<full_blob_digest>.group.map` records which blob_meta groups have been decoded
-	(a shared readiness bitmap, see
+- `<full_blob_digest>.chunk.map` records, for a CDC data blob, which CDC chunk
+	records have been filled into the logical cache file (a shared readiness
+	bitmap, one bit per record, same on-disk format as the group map, see
+	[Cross-process cache sharing](#cross-process-cache-sharing-and-prefetch-dedup)).
+- `<full_blob_digest>.group.map` records, for a groups-only (non-CDC) blob,
+	which blob_meta groups have been decoded (a shared readiness bitmap, see
 	[Cross-process cache sharing](#cross-process-cache-sharing-and-prefetch-dedup)).
 - `<full_blob_digest>.prefetch.lock` is the cross-process prefetch lock file
 	(empty; only its `flock` state matters).
 - `<full_blob_digest>.flight.lock` is the cross-process fetch lock file for
 	single groups (empty; only its byte-range lock state matters, see
 	[Cross-process cache sharing](#cross-process-cache-sharing-and-prefetch-dedup)).
 
-The cache data file mirrors the decoded address space one-to-one, so a group's
-bytes land at `uncompressed_block_offset * 4096` and EROFS chunk `blkaddr`
-offsets index into it directly:
+The cache data file mirrors the decoded address space one-to-one. For a
+groups-only blob that space is the group stream itself, so a group's bytes
+land at `uncompressed_block_offset * 4096`; for a CDC data blob it is the
+logical space (`logical_block_count * 4096` bytes), filled record by record
+from decoded unique-stream groups, and EROFS chunk `blkaddr` offsets index
+into it directly. The figure below shows the groups-only shape; a CDC blob
+replaces `.group.map` with `.chunk.map` (one bit per CDC record) and fills
+`.blob.data` at each record's `logical_byte_offset`:
 
 ```text
 cache directory, artifacts named by SHA256(full blob) = <hex>

nydus-backend/src/local.rs

@@ -270,7 +270,7 @@ fn probe_full_blob_source(
 mod tests {
     use super::*;
     use crate::ReadKind;
-    use nydus_format::blob::{BlobMetadataChunk, BlobMetadataGroup};
+    use nydus_format::blob::BlobMetadataGroup;
     use nydus_format::utils::sha256_bytes;
     use tempfile::tempdir;
 
@@ -279,7 +279,6 @@ mod tests {
             blob_id,
             1,
             vec![BlobMetadataGroup::new(0, 1, 0, 4096, crc32c::crc32c(payload)).unwrap()],
-            vec![BlobMetadataChunk::new(*blake3::hash(payload).as_bytes(), 0, 1).unwrap()],
         )
         .unwrap()
     }
@@ -310,7 +309,8 @@ mod tests {
             )
             .unwrap();
 
-        assert_eq!(blob_metadata.header().chunk_count(), 1);
+        assert_eq!(blob_metadata.header().chunk_count(), 0);
+        assert_eq!(blob_metadata.groups().len(), 1);
         assert_eq!(data, payload);
     }
 
@@ -338,7 +338,8 @@ mod tests {
             )
             .unwrap();
 
-        assert_eq!(blob_metadata.header().chunk_count(), 1);
+        assert_eq!(blob_metadata.header().chunk_count(), 0);
+        assert_eq!(blob_metadata.groups().len(), 1);
         assert_eq!(data, payload);
         assert!(backend.blob_metadata(&data_blob_id).is_err());
     }