From 77eabffe357b7c85be404694bad3c49e9f2967a1 Mon Sep 17 00:00:00 2001 From: imeoer Date: Mon, 23 Mar 2026 10:01:55 +0000 Subject: [PATCH] docs: add nydus v3 proposal Signed-off-by: imeoer --- docs/nydus-v3-proposal.md | 278 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 docs/nydus-v3-proposal.md diff --git a/docs/nydus-v3-proposal.md b/docs/nydus-v3-proposal.md new file mode 100644 index 00000000000..d022048247c --- /dev/null +++ b/docs/nydus-v3-proposal.md @@ -0,0 +1,278 @@ +# Nydus V3 Proposal + +## Overall Improvements + +- Consolidate and simplify command-line arguments, APIs, and configuration formats. +- Remove features that are no longer needed or have been effectively unmaintained. +- Drop RAFS v5 support on the build side and standardize image creation on the EROFS-based format, while preserving RAFS v5 compatibility at runtime. + +## Build Side + +### Functional Changes + +- Embed CRC32 into chunk metadata by default. +- Support build-time deduplication based on chunk dictionaries. +- Make `nydusify` depend only on the containerd converter, removing the acceleration service dependency. +- [Performance] Simplify the build pipeline to improve build speed and reduce memory consumption. + +## Runtime Side + +### Functional Changes + +- Simplify the API surface and adopt a more mature HTTP server framework to improve concurrency. +- Switch to YAML as the configuration format and simplify the configuration model. +- Stop persisting runtime configuration to disk to avoid leaking sensitive information, while supporting hot configuration updates. +- Support mounting individual image layers independently. +- Use CRC32 as the default data integrity check. +- Expose HTTP `/metrics` in standard Prometheus format. +- Provide native live upgrade and failover support for FUSE deployments. +- Remain compatible with direct kernel EROFS mounts. +- Support a higher-performance `pmem + uffd + nydusd` solution for Kata scenarios. +- [Performance] Replace chunk-level prefetching with blob-stream prefetching. +- [Performance] Support runtime ondemand-pattern collection to optimize chunk layout in images. +- [Performance] Make the full I/O stack asynchronous across `fuse-backend-rs` and `nydusd` storage. +- [Performance] Optimize storage-layer requests and improve deep integration with the Dragonfly SDK. + +### Quality + +- Achieve at least 80% test coverage for core functionality. +- Introduce a stricter filesystem test suite. +- Reduce CI runtime for integration tests. + +## Nydus Build Tool (nydus-image) + +### nydus-image create + +``` shell +nydus-image create -h +Create nydus filesystems from layer directories or tar files + +Usage: nydus-image create [OPTIONS] + +Arguments: + source from which to build the nydus filesystem + path to generated nydus bootstrap blob + +Options: + --log-level + Log level: [default: info] [possible values: trace, debug, info, warn, error] + --type + Conversion type: [default: dir-nydus] [possible values: dir-nydus, tar-nydus] + --blob + File path to save the generated nydus data blob + --chunk-size + Set the size of data chunks, must be power of two and between 0x1000-0x1000000: + --compressor + Algorithm to compress data chunks: [default: zstd] [possible values: none, lz4_block, zstd] + --digester + Algorithm to digest data chunks: [default: blake3] [possible values: blake3, sha256] + --output-json + File path to save operation result in JSON format +``` + +### nydus-image merge + +``` shell +nydus-image merge -h +Merge multiple nydus bootstraps into a overlaid bootstrap + +Usage: nydus-image merge [OPTIONS] ... + +Arguments: + ... bootstrap paths (allow one or more) + ... overlaid bootstrap path + +Options: + --log-level + Log level: [default: info] [possible values: trace, debug, info, warn, error] +``` + +### nydus-image check + +``` shell +nydus-image check -h +Check nydus layout from bootstrap file. + +Usage: nydus-image check [OPTIONS] + +Arguments: + ... bootstrap path + +Options: + --log-level + Log level: [default: info] [possible values: trace, debug, info, warn, error] +``` + +### nydus-image mount + +``` shell +nydus-image mount -h +Mount nydus layout from bootstrap file. + +Usage: nydus-image mount [OPTIONS] + +Arguments: + ... bootstrap path + ... mountpoint directory path + +Options: + --log-level + Log level: [default: info] [possible values: trace, debug, info, warn, error] +``` + +## Nydus Runtime Tool (nydusd) + +``` shell +nydusd -h +Nydus daemon to provide nydus image mount + +Usage: nydusd [OPTIONS] + +Commands: + fuse [default] Run as a FUSE daemon to mount nydus image + prefetch Load a nydus bootstrap and prefetch all blobs to cache + help Print this message or the help of the given subcommand(s) + +Options: + --log-level + Log level: [default: info] [possible values: trace, debug, info, warn, error] + + # For fuse, prefetch subcommand + --apisock + Path to the nydus daemon administration API socket + --config + Path to the nydus daemon configuration file (.yaml format) + --bootstrap + Path to the nydus filesystem metadata file + + # For fuse subcommand + --mountpoint + Mountpoint for the FUSE filesystem, target for `mount.fuse` + --failover-policy + FUSE server failover policy [default: resend] [possible values: none, resend, flush] + --fuse-threads + Number of worker threads to serve FUSE I/O requests [default: 4] + --supervisor + Path to the nydus daemon supervisor API socket + --upgrade + Start nydus daemon in upgrade mode +``` + +### Configuration + +``` yaml +# Nydusd configuration + +backend: + # Type: registry | localfs | oss + type: registry + config: + # Drop the read request once http request timeout, in seconds + timeout: 5 + # Drop the read request once http connection timeout, in seconds + connect_timeout: 5 + # Retry count when read request failed + retry_limit: 0 + proxy: + url: + +cache: + # Type: blobcache | dummy + type: blobcache + config: + # Directory of cache files, only for local + work_dir: /cache + +prefetch: + # Enable blob prefetch + enable: false + # Prefetch thread count + threads_count: 10 + # Limit prefetch bandwidth to 1MiB/s + bandwidth_rate: 1048576 +``` + +### API + +``` shell +GET /daemon + Returns general information about the nydus daemon + +GET /metrics + Returns daemon metrics: files, pattern, backend, cache, prefetch + +POST /mount + Mount an image instance or an individual layer + +DELETE /mount + Unmount a mounted image instance or layer + +POST /exit + Let nydus daemon exit + +PUT /fuse/sendfd +PUT /fuse/takeover + For FUSE daemon live upgrade / failover +``` + +## Nydusify + +```shell +nydusify -h +NAME: + Nydusify - Nydus utility tool to convert, check and mount nydus image + +USAGE: + Nydusify [global options] command [command options] + +COMMANDS: + convert Generate a Nydus image from an OCI image + check Verify nydus image format and content + mount Mount the nydus image as a filesystem + copy Copy an image from source to target + +GLOBAL OPTIONS: + --log-level value, -l value Set log level (trace, debug, info, warn, error) (default: "info") [$LOG_LEVEL] +``` + +## Kata Support + +### Current Approaches + +Nydus images can be used with Kata through several methods: + +| Approach | Description | Limitations | +|----------|-------------|------------| +| FUSE + Virtiofs | Mount meta and blob files as FUSE, expose through Virtiofs to guest | Frequent kernel/user context switches, high file operation overhead | +| Virtiofs + Blobs | Expose blob directory via Virtiofs, guest mounts meta + blobs as EROFS | EROFS DAX integration causes compatibility issues | +| Virtio-blk + Blobs | Build meta + blobs into virtio-blk device, guest mounts as EROFS | No DAX support, causes guest/host double page cache waste | +| Virtio-pmem + Local Blobs | Map local meta + blob files to mmap region as virtio-pmem device | Requires pre-downloaded blobs, no on-demand fetching | + +### Proposed Solution: pmem + uffd + nydusd + +This proposal implements a hybrid approach combining pmem, userfaultfd, and nydusd to achieve: + +- Improved runtime performance +- Full feature compatibility +- Shared page cache between host and guest +- Complete DAX support +- On-demand blob fetching for faster startup + +``` ++---------------------------------------+ +| +-----------VMM event loop----------+ | +| |recv [blobfd, off, len, fdoff] | | +| | * mmap fd range into pmem backend| | +| | * UFFDIO_WAKE vcpu on the range | | +| +-----------------A-----------------+ | +| |sock(json+fd in FC)| +| V | +| +------Runtime (uffd and proxy?)----+ | +-------------Nydusd-----------+ +| |handle userfaultfd event on fault | | |recv block request | +| | * send block request to nydusd |<-sock->|pull chunk from remote/local | +| |recv [blobfd, off, len, fdoff] | |(nbd) |send [blobfd, off, len, fdoff]| +| | * transfer to VMM | | +------------------------------+ +| | * trap UFFD_EVENT_UNMAP | | +| +-----------------------------------+ | ++---------------------------------------+ +```