Enhance blob metadata validation and simplify API structure - #2042
Merged
Conversation
…mats Introduce a `blob::flag` module that centralises the EROFS-style incompatible/compatible flag split (low 16 bits incompat, high 16 bits compat) and a `validate_incompat_flags` helper that rejects records carrying unknown incompat bits with a descriptive error. This replaces ad-hoc per-format checks and ensures consistent forward-compatibility enforcement across blob meta and blob footer readers. Signed-off-by: Gaius <gaius.qi@gmail.com>
…tency Rename `compressed_byte_offset` to `compressed_offset`, "record" to "entry" throughout the blob metadata spec and implementation, add a `NYDUS_` prefix to public constants, drop the `_byte_` infix from method names, introduce typed algorithm enums for blob meta compressor and digester, restructure header serialization to use byte-array methods, and update docs and tests accordingly. Signed-off-by: Gaius <gaius.qi@gmail.com>
…nd block group size to 4 MiB Rename blob metadata constants to use the `DEFAULT_` prefix for clarity, rename `block_group_block_bits` fields to `block_group_block_count_bits`, expand the default block group size to 4 MiB, move validation logic into methods, and add the clap `string` feature. Signed-off-by: Gaius <gaius.qi@gmail.com>
Merge `validate_padding` and `validate_blob_metadata_crc32` free functions into a single `BlobMetadata::validate_bytes` method, and simplify chunk/block-group parsing to use `chunks_exact` instead of manual index arithmetic. Signed-off-by: Gaius <gaius.qi@gmail.com>
Replace builder-pattern blob metadata loading with direct `from_bytes` calls, rename and inline private helpers, add doc comments, and unify method naming across blob metadata, cache, and storage layers for improved clarity. Signed-off-by: Gaius <gaius.qi@gmail.com>
Remove the `blob_id` field from `BlobMetadata` and eliminate redundant `blob_id`/`data_digest` parameters from related APIs, simplifying the blob metadata construction and usage across build, check, optimize, backend, format, and storage subsystems. Signed-off-by: Gaius <gaius.qi@gmail.com>
gaius-qi
requested review from
ClementMaH,
CormickKneey,
EvanCley,
bergwolf,
fcgxz2003 and
hhhhsdxxxx
and removed request for
a team
August 25, 2026 08:58
…meatdata # Conflicts: # docs/nydus.md # nydus-format/src/blob/algorithm.rs # nydus-format/src/blob/metadata.rs # nydus-format/src/blob/mod.rs # nydus-storage/src/cache/local.rs # nydus-storage/src/cache/mod.rs # nydus-storage/src/cache/remote.rs # nydus/src/bin/nydus/build.rs # nydus/src/build/blob_chunk.rs # nydus/src/check/mod.rs # nydus/src/optimize/mod.rs # nydus/tests/testsuite/nydus_core.rs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lobMetadataDigester Clarify that the Display impl emits the lowercase algorithm name used in build and check summaries. Signed-off-by: Gaius <gaius.qi@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This pull request includes a number of improvements and clarifications to the Nydus documentation and codebase, focusing on terminology consistency, code modernization, and test simplification. The most significant changes are terminology updates in the documentation for greater clarity, refactoring of code to use updated APIs and naming, and improvements to test code for maintainability.
Documentation terminology and clarity updates:
docs/nydus.md, replacing "record" with "entry" for block groups and chunks, and updating field names for better consistency (e.g.,compressed_byte_offset→compressed_offset,block_group_block_bits→block_group_block_count_bits). Also clarified various explanations and updated example code and tables accordingly. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]Codebase modernization and improvements:
clapdependency inCargo.tomlto include thestringfeature, enabling additional command-line parsing capabilities.nydus-backend/src/local.rsto use the newNYDUS_BLOB_METADATA_SUFFIXand updatedBlobMetadata::from_bytesAPI, simplifying metadata loading and improving code consistency. [1] [2] [3]BlobMetadata::from_bytesAPI for loading metadata, improving error handling and consistency with local backend changes.is_redirect_blob()tois_redirect()innydus-core/src/blob.rsfor consistency with refactored naming.Test code simplification:
nydus-backend/src/local.rsto use updated constructors and removed unnecessary parameters, making tests easier to read and maintain. [1] [2] [3]Related Issues
Please link to the relevant issue. For example:
Fix #123orRelated #456.Change Details
Please describe your changes in detail:
Test Results
If you have any relevant screenshots or videos that can help illustrate your changes, please add them here.
Change Type
Please select the type of change your pull request relates to:
Self-Checklist
Before submitting a pull request, please ensure you have completed the following: