Repository navigation
(#294) [v0.10] fix(s3): keep child objects visible beneath folder markers - #296
Conversation
| out.push((fi.name.clone(), fi)); | ||
| // An object directory can also contain child objects. | ||
| true | ||
| } |
There was a problem hiding this comment.
🟠 Medium · Recursive listing now walks every object's internal files and data dirs
descend = true for every object means each listed object's directory gets another list_dir. That listing holds the object's own storage files: xl.meta, xl.meta.bkp and the data dir(s), which contain part.N.
Each of those entries then goes through read_version:
folder/xl.metaandfolder/xl.meta.bkptry to open…/xl.meta/xl.meta, which fails with ENOTDIR and hitsErr(_) => continue.- A data dir returns
FileNotFound. Because it passes thesymlink_metadatais_dir check, the walk recurses into it and runslist_diragain, plus a failingread_versionfor eachpart.N.
The results are still correct. The cost is that a full-bucket listing now adds, per object, a blocking std::fs::read_dir and a failed metadata open. Objects that aren't inline add another stat, readdir and failed open for each part. This is on every disk in the set, and the listing path used to do none of this work.
Fix: when descending from an object directory, skip reserved entries before calling read_version, i.e. META_FILENAME, META_BACKUP_FILENAME and the record's data dir(s). One option is to pass the set of names to skip into the recursive call. Another is to only recurse into children whose own xl.meta exists or that are plain key-prefix directories.
Fixes #294.
What changed
folder/child.txtbeforefolder/, then checks prefix and root listings, pagination, and reads.Validation
cargo test --offline --locked -p openlake_storage list_(6 passed)cargo test --offline --locked -p openlake_io local_fs::tests::(13 passed)cargo test --offline --locked -p openlake_server --bin openlaked(68 passed)cargo clippy --offline --locked --workspace --all-targets -- -D warningscargo fmt --all -- --checkgit diff --check