Skip to content

(#294) [v0.10] fix(s3): keep child objects visible beneath folder markers - #296

Merged
ArnavBalyan merged 2 commits into
openlake-project:mainfrom
Keda710:fix/folder-marker-listing-294
Sep 29, 2026
Merged

ArnavBalyan merged 2 commits into
openlake-project:mainfrom
Keda710:fix/folder-marker-listing-294

Conversation

@Keda710

@Keda710 Keda710 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Fixes #294.

What changed

  • Continue walking child directories when a folder marker has its own object metadata.
  • Skip recursion into non-directory entries when no object metadata is present.
  • Add a regression test that writes folder/child.txt before folder/, then checks prefix and root listings, pagination, and reads.

Validation

  • cargo test --offline --locked -p openlake_storage list_ (6 passed)
  • cargo test --offline --locked -p openlake_io local_fs::tests:: (13 passed)
  • cargo test --offline --locked -p openlake_server --bin openlaked (68 passed)
  • cargo clippy --offline --locked --workspace --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 1 medium-severity finding

Reviewed commit c3eed84.

Comment thread crates/openlake_io/src/local_fs.rs Outdated
out.push((fi.name.clone(), fi));
// An object directory can also contain child objects.
true
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Medium · Recursive listing now walks every object's internal files and data dirs

descend = true for every object means each listed object's directory gets another list_dir. That listing holds the object's own storage files: xl.meta, xl.meta.bkp and the data dir(s), which contain part.N.

Each of those entries then goes through read_version:

  • folder/xl.meta and folder/xl.meta.bkp try to open …/xl.meta/xl.meta, which fails with ENOTDIR and hits Err(_) => continue.
  • A data dir returns FileNotFound. Because it passes the symlink_metadata is_dir check, the walk recurses into it and runs list_dir again, plus a failing read_version for each part.N.

The results are still correct. The cost is that a full-bucket listing now adds, per object, a blocking std::fs::read_dir and a failed metadata open. Objects that aren't inline add another stat, readdir and failed open for each part. This is on every disk in the set, and the listing path used to do none of this work.

Fix: when descending from an object directory, skip reserved entries before calling read_version, i.e. META_FILENAME, META_BACKUP_FILENAME and the record's data dir(s). One option is to pass the set of names to skip into the recursive call. Another is to only recurse into children whose own xl.meta exists or that are plain key-prefix directories.

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues found

Reviewed commit bc69887.

@ArnavBalyan ArnavBalyan changed the title fix(s3): keep child objects visible beneath folder markers (#294) [v0.10] fix(s3): keep child objects visible beneath folder markers Sep 29, 2026
@ArnavBalyan
ArnavBalyan merged commit b416e92 into openlake-project:main Sep 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Folder marker objects hide child objects from listings

2 participants