fix(storage): make GCS storage layer importName-subdirectory aware

[jm-rivera]

What

The DCP platform stores each import under a shared input prefix, ingestion/input/<importName>/. The storage layer assumed a flat, single-import layout. Three defects followed.

1. Nested config.json was skipped on upload. _SKIP_IN_SUBDIR dropped any .json that was not in the upload root, so per-import configs never reached GCS and the load job had nothing to read.
2. --sync deleted sibling imports. sync_directory_to_gcs listed every blob under the shared prefix and deleted anything without a local counterpart, wiping every other import's data. Permanent data loss.
3. Registration checks misreported. get_unregistered_csv_files and get_missing_csv_files compared <importName>/data.csv blob paths against flat config keys, so every CSV read as both unregistered and missing.

How

Sync deletion now scopes to the import subdirectories present locally. Two rules decide what is stale. A subtree rule prunes any depth under a present import subdir. A direct-child rule prunes legacy root-level blobs. A blob under an import you did not bring locally is never a deletion candidate, which is the data-loss fix.

_SKIP_IN_SUBDIR is gone, so nested config.json uploads. The two registration functions take an optional import_name that scopes the check to one import's subdirectory; omit it and they behave as before. A fresh import with nothing uploaded yet now returns the right answer instead of raising.

Supporting changes are a shared _remote_path helper so upload and sync compute paths in one place, a KGSettings validator that strips stray slashes from the folder paths, and kwargs-only signatures on the public storage functions.

pipeline.py is unchanged. Sync became subset-safe by discovering import names from the directory tree, so the caller needs no new argument.

Testing

uv run pytest passes 96 tests. ruff check and ruff format --check are clean. Eight new or updated tests cover sibling-import preservation, legacy pruning, per-import registration, the fresh-import case, slash handling, and kwargs-only enforcement.

Closes #108

Co-authored-by: Claude noreply@anthropic.com

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.38650% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.64%. Comparing base (48aecf8) to head (e12bb7a).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...bblocks/datacommons_tools/gcp_utilities/storage.py	97.87%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #114      +/-   ##
==========================================
+ Coverage   91.06%   93.64%   +2.58%     
==========================================
  Files          37       38       +1     
  Lines        1801     2157     +356     
==========================================
+ Hits         1640     2020     +380     
+ Misses        161      137      -24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Updates the GCS storage utilities to be aware of the ingestion/input/<importName>/ subdirectory layout, fixing prior incorrect uploads, dangerous --sync deletion behavior, and incorrect CSV registration checks under multi-import prefixes.

Changes:

• Remove subdir JSON skipping and standardize remote path construction for uploads/sync.
• Make sync deletion subset-safe by only pruning within locally-present import scopes (plus legacy root-level pruning).
• Add optional per-import scoping to CSV registration checks, strip stray slashes in KGSettings, and enforce kwargs-only gcs_folder_name in public storage APIs.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
src/bblocks/datacommons_tools/gcp_utilities/storage.py	Adds import-subdir-aware sync scoping, shared remote-path helpers, kwargs-only APIs, and per-import registration checks.
src/bblocks/datacommons_tools/gcp_utilities/settings.py	Adds validators to normalize GCS folder path settings by stripping leading/trailing slashes.
tests/test_storage.py	Expands tests for nested JSON uploads, subset-safe sync behavior, per-import registration, fresh-import behavior, slash normalization, and kwargs-only enforcement.
tests/test_explicit_path_characterization.py	Updates callers for the new kwargs-only storage function signatures.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.