Skip to content

feat(files): begin Filesystem Collections ingestion - #134

Merged
devnull03 merged 38 commits into
mainfrom
feat/filesystem-collections-plan
Sep 22, 2026
Merged

devnull03 merged 38 commits into
mainfrom
feat/filesystem-collections-plan

Conversation

@devnull03

@devnull03 devnull03 commented Sep 13, 2026 •

Copy link
Copy Markdown
Owner

Overview

This draft begins the Filesystem Collections release: qrate will build archival components from files and folders, accept dropped material in open projects, and expose nested descriptions as collapsible rows.

The first implementation slice adds a shared filesystem inventory used by project creation and open-project file resolution. Recursive discovery now preserves relative paths, so files such as photographs/001.jpg and documents/001.jpg remain distinct instead of collapsing to the same basename. The photo index understands those relative paths while retaining basename matching for existing projects.

The design plan in docs/dev/filesystem-collections-plan.md covers the remaining schema v4 migration, standards-aware field roles and labels, import preview, drag-and-drop surfaces, hierarchy editing, interchange, and release sequencing. Parent descriptions are normal archival components and will participate in search, diagnostics, galleries, exports, plugins, and Google Sheets.

Implemented in this draft

  • Add a UI-independent file-ingest crate with deterministic recursive discovery.
  • Preserve directories, parent relationships, and relative paths in the inventory.
  • Skip filesystem cruft and directory symlinks, with bounded warnings for partial scans.
  • Use the shared scanner in project-wizard folder discovery.
  • Use the shared scanner in the table photo index and resolve duplicate basenames by relative path.
  • Add focused coverage for nesting, recursion controls, ordering, ignored files, and duplicate filenames.

Verification

  • cargo fmt --all --check — passed
  • cargo test -p file-ingest -p project-wizard -p table --target-dir target/rust-analyzer — passed (101 tests)
  • cargo clippy -p file-ingest -p project-wizard -p table --all-targets --target-dir target/rust-analyzer -- -D warnings -A dead_code — passed
  • git diff --check — passed

The PR remains a draft while the higher-level schema and UI slices are implemented.


Import duplicates (ASNT-77)

Closes #87 (Notion ID: 3b821d32-b13b-818d-a1b6-c542b16fe2d7)

Re-importing a batch folder no longer copies what the project already holds. file_ingest::duplicates matches an import against existing components by stored source path, then by a single filename-key match; anything several rows could claim stays ambiguous and is never merged automatically.

  • Skip (default), Update, or Add as new decide only the matches. Update re-links the row and leaves its metadata and its place in the arrangement untouched — qrate links files in place, so nothing is overwritten.
  • A skipped folder still adopts new files below it, so a second batch lands inside the series the first one made.
  • The wizard asks only when a folder holds a file that several rows name; the open table asks only when a drop repeats material. The answer is saved per project.
  • The wizard and the table now match files with the same key rule, so a multi-part item counted as matched in the Files step is also linked at creation.

Plan and the two deliberate deviations: docs/dev/import-duplicates-plan.md.

Audit fixes in this branch

  • Hierarchy is written in the same transaction as its rows. It was saved ahead of them, so structure naming an unsaved row failed its foreign key and the next save restored the stale copy — imported hierarchy was lost on reopen.
  • Inserted and duplicated rows appear beside their neighbour instead of at the end of the table.
  • Dropped roots queue after existing siblings rather than landing among them.
  • Deleting a parent promotes its children into its place, not to the top level.
  • Export fills structure columns only where the cell is blank, so typed values survive.
  • Loading, creating, importing and saving log their row and component counts; skipped paths are named.

Verification (latest)

  • ./scripts/ci.sh — passed (fmt, clippy -D warnings, full workspace tests)

@devnull03 devnull03 changed the title docs: plan Filesystem Collections release feat(files): begin Filesystem Collections ingestion Sep 14, 2026
@devnull03
devnull03 force-pushed the feat/filesystem-collections-plan branch from c2b140a to 0591ff2 Compare September 14, 2026 21:19
The hierarchy was written ahead of the dataset save, so structure naming an
unsaved row failed its foreign key and the next save restored the stale copy.
save_dataset now writes the live structure in the same transaction.

Also keeps inserted and duplicated rows beside their neighbour, queues dropped
roots after existing siblings, promotes children into a deleted parent's place,
stores dropped source paths relative to the files root, and indexes hierarchy
traversal instead of rescanning every row.
Structure projection only fills blank cells, and the flat JSON-LD writer that
the hierarchy-aware one replaced is removed.
Opening, creating, and dropping into a project now record their row and
component counts and the active description profile, and the paths that get
skipped are named rather than only counted.

A failed folder scan, a row that cannot be placed beside its neighbour, and a
deleted row missing its structure each say so instead of passing silently.
Drops the unused subtree delete mode the row-deletion path replaced.
A pure matcher for ASNT-77: the same stored source path, then exactly one
filename-key match, and anything matching several rows stays ambiguous rather
than merging. Skip, update, or add-as-new decide only the matches.

A path reached twice in one import collapses on its own, and a skipped folder
still adopts the files below it that are new, so a second batch lands inside
the series the first batch made.
Project creation resolves its folder against the spreadsheet rows instead of
claiming each file for the last row that named it, and the Files step offers
the choice only when the folder actually holds such a file.

Files are matched with the same key rule the Files step counts with, so a
multi-part item counted as matched is also linked at creation. The answer is
saved per project for the next import.
A drop resolves against the open project first. The prompt says how much of it
is already catalogued and offers skip, update, or add as new; a drop with
nothing in common still asks only whether to import.

Update re-links the row and leaves its metadata and its place in the
arrangement alone, and the whole import stays one undo step.
Hand GIFs to gpui at every size. The thumbnail ladder decoded frame 0 only, so a GIF only moved in the fullscreen viewer.
Extract the wizard's config dialog into ColumnConfigLoader, a self-rendering view with the file and Sheet tabs and the column mapping. The wizard dialog holds it as a child, which removes the close-and-reopen refresh. Data > Load Column Config... applies it to the open project, writing only what the config states.
- Group selected rows under a new parent, undone as one step.
- Draw nesting in the row gutter the way the Problems panel does: tint, guide lines, and a chevron with a child count.
- Offer the child-aware delete commands only on rows with children.
- Double-click a column header to rename it.
- Push added, removed, and renamed columns into the open project so Settings sees them.
- Write a path the preview can resolve when a file is picked from outside the files folder.
@devnull03
devnull03 force-pushed the feat/filesystem-collections-plan branch from 1316c1e to 1b9e61d Compare September 21, 2026 07:45
@devnull03
devnull03 marked this pull request as ready for review September 21, 2026 16:57
The target's position was counted with the moving row still among its
siblings, so a row moved below a later sibling landed one slot too far.
@devnull03
devnull03 merged commit d48f534 into main Sep 22, 2026
3 checks passed
@devnull03
devnull03 deleted the feat/filesystem-collections-plan branch September 22, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ASNT-77: Project wizard: duplicate-handling policy on import

1 participant