[devnull03]

Visual search and search inside linked PDFs. Design doc: Visual search, similar items, and linked-file text search.

What works

Search inside linked PDFs

• The find bar has an Include text inside linked files toggle, off by default.
• With it on, each linked PDF's text is read once in the background (preview::document_text) and cached beside the thumbnails.
• Matching follows the same case, whole-word and regex rules as cells. The view narrows to the rows with a hit.

Visual search

• New visual-search crate: CLIP ViT-B/32 on the CPU through candle (pure Rust), with a hand-written CLIP tokenizer.
• Weights: a pinned download (openai/clip-vit-base-patch32@b33cedf, 605 MB), checked against its SHA-256 before it's moved into place.
• Image toggle in the find bar: rows are ranked by how well their linked file's thumbnail matches the query. The view temporarily shows only the hits, best first.
• Breadth slider: keeps hits scoring within a proportion of the best one, 15% by default, adjustable from 2% to 40% while you watch. Replace is disabled during a visual search.
• Find similar items: in the row and card context menu. It ranks by that file's own vector.
• What gets embedded: thumbnails (preview::thumbnail_pixels), so it covers everything qrate can preview. PDFs use their first page, video its poster frame, audio its cover art.

Works in every view

• The find bar is drawn by ViewsPanel above Table or Gallery. A search survives switching between them, and Find / Ctrl+F / Ctrl+H work in both.
• The gallery scrolls to follow the selected hit.
• Fullscreen viewer: it follows the table selection, so stepping through results from the find bar changes the image. Left/right row arrows in the bottom-right corner (and ↑ / ↓) step through the view's rows, which during a search are the results. Rows with nothing to preview are skipped.

Storage

• Vectors live in the project file, in a __visual_index table created on first write. The model that built them is recorded in __settings, and a different model clears them.
• Paths are relative to the files folder, and a file counts as changed only when its length changes. A project copied with its files keeps its index.
• Each batch is written as it's embedded.

Docs

• New docs/visual-search.md: what it searches, installing the model, searching and the breadth slider, find similar items, indexing and where the index lives, searching inside linked PDFs, and the limits (one page or frame per file, no handwriting or printed text, English queries, never writes a cell).
• Updated: docs/index.md, docs/grid.md (the two search toggles, and the bar following the view), docs/files-and-photos.md (the viewer's row arrows), and the README (feature bullet, privacy paragraph, docs list).

Crash fix: fast scrolling through thumbnails

This fixes a bug already on main, not one this branch introduced. Scrolling quickly through a large gallery could abort the app with a panic in gpui's DirectX atlas (directx_atlas.rs:255, Option::unwrap() on None). Windows reports the abort as STATUS_STACK_BUFFER_OVERRUN.

• Cause:
  • preview::retain enforces the 192 MB thumbnail memory cap. It called cx.drop_image straight from the image loader, and gpui runs that loader while painting.
  • Freeing an image mid-paint could release the GPU texture behind a sprite that the same frame had already painted. Presenting that frame then looked up a texture that no longer existed.
  • The cap was also first-in-first-out: an image already held was never moved back. A card still on screen could be the "oldest" one freed.
• Fix:
  • Every draw now moves the image to the back of the queue, so eviction takes whatever has been off screen longest.
  • retain never frees anything itself. It queues a release with window.on_next_frame, which runs before the next frame is laid out.
  • After freeing, release repaints every window. Dock panels are cached views that replay their last paint's sprites, and those could include a freed image.
• Test: the eviction test now checks three things. Nothing is freed inside the frame that drew it. The release keeps the most recently drawn images. An image drawn again survives.

Decisions worth reviewing

• Search narrowing in the delegate: set_search_rows overrides the view's rows. Searches always scan the rows the column filters allow (filtered_rows), not the rows the previous search left, so typing more can widen results again. An insert or delete drops the narrowing, because it shifts source indices.
• candle instead of ONNX Runtime: no native runtime to ship, so all three CI platforms build it unchanged.
• candle held at 0.9.2: from 0.10 it pulls in tokenizers and its C oniguruma build.
• No FAISS or sqlite-vec: ranking is a dot product over the filtered rows, which takes milliseconds at collection scale.
• Cutoff and staleness are ponytail: shortcuts:
  • The proportional cutoff was tuned on one collection.
  • Staleness is checked on length only.

Measured on this machine (16-core, CPU only)

• Speed: 86 ms per image to embed, about 30 ms per text query.
• Quality: on the Islandora sample photographs, "a building" and "a group of people standing outside" each ranked the right photo first. I checked both by eye.

Not done yet

1. Optional components (Notion task): the model weights should install through that mechanism.
2. Remote embedding endpoint: Infinity-style POST /embeddings, for institutions hosting their own. The Islandora server-side design is in the Notion doc "Repository-wide visual search for Islandora 2".
3. Faster models: SigLIP 2 base is the candidate. MobileCLIP2's licence is research-only.
4. OCR for scans without a text layer, and captioning / suggested subjects: both as staged drafts the archivist accepts.
5. Agent bridge: QuerySource::Search still matches cells only. Extending it means updating AGENTS.md.

Known limits

• PDF lock: reading a very long PDF holds the PDFium lock, which can pause gallery thumbnails (ponytail:).
• Deleted files: their vectors stay in __visual_index.

Verification

• Rebased on main at 0.5.0-beta.1, with no conflicts.
• Local CI: ./scripts/ci.sh (fmt, clippy -D warnings, cargo test --workspace) passed on Windows.
• Cross-platform lock: Cargo.lock resolves with --locked for x86_64/aarch64 macOS and x86_64 Linux.
• PDF tests: ran against real PDFium, with no skips.
• visual-search tests: ran against the downloaded weights.
• App: a real 1,893-row project was indexed and searched by hand. That testing found an observer loop that froze the UI (fixed in badf6a3) and the ranking and cutoff problems addressed here.
• Crash fix: reasoned from gpui's atlas, image and frame-loop source. The panic could not be reproduced in tests, because gpui's test renderer has no DirectX atlas.
• Not clicked through after the latest changes: narrowing, the breadth slider, Gallery search, the viewer following the selection, storage in .qrate, and fast scrolling after the crash fix are covered by unit tests only.

🤖 Generated with Claude Code