From 9cbc1ac633d207eedd0f2514d58eb9d13af982cb Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 22:42:51 +0000 Subject: [PATCH 1/3] Add unified cross-source search and macOS desktop organizer skills Proposal + two Cowork skills for indexing/searching across sources and safely organizing local files. - docs/unified-search-and-organization/PROPOSAL.md: master design covering the connector contract, normalized loom.doc.v1 record, hybrid BM25+dense ranking with live citations, local-first privacy model, and a phased rollout that maps onto the M365/Slack/Calendar tools already available. - skills/unified-search: read-only skill that fans out across email, chat, cloud docs, calendar and local files, dedupes by content hash, and returns ranked, cited results. Bundles architecture, connector map, and the loom.doc.v1 JSON schema. - skills/desktop-organizer: scan -> classify -> propose -> confirm -> execute -> report pipeline with a mandatory dry-run, undo manifest, never-delete quarantine, protected-path skipping, and an optional user-defined directory hierarchy. Includes a runnable reference implementation (organize.py) and an example config. Both skills follow the house skills//SKILL.md convention and reuse the existing MCP tool-loading model; the organizer script uses only stdlib + pyyaml (already a dependency). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WcZuofEEnVYEhZzKk4XAjm --- .../PROPOSAL.md | 174 +++++++++ skills/desktop-organizer/SKILL.md | 136 +++++++ .../reference/organize.config.example.yaml | 63 ++++ .../desktop-organizer/reference/taxonomy.md | 91 +++++ skills/desktop-organizer/scripts/organize.py | 342 ++++++++++++++++++ skills/unified-search/SKILL.md | 118 ++++++ .../unified-search/reference/architecture.md | 133 +++++++ skills/unified-search/reference/connectors.md | 53 +++ .../unified-search/reference/doc-schema.json | 104 ++++++ 9 files changed, 1214 insertions(+) create mode 100644 docs/unified-search-and-organization/PROPOSAL.md create mode 100644 skills/desktop-organizer/SKILL.md create mode 100644 skills/desktop-organizer/reference/organize.config.example.yaml create mode 100644 skills/desktop-organizer/reference/taxonomy.md create mode 100644 skills/desktop-organizer/scripts/organize.py create mode 100644 skills/unified-search/SKILL.md create mode 100644 skills/unified-search/reference/architecture.md create mode 100644 skills/unified-search/reference/connectors.md create mode 100644 skills/unified-search/reference/doc-schema.json diff --git a/docs/unified-search-and-organization/PROPOSAL.md b/docs/unified-search-and-organization/PROPOSAL.md new file mode 100644 index 0000000..1fab42e --- /dev/null +++ b/docs/unified-search-and-organization/PROPOSAL.md @@ -0,0 +1,174 @@ +# Unified Search, Indexing & Desktop Organization — Proposal + +> A design for weaving a **single searchable memory** across a user's email, local +> files, Teams, Slack, cloud docs and calendar — plus a **safe macOS desktop +> organizer** — delivered as two Claude Cowork skills that run against the +> connectors this environment already exposes. + +**Status:** Proposal / design. Two runnable skills ship alongside this document +(`skills/unified-search/`, `skills/desktop-organizer/`). Nothing in this proposal +touches user data until the user explicitly runs a skill. + +--- + +## 1. Goals + +1. **One search box over everything.** Ask a natural-language question and get + ranked, deduplicated, *cited* results drawn from every source the user has + connected — email, chat, files, docs, calendar. +2. **User-extensible sources.** Adding a new source (Notion, Jira, a local + folder) is dropping in a connector, not rewriting the engine. +3. **A trustworthy desktop organizer.** Claude can tidy a messy Mac — Downloads, + Desktop, Documents — into a clean, *optional* directory hierarchy, and it can + never lose or silently delete a file. +4. **Local-first & privacy-respecting.** The index lives on the user's machine; + source ACLs are honored; secrets are never indexed. + +## 2. Why two skills, not one + +Search/retrieval and file *mutation* have opposite risk profiles. Retrieval is +read-only and benefits from broad reach; organization moves bytes on disk and +demands a hard confirmation gate. Splitting them keeps the dangerous surface +small and independently auditable, and lets a user adopt search without ever +granting write access — and vice versa. + +| Skill | Direction | Risk | Trigger examples | +|-------|-----------|------|------------------| +| `unified-search` | read-only | low | "find the budget thread with Jane", "what did we decide about the launch date" | +| `desktop-organizer` | writes files | gated | "organize my Downloads", "clean up my Desktop" | + +## 3. What this environment already gives us + +Phase 0 is buildable **today** — the connectors below are already present, so the +search skill can wrap them as read-only sources with zero new auth: + +| Source | Tools already available | Normalized type | +|--------|-------------------------|-----------------| +| Outlook / Exchange email | `outlook_email_search` | `email` | +| Microsoft Teams chat | `chat_message_search`, `teams_list_chats` | `chat_message` | +| SharePoint / OneDrive | `sharepoint_search`, `sharepoint_folder_search` | `file` / `doc` | +| Outlook calendar | `outlook_calendar_search` | `calendar_event` | +| Slack + other chat (Macro) | `ContentSearch`, `ReadChannelMessages`, `ReadChat` | `chat_message` | +| Google Calendar | `list_events`, `search_events` | `calendar_event` | +| Local files (Cowork sandbox) | `Glob`, `Grep`, `Read` | `file` | + +Cowork adds Gmail, Google Drive and Slack as first-class OAuth connectors, which +Phase 1+ can promote to native delta-sync connectors. + +--- + +## 4. Unified search architecture + +Three pillars (full spec in [`../../skills/unified-search/reference/architecture.md`](../../skills/unified-search/reference/architecture.md)): + +### 4.1 A thin connector contract +Every source implements the same capability-negotiated interface — `list(cursor)`, +`fetchMetadata`, `fetchContent`, `fetchACL` — and declares what it can do +(`delta`, `contentFetch`, `acl`). The engine degrades gracefully when a +capability is missing (e.g. poll-on-query when there's no delta cursor). +Connectors register via a `connector.yaml` manifest, so users add sources without +touching the core. + +### 4.2 One normalized record — `loom.doc.v1` +Everything — an email, a Slack message, a PDF, a calendar invite — maps to a +single JSON record with a stable `id`, `source`, `type`, `title`, `participants`, +`timestamps`, a deep-link `location` for citation, a `snippet`, a `content_hash` +for dedup, `tags`, and `acl`. Source-specific fields survive in `raw`. Schema: +[`../../skills/unified-search/reference/doc-schema.json`](../../skills/unified-search/reference/doc-schema.json). + +### 4.3 Hybrid retrieval with live citations +- **BM25/FTS5** for precise term, name and ID matching ("invoice #4471"). +- **Dense embeddings** (prefer a *local* model) for semantic recall ("that + thread about the budget cut"). +- **Reciprocal Rank Fusion** (`score = Σ 1/(k+rank)`, k≈60) merges the two; + optional cross-encoder rerank on the top 50. +- Results are **deduplicated** across sources by `content_hash` (the same file in + Drive + local + as an email attachment collapses to one record with an + `also_in` list) and returned as **pointers with deep links** — the assistant + re-fetches live content before quoting, so it never serves stale data. + +### 4.4 Storage & sync +SQLite (records + FTS5) + a content-addressed blob store for extracted text + a +local vector index (sqlite-vec / LanceDB). Each `(connector, account)` keeps a +`sync_state` cursor; only changed etags trigger content fetch; tombstones +propagate deletes. Large binaries are text-extracted (Tika / pdfplumber / OCR), +never embedded raw. + +### 4.5 Privacy +Local-first and encrypted at rest; OAuth tokens in the OS keychain, never in +SQLite or logs; every query filters to what the connected identity may see; +`.env`, `id_rsa`, `*.pem`, muted channels and user exclude-globs are **never +indexed**, and a redaction pass strips detected secrets before embedding. + +### 4.6 Rollout +- **Phase 0 — MVP:** wrap the existing M365 / Slack / Calendar tools as read-only + connectors; normalized schema; FTS5/BM25; unified query + citations; + poll-on-query. Zero local-storage risk. +- **Phase 1:** persistent local index, incremental cursors, hybrid RRF ranking, + cross-source dedup. +- **Phase 2:** native macOS FSEvents connector (text extraction/OCR) + native + Gmail/Drive delta sync. +- **Phase 3:** publish the `SourceConnector` SDK + `connector.yaml` so users add + their own sources. + +--- + +## 5. Desktop organization system + +The organizer is a **scan → classify → propose → confirm → execute → report** +pipeline where every phase before "execute" is read-only and "execute" is gated +on explicit user approval. Full spec: +[`../../skills/desktop-organizer/reference/taxonomy.md`](../../skills/desktop-organizer/reference/taxonomy.md). + +### 5.1 Optional directory hierarchy +A sensible default tree (`~/Documents/Organized/` → Domain → Project → Year) that +the user can **fully override** by dropping an `organize.config.yaml` in the scan +root. Downloads and Desktop are staging areas drained into the tree; anything +ambiguous lands in `_Review/` with a reason. Example config: +[`../../skills/desktop-organizer/reference/organize.config.example.yaml`](../../skills/desktop-organizer/reference/organize.config.example.yaml). + +### 5.2 Classification +Destination is decided by combining signals, highest-confidence first: +extension + MIME (magic bytes, not just the extension) → type bucket; filename +tokens and content sniff → domain/project; EXIF/mtime/name dates → year-month +bucket; download-origin xattr → routing hint. Screenshots, junk (`.DS_Store`, +`*.part`, zero-byte, lock files), installers and duplicates (SHA-256) are each +handled explicitly. + +### 5.3 Non-negotiable safety invariants +1. **Mandatory dry-run.** A human-readable `PLAN.md` + machine `plan.json` is + produced first; no bytes move during scan or plan. +2. **Explicit confirmation** before any move; partial approvals honored. +3. **Undo manifest.** Every move is appended to `undo-.json` *before* + it commits; one command reverses everything. +4. **Never delete.** Installers/junk/duplicates are *quarantined* to `_Review/` + (or Trash if configured); deletion is only ever a suggestion the user runs. +5. **Protected paths skipped:** `~/Library`, `/System`, `/Applications`, `.app` + and `.photoslibrary` bundles, git repos (moved whole), `node_modules`, + dotfiles, symlinks, `*.icloud` placeholders. +6. **Locked/open files** are detected and deferred; cross-volume moves are + copy → verify hash → remove. + +A runnable reference implementation of this pipeline (dry-run planner + executor ++ undo) ships at +[`../../skills/desktop-organizer/scripts/organize.py`](../../skills/desktop-organizer/scripts/organize.py). + +--- + +## 6. How it maps onto this codebase + +- Both skills follow the house `skills//SKILL.md` convention (YAML + frontmatter: `name` + block-scalar `description` with trigger bullets and an + ``), matching `skills/long-runner-acceptance-test/`. +- Bundled scripts follow the `tasks/*/processor.py` precedent (a self-contained + Python module using only stdlib + `pyyaml`, which is already a dependency). +- Connectors reuse `client.py`'s MCP loading model (`mcp____*` tool + names, global + project resolution) — no new configuration mechanism. + +## 7. Open questions for the user + +1. **Local embedding model** — ship with a local embedder (privacy, offline) or + allow a remote one for quality? Default proposed: local-first. +2. **Organizer destination** — organize into a *copy tree* (`Documents/Organized`, + safest) or restructure `~/Documents` in place? Default proposed: copy tree. +3. **Which sources first** beyond the Phase-0 set already wired here? diff --git a/skills/desktop-organizer/SKILL.md b/skills/desktop-organizer/SKILL.md new file mode 100644 index 0000000..9a4c3c6 --- /dev/null +++ b/skills/desktop-organizer/SKILL.md @@ -0,0 +1,136 @@ +--- +name: desktop-organizer +description: | + Safely organize a messy macOS desktop system — Downloads, Desktop, Documents — + into a clean, OPTIONAL directory hierarchy. Classifies files by type, content, + and date; proposes a plan; and only moves files after explicit confirmation. + Never deletes: junk and duplicates are quarantined, and every move is reversible + via an undo manifest. + + Use this skill when: + - The user asks to "organize", "clean up", "tidy", "sort", or "file away" their + Downloads, Desktop, Documents, or a specific folder. + - The user wants a consistent folder structure or naming scheme for local files. + - The user complains about clutter, duplicate files, or "can't find anything". + - The user wants to define or apply their own folder taxonomy. + + + Context: The user's Downloads folder is a mess. + user: "my downloads folder is chaos, can you sort it out?" + assistant: I'll use the desktop-organizer skill. First I'll scan read-only and + show you a plan of every proposed move — nothing changes until you approve it. + + + + Context: The user wants their own structure. + user: "file my desktop into folders by project and year" + assistant: I'll run desktop-organizer with a project/year hierarchy, produce a + dry-run plan for your review, then execute with a reversible undo manifest. + +--- + +# macOS Desktop Organizer + +You are a careful macOS file-organization agent. You MOVE files into a tidy tree; +you NEVER delete and you NEVER touch protected locations. Reversibility is +non-negotiable. + +## What to do + +Run this pipeline in order. **Never skip a phase, and never move a byte before +Phase 3's plan has been approved.** + +### 1. SCAN (read-only) +Walk the configured scan paths. Load `organize.config.yaml` from the scan root or +`~/.config/cowork-organize/`; if absent, use the built-in defaults (see +[reference/taxonomy.md](reference/taxonomy.md)). For each file gather: name, +extension, MIME (magic bytes, not just extension), size, mtime, EXIF date, +content sniff for PDFs/images, download-origin xattr, and SHA-256. +**Skip entirely:** `~/Library`, `/System`, `/Applications`, `.app` and +`.photoslibrary` bundles, git repos (treat as one unit), `node_modules`, +dotfiles/hidden files, symlinks, `*.icloud` placeholders, and anything matching +the config's ignore globs. + +### 2. CLASSIFY +Apply the rubric in [reference/taxonomy.md](reference/taxonomy.md): extension + +MIME → type bucket; filename tokens and content → domain/project; dates → +year/month bucket. Detect screenshots, junk (`.DS_Store`, `*.part`, zero-byte, +lock files), installers, and duplicates (identical SHA-256). Assign each file a +destination, a proposed rename, and a confidence (high/medium/low). Low-confidence +or unmatched → `_Review/unsorted`. Duplicates → `_Review/duplicates` (keep the +best-located copy). Junk/installers → `_Review`, flagged "suggest delete" — do +NOT delete. + +### 3. PROPOSE (mandatory dry-run) +Produce `PLAN.md` (human-readable, grouped by action, with per-file reason + +confidence + flags + rename) and `plan.json` (machine-readable). Show counts and +total bytes. Make NO changes. Present the plan and STOP. + +### 4. CONFIRM +Wait for explicit user approval. Honor partial approvals and edits ("just do +Finance and Screenshots"). Do not proceed on silence. + +### 5. EXECUTE +For each approved move: recompute the destination, apply naming +(`YYYY-MM-DD_slug`, kebab-case, max 80 chars, `on_conflict` policy), check the +file isn't open/locked (skip + log if busy), and ensure disk space. **APPEND the +move to `undo-.json` BEFORE committing it.** Cross-volume moves are +copy → verify hash → remove source. Move whole git repos intact. + +### 6. REPORT +Summarize: moved, renamed, quarantined, skipped, deferred, errors. State the undo +command. List `_Review` contents with *suggested* (not executed) deletions. Never +claim a file was deleted — only moved or quarantined. + +## Configuration & taxonomy + +The default hierarchy, full classification rubric, naming rules, and the +user-overridable config schema are in +[reference/taxonomy.md](reference/taxonomy.md). A ready-to-copy config template is +[reference/organize.config.example.yaml](reference/organize.config.example.yaml). + +## Reference implementation + +[scripts/organize.py](scripts/organize.py) implements this exact pipeline +(stdlib + pyyaml only): +- `python organize.py scan` → dry-run, writes `PLAN.md` + `plan.json` +- `python organize.py execute --plan plan.json` → applies, writes `undo-.json` +- `python organize.py undo --manifest undo-.json` → reverses everything + +Use it to do the mechanical work deterministically, or follow the pipeline +manually with the filesystem tools — either way the safety invariants below hold. + +## Rules + +- Destructive actions are FORBIDDEN. When uncertain, quarantine to `_Review` and + ask. +- The dry-run plan and explicit confirmation are mandatory — never move a file the + user hasn't seen in a plan. +- Every executed move MUST be recorded in the undo manifest before it happens. +- Never delete: junk, installers, and duplicates are quarantined, deletion is only + ever a suggestion the user runs manually. +- Never touch protected paths (`~/Library`, `/System`, `/Applications`, `.app` / + `.photoslibrary` bundles, git repos, `node_modules`, dotfiles, symlinks, + `*.icloud`). +- The directory hierarchy is OPTIONAL and user-owned: config rules override + defaults, first match wins. +- Prefer organizing into a copy-tree (`~/Documents/Organized`) over restructuring + a folder in place, unless the user's config says otherwise. + +## Output + +After the dry-run, present the plan summary and ask for confirmation. After +execution, emit: + +```json +{ + "moved": 0, + "renamed": 0, + "quarantined": 0, + "skipped": 0, + "deferred": 0, + "errors": 0, + "undo_manifest": "undo-2026-08-01T12-00-00.json", + "review_suggested_deletions": ["_Review/junk/.DS_Store", "_Review/duplicates/..."] +} +``` diff --git a/skills/desktop-organizer/reference/organize.config.example.yaml b/skills/desktop-organizer/reference/organize.config.example.yaml new file mode 100644 index 0000000..0b13713 --- /dev/null +++ b/skills/desktop-organizer/reference/organize.config.example.yaml @@ -0,0 +1,63 @@ +# organize.config.yaml — user-configurable taxonomy for the desktop-organizer skill. +# +# Drop this file in your scan root (e.g. ~/Downloads) or at +# ~/.config/cowork-organize/organize.config.yaml. If it is absent, the built-in +# defaults apply. User rules are merged OVER defaults; the first matching rule +# wins. The directory hierarchy is entirely optional — delete `folders`/`rules` +# to fall back to defaults. + +version: 1 + +# Destination tree. The organizer moves files here rather than restructuring +# source folders in place. +root: ~/Documents/Organized + +# Folders to drain into the tree (staging areas). +scan: + - ~/Downloads + - ~/Desktop + +naming: + date_format: "%Y-%m-%d" # strftime; prefixed to slug when a date is meaningful + case: kebab # kebab | snake | title + max_len: 80 + +# Globs never touched, in addition to the always-protected system paths. +ignore: + - "**/.git/**" + - "**/*.app/**" + - "~/Library/**" + - "**/node_modules/**" + - "**/*.icloud" # iCloud placeholders (not downloaded) + - "**/.DS_Store" # matched here only to keep it out of scans; still quarantined if found + +# Your own taxonomy. `{project}`, `{year}`, `{month}` are filled from +# classification. These override the default tree. +folders: + Work: { path: "Work/{project}/{year}" } + Finance: { path: "Personal/finance/{year}" } + Shots: { path: "Media/Screenshots/{year}-{month}" } + Media: { path: "Media/Photos/{year}/{year}-{month}" } + Code: { path: "Code/{project}" } + +# First match wins. `to` references a key in `folders`, or a special target. +rules: + - match: { name_regex: "(?i)invoice|receipt|statement" } + to: Finance + - match: { ext: [png], name_regex: "(?i)screenshot|cleanshot|screen shot" } + to: Shots + - match: { ext: [jpg, jpeg, heic, png] } + to: Media + - match: { ext: [py, js, ts, go, rs, java] } + to: Code + - match: { ext: [dmg, pkg] } + to: quarantine # → Installers, flagged "suggest delete" + - match: { mime: "application/zip" } + to: "Archives/{year}" + +defaults: + unmatched: "_Review/unsorted" + quarantine: "_Review" + on_conflict: version # version | skip | hash-suffix + duplicates: quarantine # exact SHA-256 dupes → _Review/duplicates + never_delete: true # hard invariant; do not set to false diff --git a/skills/desktop-organizer/reference/taxonomy.md b/skills/desktop-organizer/reference/taxonomy.md new file mode 100644 index 0000000..a454737 --- /dev/null +++ b/skills/desktop-organizer/reference/taxonomy.md @@ -0,0 +1,91 @@ +# Desktop Organizer — Taxonomy, Rubric & Naming + +## 1. Default hierarchy (a sensible default, fully overridable) + +The default root is `~/Documents/Organized/` — the organizer never restructures +`~/Documents` in place unless the config says so. Pattern: **Domain → Project / +Category → Year → files**. Downloads and Desktop are *staging areas* drained into +this tree. + +``` +~/Documents/Organized/ +├── Work/ +│ ├── acme-corp/2026/2026-07-14_q3-budget.xlsx +│ └── _Unsorted/ # domain known, project unknown +├── Personal/ +│ ├── finance/2026/2026-06-01_bank-statement.pdf +│ ├── health/2026/ +│ └── travel/2026/ +├── Media/ +│ ├── Photos/2026/2026-07/ +│ ├── Screenshots/2026-07/ +│ └── Video/ +├── Code/ +│ └── / # left intact if it's a git repo +├── Reference/ # ebooks, manuals, papers +├── Archives/ # zips/tars +├── Installers/ # .dmg/.pkg → quarantine, suggest delete +└── _Review/ # ambiguous / duplicates / junk — NEVER auto-deleted + ├── unsorted/ + ├── duplicates/ + └── junk/ +``` + +Anything unclassifiable lands in `_Review/unsorted/` with a reason note. + +## 2. Classification rubric + +Decide the destination by combining signals, highest-confidence first: + +| Signal | Use | +|--------|-----| +| **Extension + MIME** | Primary type bucket. `.pdf`→docs, `.png/.jpg/.heic`→images, `.mov/.mp4`→video, `.zip/.tar/.gz`→archives, `.dmg/.pkg/.app`→installers, `.py/.js/.ts`→code, `.docx/.xlsx/.pptx`→documents. Read magic bytes — do not trust the extension. | +| **Content sniff** | PDF text scan → "Invoice/Statement/Boarding pass" → finance/travel. Light image inspection → screenshot vs. photo. | +| **Filename tokens** | Regex for `invoice`, `receipt`, `resume`, `screenshot`, project codes, client names → domain/project. | +| **Dates** | EXIF DateTaken (photos) > filesystem mtime > date parsed from name. Drives the `/YYYY/` and `YYYY-MM-DD` buckets. | +| **Source (optional)** | Download origin (`kMDItemWhereFroms` xattr), AirDrop/Slack tags → routing hints. | + +**Screenshots:** name matches `Screenshot* / CleanShot* / Screen Shot*`, or +dir=Desktop + PNG + no EXIF → `Media/Screenshots/YYYY-MM/`. + +**Desktop clutter:** loose files >7 days old on the Desktop are candidates; +app aliases / `.app` / in-use folders are skipped. + +**Junk:** `.DS_Store`, `*.crdownload`, `*.part`, zero-byte files, `~$*` lock +files → `_Review/junk/` (never deleted). + +**Duplicates:** SHA-256 of contents. Exact match → keep the copy in the best +location, move the others to `_Review/duplicates/` with a pointer to the kept +copy. Near-dupes (same name, differing size) are flagged, never auto-removed. + +## 3. Naming conventions + +- **Date prefix when meaningful:** `YYYY-MM-DD_.` (e.g. + `2026-07-14_q3-budget.xlsx`). Photos use their EXIF date. +- **Slug:** lowercase; spaces/underscores → `-`; strip diacritics and unsafe + chars `/\:*?"<>|`; collapse repeats; trim to `max_len`; keep the (lowercased) + original extension. +- **Never destroy the original name** — it is stored in the undo manifest so every + rename is reversible. +- **Collisions (`on_conflict`):** `version` → append `-v2`, `-v3`; `hash-suffix` → + append the first 8 of the content hash; `skip` → leave in place and log. The + collision check is case-insensitive (APFS is case-insensitive by default). + +## 4. Protected / skipped paths (always) + +`~/Library`, `/System`, `/Applications`, `.app` bundles, `.photoslibrary`, git +repositories (moved whole, never split), `node_modules`, dotfiles/hidden files, +symlinks, and `*.icloud` placeholders (not yet downloaded). Locked/open files are +detected (`lsof`/flock) and deferred with a note. + +## 5. Safety invariants (restated — non-negotiable) + +1. Read-only scan; mandatory dry-run plan (`PLAN.md` + `plan.json`) before any + move. +2. Explicit confirmation required; partial approval supported. +3. Every move appended to `undo-.json` *before* it commits; an `undo` + run replays it in reverse. +4. Never delete — quarantine to `_Review/` (or macOS Trash if configured); + deletion is only ever a suggestion. +5. Cross-volume moves = copy → verify hash → remove source. Verify free disk + space first. diff --git a/skills/desktop-organizer/scripts/organize.py b/skills/desktop-organizer/scripts/organize.py new file mode 100644 index 0000000..c443b4c --- /dev/null +++ b/skills/desktop-organizer/scripts/organize.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +"""Reference implementation of the desktop-organizer safety pipeline. + +Deliberately dependency-light (stdlib + optional pyyaml) so it runs anywhere the +rest of this project runs. It enforces the skill's non-negotiable invariants: + + scan -> read-only; writes PLAN.md + plan.json, moves nothing + execute-> applies an approved plan.json; records undo-.json BEFORE each move + undo -> reverses a run from its undo manifest + +It NEVER deletes: junk/installers/duplicates are quarantined under _Review/. + +Usage: + python organize.py scan [--config organize.config.yaml] [--root DEST] + python organize.py execute --plan plan.json + python organize.py undo --manifest undo-.json +""" +from __future__ import annotations + +import argparse +import fnmatch +import hashlib +import json +import os +import re +import shutil +import sys +from datetime import datetime, timezone +from pathlib import Path + +try: + import yaml # pyyaml is already a project dependency +except ImportError: # pragma: no cover - graceful fallback + yaml = None + +# --- Always-protected paths, independent of user config ----------------------- +PROTECTED_PARTS = {"Library", "System", "Applications", "node_modules"} +PROTECTED_SUFFIXES = (".app", ".photoslibrary") +JUNK_NAMES = {".DS_Store"} +JUNK_GLOBS = ("*.crdownload", "*.part", "~$*") + +DEFAULT_CONFIG = { + "version": 1, + "root": "~/Documents/Organized", + "scan": ["~/Downloads", "~/Desktop"], + "naming": {"date_format": "%Y-%m-%d", "case": "kebab", "max_len": 80}, + "ignore": ["**/.git/**", "**/*.app/**", "~/Library/**", "**/node_modules/**", "**/*.icloud"], + "folders": {}, + "rules": [], + "defaults": { + "unmatched": "_Review/unsorted", + "quarantine": "_Review", + "on_conflict": "version", + "duplicates": "quarantine", + "never_delete": True, + }, +} + +EXT_BUCKETS = { + "pdf": "Reference", "docx": "Work", "xlsx": "Work", "pptx": "Work", + "png": "Media", "jpg": "Media", "jpeg": "Media", "heic": "Media", + "mov": "Media", "mp4": "Media", + "zip": "Archives", "tar": "Archives", "gz": "Archives", + "dmg": "quarantine", "pkg": "quarantine", + "py": "Code", "js": "Code", "ts": "Code", "go": "Code", "rs": "Code", +} + + +def load_config(path: str | None) -> dict: + cfg = json.loads(json.dumps(DEFAULT_CONFIG)) # deep copy + if path and Path(path).expanduser().exists(): + if yaml is None: + print("warning: pyyaml not installed; using defaults", file=sys.stderr) + else: + user = yaml.safe_load(Path(path).expanduser().read_text()) or {} + cfg.update({k: v for k, v in user.items() if v is not None}) + return cfg + + +def sha256(p: Path, limit: int = 64 * 1024 * 1024) -> str: + h = hashlib.sha256() + with p.open("rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + if f.tell() > limit: + break + return "sha256:" + h.hexdigest() + + +def is_protected(p: Path) -> bool: + if any(part in PROTECTED_PARTS for part in p.parts): + return True + if any(str(p).find(sfx + os.sep) != -1 or p.name.endswith(sfx) for sfx in PROTECTED_SUFFIXES): + return True + if p.is_symlink() or p.name.startswith("."): + return True + if p.name.endswith(".icloud"): + return True + # git repos are treated as one unit; skip files inside them + if any((anc / ".git").exists() for anc in list(p.parents)[:6]): + return True + return False + + +def matches_ignore(p: Path, globs: list[str]) -> bool: + s = str(p) + return any(fnmatch.fnmatch(s, os.path.expanduser(g)) for g in globs) + + +def is_junk(p: Path) -> bool: + if p.name in JUNK_NAMES or any(fnmatch.fnmatch(p.name, g) for g in JUNK_GLOBS): + return True + try: + return p.stat().st_size == 0 + except OSError: + return False + + +def slugify(name: str, case: str, max_len: int) -> str: + stem, ext = os.path.splitext(name) + stem = re.sub(r"[/\\:*?\"<>|]", "", stem) + stem = re.sub(r"[\s_]+", "-", stem.strip()) + stem = re.sub(r"-{2,}", "-", stem).strip("-").lower() + if case == "snake": + stem = stem.replace("-", "_") + return (stem[:max_len] or "file") + ext.lower() + + +def file_date(p: Path, fmt: str) -> str: + ts = p.stat().st_mtime + return datetime.fromtimestamp(ts, tz=timezone.utc).strftime(fmt) + + +def classify(p: Path, cfg: dict) -> tuple[str, str]: + """Return (destination_subpath, reason).""" + name = p.name + ext = p.suffix.lower().lstrip(".") + dfmt = cfg["naming"]["date_format"] + year = file_date(p, "%Y") + month = file_date(p, "%m") + + # user rules first (first match wins) + for rule in cfg.get("rules", []): + m = rule.get("match", {}) + if "ext" in m and ext not in [e.lower() for e in m["ext"]]: + continue + if "name_regex" in m and not re.search(m["name_regex"], name): + continue + target = rule.get("to") + folder = cfg.get("folders", {}).get(target, {"path": target}) + path_tmpl = folder["path"] if isinstance(folder, dict) else str(folder) + if target == "quarantine": + return f"{cfg['defaults']['quarantine']}/installers", "rule:quarantine" + return path_tmpl.format(project="_Unsorted", year=year, month=month), f"rule:{target}" + + # screenshot heuristic + if ext == "png" and re.search(r"(?i)screenshot|cleanshot|screen shot", name): + return f"Media/Screenshots/{year}-{month}", "screenshot" + + # extension bucket + bucket = EXT_BUCKETS.get(ext) + if bucket == "quarantine": + return f"{cfg['defaults']['quarantine']}/installers", "installer" + if bucket in ("Media",): + return f"Media/Photos/{year}/{year}-{month}", "ext:media" + if bucket: + return f"{bucket}/{year}", f"ext:{ext}" + + return cfg["defaults"]["unmatched"], "unmatched" + + +def dest_path(root: Path, sub: str, p: Path, cfg: dict) -> Path: + d = file_date(p, cfg["naming"]["date_format"]) + slug = slugify(p.name, cfg["naming"]["case"], cfg["naming"]["max_len"]) + if not slug.lower().startswith(d): + slug = f"{d}_{slug}" + return root / sub / slug + + +def resolve_conflict(dest: Path, policy: str, content_hash: str) -> Path | None: + if not dest.exists(): + return dest + if policy == "skip": + return None + stem, ext = os.path.splitext(dest.name) + if policy == "hash-suffix": + return dest.with_name(f"{stem}-{content_hash.split(':')[1][:8]}{ext}") + n = 2 + while True: + cand = dest.with_name(f"{stem}-v{n}{ext}") + if not cand.exists(): + return cand + n += 1 + + +def cmd_scan(args) -> None: + cfg = load_config(args.config) + root = Path(args.root or cfg["root"]).expanduser() + seen_hashes: dict[str, str] = {} + moves, quarantines, junk, skipped = [], [], [], [] + + for scan_dir in cfg["scan"]: + base = Path(scan_dir).expanduser() + if not base.exists(): + continue + for p in base.rglob("*"): + if not p.is_file(): + continue + if is_protected(p) or matches_ignore(p, cfg["ignore"]): + skipped.append({"path": str(p), "reason": "protected/ignored"}) + continue + if is_junk(p): + junk.append({"src": str(p), "dst": str(root / "_Review/junk" / p.name), + "reason": "junk", "action": "quarantine"}) + continue + ch = sha256(p) + if ch in seen_hashes: + quarantines.append({"src": str(p), + "dst": str(root / "_Review/duplicates" / p.name), + "reason": f"duplicate of {seen_hashes[ch]}", + "action": "quarantine", "hash": ch}) + continue + seen_hashes[ch] = str(p) + sub, reason = classify(p, cfg) + dest = dest_path(root, sub, p, cfg) + dest = resolve_conflict(dest, cfg["defaults"]["on_conflict"], ch) + if dest is None: + skipped.append({"path": str(p), "reason": "conflict:skip"}) + continue + moves.append({"src": str(p), "dst": str(dest), "reason": reason, + "action": "move", "hash": ch}) + + plan = { + "generated": file_date(Path(cfg["scan"][0]).expanduser(), "%Y-%m-%dT%H-%M-%S") + if Path(cfg["scan"][0]).expanduser().exists() else "unknown", + "root": str(root), + "moves": moves, "quarantines": quarantines, "junk": junk, "skipped": skipped, + } + Path("plan.json").write_text(json.dumps(plan, indent=2)) + _write_plan_md(plan) + print(f"DRY RUN — nothing moved.\n moves: {len(moves)}\n" + f" duplicates: {len(quarantines)}\n junk: {len(junk)}\n" + f" skipped: {len(skipped)}\n\nReview PLAN.md, then:\n" + f" python organize.py execute --plan plan.json") + + +def _write_plan_md(plan: dict) -> None: + lines = [f"# Organization Plan (DRY RUN)\n", f"Destination root: `{plan['root']}`\n"] + for title, key in [("Moves", "moves"), ("Duplicates → _Review", "quarantines"), + ("Junk → _Review", "junk"), ("Skipped (protected/ignored)", "skipped")]: + items = plan.get(key, []) + lines.append(f"\n## {title} ({len(items)})\n") + for it in items[:1000]: + if "src" in it: + lines.append(f"- `{it['src']}` → `{it['dst']}` _({it.get('reason','')})_") + else: + lines.append(f"- `{it['path']}` _({it.get('reason','')})_") + lines.append("\n> Nothing has been moved. Deletion is never performed — " + "junk and duplicates are quarantined under `_Review/`.\n") + Path("PLAN.md").write_text("\n".join(lines)) + + +def _do_move(src: Path, dst: Path) -> None: + dst.parent.mkdir(parents=True, exist_ok=True) + if src.stat().st_dev != dst.parent.stat().st_dev: # cross-volume: copy+verify+remove + shutil.copy2(src, dst) + if sha256(src) != sha256(dst): + dst.unlink(missing_ok=True) + raise IOError(f"hash mismatch copying {src}") + src.unlink() + else: + shutil.move(str(src), str(dst)) + + +def cmd_execute(args) -> None: + plan = json.loads(Path(args.plan).read_text()) + ts = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H-%M-%S") if _now_ok() else "run" + undo_path = Path(f"undo-{ts}.json") + undo: list[dict] = [] + stats = {"moved": 0, "quarantined": 0, "deferred": 0, "errors": 0} + + for group, kind in [("moves", "moved"), ("quarantines", "quarantined"), ("junk", "quarantined")]: + for it in plan.get(group, []): + src, dst = Path(it["src"]), Path(it["dst"]) + if not src.exists(): + stats["deferred"] += 1 + continue + entry = {"src": str(src), "dst": str(dst), "orig_name": src.name} + undo.append(entry) + undo_path.write_text(json.dumps(undo, indent=2)) # record BEFORE the move + try: + _do_move(src, dst) + stats[kind] += 1 + except Exception as e: # noqa: BLE001 + stats["errors"] += 1 + entry["error"] = str(e) + undo.pop() + undo_path.write_text(json.dumps(undo, indent=2)) + + print(json.dumps({**stats, "undo_manifest": str(undo_path)}, indent=2)) + print(f"\nTo reverse: python organize.py undo --manifest {undo_path}") + + +def cmd_undo(args) -> None: + entries = json.loads(Path(args.manifest).read_text()) + restored = 0 + for entry in reversed(entries): + dst, src = Path(entry["dst"]), Path(entry["src"]) + if dst.exists(): + src.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(dst), str(src)) + restored += 1 + print(f"Reversed {restored} move(s) from {args.manifest}.") + + +def _now_ok() -> bool: + try: + datetime.now() + return True + except Exception: # pragma: no cover + return False + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + sub = ap.add_subparsers(dest="cmd", required=True) + s = sub.add_parser("scan", help="dry-run: write PLAN.md + plan.json") + s.add_argument("--config") + s.add_argument("--root") + s.set_defaults(func=cmd_scan) + e = sub.add_parser("execute", help="apply an approved plan.json") + e.add_argument("--plan", required=True) + e.set_defaults(func=cmd_execute) + u = sub.add_parser("undo", help="reverse a run from its undo manifest") + u.add_argument("--manifest", required=True) + u.set_defaults(func=cmd_undo) + args = ap.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/skills/unified-search/SKILL.md b/skills/unified-search/SKILL.md new file mode 100644 index 0000000..5198fb5 --- /dev/null +++ b/skills/unified-search/SKILL.md @@ -0,0 +1,118 @@ +--- +name: unified-search +description: | + Search and retrieve across ALL the user's connected sources at once — email + (Outlook/Gmail), chat (Slack, Teams), cloud docs (SharePoint, Drive, OneDrive), + local files, and calendar — then return ranked, deduplicated, cited results. + New sources plug in without changing the engine. + + Use this skill when: + - The user asks to "find", "search", "look up", "where is", or "pull up" + something that could live in more than one place. + - The user references content vaguely ("that thread with Jane about the budget", + "the deck from last week") and you must locate the source of truth. + - The user wants a consolidated answer drawn from several tools/systems. + - The user wants to build or extend a cross-source index/connector. + + + Context: The user can't remember where a decision was recorded. + user: "where did we land on the launch date? it was either slack or email" + assistant: I'll use the unified-search skill to query Slack and email together, + dedupe, and cite the source of the decision. + + + + Context: The user wants a single view across systems. + user: "find everything about the Q3 budget from the last month" + assistant: I'll run unified-search across email, chat, and cloud docs, merge and + rank the results, and give you cited links back to each source. + +--- + +# Unified Cross-Source Search + +You are a retrieval agent. Your job is to answer "where is / find / what did we +decide" questions by querying **every relevant connected source**, merging the +results into one ranked, deduplicated list, and **citing each result with a deep +link** back to its origin. You retrieve pointers — you re-fetch live content +before quoting so you never serve stale data. + +## What to do + +Follow this procedure on every search request: + +### 1. Plan the query +Parse the user's request into: +- **terms** — keywords, names, IDs, quoted phrases. +- **filters** — sources, item types, date range, people involved. +- **intent** — a specific item ("invoice #4471" → keyword) vs. a fuzzy recall + ("that thread about the budget cut" → semantic). Pick `mode`: `keyword`, + `semantic`, or `hybrid` (default `hybrid`). + +### 2. Fan out to sources (read-only) +Query only the sources that match the filters, in parallel where possible. Map +each source to the tools available in this environment — see +[reference/connectors.md](reference/connectors.md) for the exact tool per source. +If a source isn't connected, note it in the results rather than failing silently. + +### 3. Normalize every hit +Map each raw result into the `loom.doc.v1` record shape (id, source, type, title, +author, participants, timestamps, `location` deep-link, snippet, content_hash, +tags, acl). Schema: [reference/doc-schema.json](reference/doc-schema.json). + +### 4. Merge, dedup, rank +- Collapse duplicates by `content_hash` (same file in Drive + local + as an email + attachment → one record with an `also_in` list of the other locations). +- Rank with hybrid fusion (Reciprocal Rank Fusion of BM25 + semantic; k≈60). +- Keep the strongest ~20 unless the user asked for more. + +### 5. Answer with citations +Return a short direct answer first, then the ranked results, each with: title, +source, timestamp, author, a one-line snippet, and a **clickable deep link**. If +you assert a fact, cite the specific item it came from. Before quoting a result +verbatim, re-fetch its live content. + +## Architecture & extension + +The full connector contract, normalized-record schema, hybrid-ranking design, +storage/sync model, privacy rules, and phased rollout are in +[reference/architecture.md](reference/architecture.md). Read it when the task is +to *build* or *extend* the index (add a source, design storage) rather than just +run a query. + +## Rules + +- DO stay read-only. This skill never moves, edits, or deletes anything. +- DO respect source permissions — only return what the connected identity can see. +- DO cite every result with a deep link; an uncited claim is a bug. +- DO note which sources were queried and which were unavailable. +- DO NOT index or surface secrets (`.env`, `id_rsa`, `*.pem`, credential files) + or content from muted/excluded channels. +- DO NOT quote cached snippets as authoritative — re-fetch before quoting. +- DO NOT invent a source link; if you can't produce a real deep link, say so. + +## Output + +Lead with a one-paragraph answer, then a results list. When a machine-readable +form is requested, emit: + +```json +{ + "answer": "One-paragraph direct answer with the key fact.", + "results": [ + { + "id": "sha256:...", + "type": "email|chat_message|file|doc|calendar_event", + "title": "Re: Q3 budget", + "source": "outlook", + "author": "Jane ", + "timestamp": "2026-07-30T14:00:00Z", + "snippet": "...cut the travel line by 15%...", + "score": 0.87, + "citation": { "url": "https://...", "also_in": ["gdrive://1AbC"] } + } + ], + "sources_queried": ["outlook", "slack", "sharepoint"], + "sources_unavailable": ["gmail"] +} +``` diff --git a/skills/unified-search/reference/architecture.md b/skills/unified-search/reference/architecture.md new file mode 100644 index 0000000..009d20b --- /dev/null +++ b/skills/unified-search/reference/architecture.md @@ -0,0 +1,133 @@ +# Unified Search — Architecture Reference + +Read this when *building or extending* the index (adding a source, designing +storage, changing ranking) — not for a routine query. + +## 1. Source-connector abstraction + +Every source implements one capability-negotiated contract. The engine knows +nothing source-specific beyond it. Connectors register via a `connector.yaml` +manifest (`id`, auth type, required scopes, capability flags). + +```typescript +interface SourceConnector { + id: string; // "outlook", "slack", "sharepoint", "local-fs" + capabilities: { + delta: boolean; // supports incremental cursor + contentFetch: boolean; + acl: boolean; + }; + + authenticate(ctx: AuthContext): Promise; // scoped token/handle + + // Enumerate items changed since cursor (full scan if cursor null) + list(cursor: string | null, opts: { pageSize: number }) + : AsyncIterator<{ items: ItemRef[]; nextCursor: string; done: boolean }>; + + fetchMetadata(ref: ItemRef): Promise; + fetchContent(ref: ItemRef): Promise<{ mime: string; bytes?: Uint8Array; text?: string }>; + fetchACL(ref: ItemRef): Promise; // optional if !capabilities.acl +} + +type ItemRef = { sourceId: string; nativeId: string; etag: string; + changeType: "upsert" | "delete" }; +``` + +Cursors are opaque per-connector strings: Outlook/Graph `deltaLink`, Gmail +`historyId`, Drive `pageToken`, Slack `oldest` ts, local-fs FSEvents id / mtime +watermark. When a capability is missing, degrade: no `delta` → poll-on-query with +short TTL; no `acl` → treat as private-to-user. + +## 2. Normalized record — `loom.doc.v1` + +Everything maps to one record; source-specific fields survive in `raw`. Formal +schema: `doc-schema.json`. + +Key fields: `id` = `sha256(source_id + native_id)`; `content_hash` = +`sha256(normalized_text)` (dedup key); `location.url` = deep link for citation; +`acl.principals` = who may see it. See the schema file for the complete shape and +an example record. + +## 3. Indexing strategy + +**Stored locally:** +- Normalized records → SQLite (`documents` table + FTS5 for BM25). +- Extracted plain text → content-addressed blob store (never store original + binaries unless the source is already local). +- Embeddings → local vector index (sqlite-vec / LanceDB), one vector per + ~512-token chunk. + +**Incremental sync.** Each `(connector, account)` keeps a `sync_state` row +`{cursor, last_full_scan, watermark}`. A scheduler polls per source (webhook/push +where available — Gmail push, Graph subscriptions). `list()` yields +upserts/deletes; only changed etags trigger `fetchContent`. Tombstones propagate +deletes to all three stores. + +**Dedup across sources.** Two levels: +1. *Exact* — `content_hash` collapses the same content (file in Drive + local + + email attachment) into one canonical record with `aliases[]` / `also_in`. +2. *Near* — MinHash/SimHash buckets catch forwarded emails and doc revisions; + keep newest as canonical, link the rest. + +**Large binaries.** Never embed raw. Pipeline: type-sniff → extract text (Tika / +pdfplumber / OCR for images) → chunk → discard bytes. Files over a size cap are +metadata-only with on-demand extraction at query time. + +**Hybrid retrieval (recommended).** Run both and fuse: +- BM25/FTS5 for exact term, name, ID matching (fast, cheap). +- Dense embeddings for semantic recall. +- **Reciprocal Rank Fusion**: `score = Σ 1/(k + rank)`, k≈60; optional + cross-encoder rerank on top 50. Embed lazily — index metadata + BM25 + immediately, backfill embeddings asynchronously. + +## 4. Query interface + +```json +// Request +{ + "q": "budget cut discussion with Jane", + "filters": { + "source": ["outlook", "slack", "sharepoint"], + "type": ["email", "chat_message"], + "date": { "gte": "2026-07-01", "lte": "2026-08-01" }, + "person": ["jane@x.com"] + }, + "mode": "hybrid", + "limit": 20 +} +``` + +Structured filters constrain the candidate set (SQL) first; hybrid ranking then +orders it. Response returns deduped canonical docs, each with a `citation.url` +deep link and `citation.also_in` for cross-source copies, plus `facets` for +drill-down (see SKILL.md "Output"). + +## 5. Privacy / security + +- **Local-first:** index, blobs, vectors on-device, encrypted at rest via a + keychain-derived key. Prefer a local embedding model so content never leaves + the machine. +- **Respect ACLs:** store effective `acl.principals`; every query filters to what + the connected identity can see; re-validate shared items on access. +- **Secrets:** OAuth/refresh tokens in the OS keychain, never in SQLite or logs; + connectors receive short-lived scoped handles. +- **Never indexed:** password vaults, `.env`, `id_rsa`, `*.pem`, user + exclude-globs, muted/private channels, un-opted-in sources. A redaction pass + strips detected secrets from extracted text before embedding. +- **User controls:** per-source enable, exclusion rules, "forget this item", + full local purge. + +## 6. Phased rollout + +- **Phase 0 — MVP:** wrap the tools already present in this environment (see + `connectors.md`) as read-only connectors. Normalized schema + SQLite/FTS5 (BM25 + only) + unified query + citations; sync = poll-on-query with short TTL. Zero + local-storage risk; proves the abstraction end to end. +- **Phase 1:** persistent local SQLite/blob/vector store, incremental cursors, + hybrid RRF ranking with local embeddings, cross-source dedup. +- **Phase 2:** macOS FSEvents connector (text extraction, OCR); native Gmail + + Google Drive delta connectors for real push sync. +- **Phase 3 — Extensibility GA:** publish the `SourceConnector` SDK + + `connector.yaml` manifest so users add sources (Notion, Jira, …) as drop-in + plugins; connector registry with capability negotiation and per-connector scope + prompts. diff --git a/skills/unified-search/reference/connectors.md b/skills/unified-search/reference/connectors.md new file mode 100644 index 0000000..97f7c6a --- /dev/null +++ b/skills/unified-search/reference/connectors.md @@ -0,0 +1,53 @@ +# Connector Map — source → tools in this environment + +How each normalized source maps to concrete tools. Tool names follow the +`mcp____` convention `client.py` already uses. When a source has no +connected tool, report it as `sources_unavailable` rather than failing the query. + +## Available today (Phase 0 — read-only, no new auth) + +| Source | Normalized `type` | Search tool(s) | Read/context tool(s) | +|--------|-------------------|----------------|----------------------| +| Outlook / Exchange email | `email` | `mcp__Microsoft_365__outlook_email_search` | `mcp__Microsoft_365__read_resource` | +| Microsoft Teams | `chat_message` | `mcp__Microsoft_365__chat_message_search` | `mcp__Microsoft_365__teams_list_chats`, `read_resource` | +| SharePoint / OneDrive | `file` / `doc` | `mcp__Microsoft_365__sharepoint_search`, `sharepoint_folder_search` | `read_resource` | +| Outlook calendar | `calendar_event` | `mcp__Microsoft_365__outlook_calendar_search` | `read_resource` | +| Slack / other chat (Macro) | `chat_message` | `mcp__Macro__ContentSearch`, `mcp__Macro__NameSearch` | `mcp__Macro__ReadChannelMessages`, `ReadChannelThread`, `ReadChat`, `ReadContent` | +| Google Calendar | `calendar_event` | `mcp__Google_Calendar__search_events` | `mcp__Google_Calendar__list_events`, `get_event` | +| Local files (sandbox) | `file` | `Grep`, `Glob` | `Read` | +| GitHub (code/issues/PRs) | `doc` / `file` | `mcp__github__search_code`, `search_issues`, `search_pull_requests` | `mcp__github__get_file_contents`, `pull_request_read` | + +## Cowork-native connectors (promote in Phase 1+) + +Claude Cowork exposes these as first-class OAuth connectors; wrap them the same +way and, where the API offers a delta cursor, upgrade from poll-on-query to real +incremental sync. + +| Source | Read | Write | Notes | +|--------|------|-------|-------| +| Gmail | full | drafts only | Google Workspace connector | +| Google Drive | full | full | delta via `pageToken` | +| Google Calendar | full | full | | +| Slack | channels/threads | post / draft | | +| Microsoft 365 (Outlook) | full | read-only by default | admin can enable write | +| Microsoft 365 (Teams) | full | read-only | always read-only | +| SharePoint / OneDrive | full | admin-gated | | + +## Adding a new source + +1. Write a `connector.yaml` (`id`, auth type, scopes, capability flags: + `delta` / `contentFetch` / `acl`). +2. Implement the `SourceConnector` contract (see `architecture.md` §1): `list`, + `fetchMetadata`, `fetchContent`, `fetchACL`. +3. Provide a `normalize()` that maps the source's raw items into `loom.doc.v1` + (`doc-schema.json`), including a real `location.url` deep link and a + `content_hash`. +4. Register it; the engine picks it up via capability negotiation — no core + changes. + +## Query-time fan-out rules + +- Only call tools for sources that pass the request's `filters.source`. +- Run source queries in parallel where the tools allow. +- Cap per-source results before merging (e.g. top 25/source) to keep fusion cheap. +- Always capture a deep link at fetch time — it is required for citation. diff --git a/skills/unified-search/reference/doc-schema.json b/skills/unified-search/reference/doc-schema.json new file mode 100644 index 0000000..049a246 --- /dev/null +++ b/skills/unified-search/reference/doc-schema.json @@ -0,0 +1,104 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "loom.doc.v1", + "title": "Unified normalized document record", + "description": "Every item from every source maps into this shape. Source-specific fields are preserved under `raw`.", + "type": "object", + "required": ["id", "source", "type", "timestamps", "location", "content_hash"], + "properties": { + "id": { + "type": "string", + "description": "sha256(source.id + native_id) — stable global identifier." + }, + "source": { + "type": "object", + "required": ["id"], + "properties": { + "id": { "type": "string", "description": "Connector id, e.g. outlook, slack, sharepoint, local-fs." }, + "account": { "type": "string", "description": "Which connected account this came from." } + } + }, + "type": { + "type": "string", + "enum": ["email", "chat_message", "file", "doc", "calendar_event", "contact"] + }, + "title": { "type": "string" }, + "author": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "email": { "type": "string" }, + "id": { "type": "string" } + } + }, + "participants": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "email": { "type": "string" }, + "role": { "type": "string", "enum": ["to", "cc", "bcc", "attendee", "member", "author"] } + } + } + }, + "timestamps": { + "type": "object", + "properties": { + "created": { "type": "string", "format": "date-time" }, + "modified": { "type": "string", "format": "date-time" }, + "event_start": { "type": ["string", "null"], "format": "date-time" }, + "event_end": { "type": ["string", "null"], "format": "date-time" } + } + }, + "location": { + "type": "object", + "description": "Where the item lives; url is the citation deep link.", + "properties": { + "url": { "type": "string", "description": "Deep link back to the source item (used for citation)." }, + "path": { "type": "string", "description": "Local filesystem path, if applicable." }, + "container": { "type": "string", "description": "Channel, folder, label, or mailbox." } + } + }, + "snippet": { "type": "string", "description": "Short preview (~280 chars) for display." }, + "body_ref": { "type": "string", "description": "Pointer to extracted full text in the blob store, e.g. blobstore://sha256..." }, + "content_hash": { "type": "string", "description": "sha256(normalized_text) — cross-source dedup key." }, + "aliases": { + "type": "array", + "items": { "type": "string" }, + "description": "Other locations of the same content (also_in)." + }, + "tags": { "type": "array", "items": { "type": "string" } }, + "acl": { + "type": "object", + "properties": { + "visibility": { "type": "string", "enum": ["private", "shared", "public"] }, + "principals": { "type": "array", "items": { "type": "string" } } + } + }, + "lang": { "type": "string" }, + "raw": { "type": "object", "description": "Verbatim source-specific fields." } + }, + "examples": [ + { + "id": "sha256:ab12...", + "source": { "id": "outlook", "account": "user@corp.com" }, + "type": "email", + "title": "Re: Q3 budget review", + "author": { "name": "Jane", "email": "jane@x.com" }, + "participants": [{ "name": "Bob", "email": "bob@x.com", "role": "to" }], + "timestamps": { "created": "2026-07-30T14:00:00Z", "modified": "2026-07-31T09:12:00Z" }, + "location": { + "url": "https://outlook.office.com/mail/id/AAMk...", + "container": "Inbox/Finance" + }, + "snippet": "First 280 chars of the message for preview...", + "body_ref": "blobstore://sha256:cd34...", + "content_hash": "sha256:cd34...", + "aliases": ["sharepoint://sites/finance/Shared%20Documents/q3-budget.xlsx"], + "tags": ["finance", "auto:has_attachment"], + "acl": { "visibility": "shared", "principals": ["user@corp.com", "group:finance"] }, + "lang": "en" + } + ] +} From bb739421b5618a5f9da6124bc0c53df0297571dc Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 2 Aug 2026 22:53:47 +0000 Subject: [PATCH 2/3] Confirm recommended defaults in proposal Convert the three open questions into confirmed decisions: local-first embeddings, copy-tree organizer destination, and Phase-0 sources first. These already match what the shipped skills and organize.py implement. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WcZuofEEnVYEhZzKk4XAjm --- .../PROPOSAL.md | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/unified-search-and-organization/PROPOSAL.md b/docs/unified-search-and-organization/PROPOSAL.md index 1fab42e..9839e76 100644 --- a/docs/unified-search-and-organization/PROPOSAL.md +++ b/docs/unified-search-and-organization/PROPOSAL.md @@ -165,10 +165,20 @@ A runnable reference implementation of this pipeline (dry-run planner + executor - Connectors reuse `client.py`'s MCP loading model (`mcp____*` tool names, global + project resolution) — no new configuration mechanism. -## 7. Open questions for the user - -1. **Local embedding model** — ship with a local embedder (privacy, offline) or - allow a remote one for quality? Default proposed: local-first. -2. **Organizer destination** — organize into a *copy tree* (`Documents/Organized`, - safest) or restructure `~/Documents` in place? Default proposed: copy tree. -3. **Which sources first** beyond the Phase-0 set already wired here? +## 7. Confirmed decisions + +These were the three defaults recommended for the user's call; all three are +**confirmed** and are what the shipped skills and `organize.py` implement: + +1. **Embeddings — local-first.** Ship with a local embedding model so content + never leaves the machine; a remote embedder stays available as an opt-in for + quality-sensitive workloads. (`unified-search/reference/architecture.md` §3, §5.) +2. **Organizer destination — copy tree.** Organize into `~/Documents/Organized` + rather than restructuring `~/Documents` in place — the safest option and the + default `organize.config.yaml` `root`. In-place remains available by changing + `root` and `scan`. (`desktop-organizer/reference/taxonomy.md` §1.) +3. **Sources — Phase-0 set first.** Start with the connectors already present in + this environment (Outlook email, Teams, SharePoint/OneDrive, Slack via Macro, + Google/Outlook calendar, local files, GitHub), then promote Cowork-native + Gmail/Drive/Slack to delta-sync connectors in Phase 1+. + (`unified-search/reference/connectors.md`.) From 7b8d418e151d077c41a67b347c9ff464c30e9f7c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 3 Aug 2026 00:21:23 +0000 Subject: [PATCH 3/3] Drop Jira/Atlassian from extensibility examples Replace with Notion/Linear in the user-extensible-source examples. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WcZuofEEnVYEhZzKk4XAjm --- docs/unified-search-and-organization/PROPOSAL.md | 2 +- skills/unified-search/reference/architecture.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/unified-search-and-organization/PROPOSAL.md b/docs/unified-search-and-organization/PROPOSAL.md index 9839e76..a965fb6 100644 --- a/docs/unified-search-and-organization/PROPOSAL.md +++ b/docs/unified-search-and-organization/PROPOSAL.md @@ -16,7 +16,7 @@ touches user data until the user explicitly runs a skill. 1. **One search box over everything.** Ask a natural-language question and get ranked, deduplicated, *cited* results drawn from every source the user has connected — email, chat, files, docs, calendar. -2. **User-extensible sources.** Adding a new source (Notion, Jira, a local +2. **User-extensible sources.** Adding a new source (Notion, Linear, a local folder) is dropping in a connector, not rewriting the engine. 3. **A trustworthy desktop organizer.** Claude can tidy a messy Mac — Downloads, Desktop, Documents — into a clean, *optional* directory hierarchy, and it can diff --git a/skills/unified-search/reference/architecture.md b/skills/unified-search/reference/architecture.md index 009d20b..56f1daa 100644 --- a/skills/unified-search/reference/architecture.md +++ b/skills/unified-search/reference/architecture.md @@ -128,6 +128,6 @@ drill-down (see SKILL.md "Output"). - **Phase 2:** macOS FSEvents connector (text extraction, OCR); native Gmail + Google Drive delta connectors for real push sync. - **Phase 3 — Extensibility GA:** publish the `SourceConnector` SDK + - `connector.yaml` manifest so users add sources (Notion, Jira, …) as drop-in + `connector.yaml` manifest so users add sources (Notion, Linear, …) as drop-in plugins; connector registry with capability negotiation and per-connector scope prompts.