diff --git a/docs/unified-search-and-organization/PROPOSAL.md b/docs/unified-search-and-organization/PROPOSAL.md new file mode 100644 index 0000000..a965fb6 --- /dev/null +++ b/docs/unified-search-and-organization/PROPOSAL.md @@ -0,0 +1,184 @@ +# Unified Search, Indexing & Desktop Organization — Proposal + +> A design for weaving a **single searchable memory** across a user's email, local +> files, Teams, Slack, cloud docs and calendar — plus a **safe macOS desktop +> organizer** — delivered as two Claude Cowork skills that run against the +> connectors this environment already exposes. + +**Status:** Proposal / design. Two runnable skills ship alongside this document +(`skills/unified-search/`, `skills/desktop-organizer/`). Nothing in this proposal +touches user data until the user explicitly runs a skill. + +--- + +## 1. Goals + +1. **One search box over everything.** Ask a natural-language question and get + ranked, deduplicated, *cited* results drawn from every source the user has + connected — email, chat, files, docs, calendar. +2. **User-extensible sources.** Adding a new source (Notion, Linear, a local + folder) is dropping in a connector, not rewriting the engine. +3. **A trustworthy desktop organizer.** Claude can tidy a messy Mac — Downloads, + Desktop, Documents — into a clean, *optional* directory hierarchy, and it can + never lose or silently delete a file. +4. **Local-first & privacy-respecting.** The index lives on the user's machine; + source ACLs are honored; secrets are never indexed. + +## 2. Why two skills, not one + +Search/retrieval and file *mutation* have opposite risk profiles. Retrieval is +read-only and benefits from broad reach; organization moves bytes on disk and +demands a hard confirmation gate. Splitting them keeps the dangerous surface +small and independently auditable, and lets a user adopt search without ever +granting write access — and vice versa. + +| Skill | Direction | Risk | Trigger examples | +|-------|-----------|------|------------------| +| `unified-search` | read-only | low | "find the budget thread with Jane", "what did we decide about the launch date" | +| `desktop-organizer` | writes files | gated | "organize my Downloads", "clean up my Desktop" | + +## 3. What this environment already gives us + +Phase 0 is buildable **today** — the connectors below are already present, so the +search skill can wrap them as read-only sources with zero new auth: + +| Source | Tools already available | Normalized type | +|--------|-------------------------|-----------------| +| Outlook / Exchange email | `outlook_email_search` | `email` | +| Microsoft Teams chat | `chat_message_search`, `teams_list_chats` | `chat_message` | +| SharePoint / OneDrive | `sharepoint_search`, `sharepoint_folder_search` | `file` / `doc` | +| Outlook calendar | `outlook_calendar_search` | `calendar_event` | +| Slack + other chat (Macro) | `ContentSearch`, `ReadChannelMessages`, `ReadChat` | `chat_message` | +| Google Calendar | `list_events`, `search_events` | `calendar_event` | +| Local files (Cowork sandbox) | `Glob`, `Grep`, `Read` | `file` | + +Cowork adds Gmail, Google Drive and Slack as first-class OAuth connectors, which +Phase 1+ can promote to native delta-sync connectors. + +--- + +## 4. Unified search architecture + +Three pillars (full spec in [`../../skills/unified-search/reference/architecture.md`](../../skills/unified-search/reference/architecture.md)): + +### 4.1 A thin connector contract +Every source implements the same capability-negotiated interface — `list(cursor)`, +`fetchMetadata`, `fetchContent`, `fetchACL` — and declares what it can do +(`delta`, `contentFetch`, `acl`). The engine degrades gracefully when a +capability is missing (e.g. poll-on-query when there's no delta cursor). +Connectors register via a `connector.yaml` manifest, so users add sources without +touching the core. + +### 4.2 One normalized record — `loom.doc.v1` +Everything — an email, a Slack message, a PDF, a calendar invite — maps to a +single JSON record with a stable `id`, `source`, `type`, `title`, `participants`, +`timestamps`, a deep-link `location` for citation, a `snippet`, a `content_hash` +for dedup, `tags`, and `acl`. Source-specific fields survive in `raw`. Schema: +[`../../skills/unified-search/reference/doc-schema.json`](../../skills/unified-search/reference/doc-schema.json). + +### 4.3 Hybrid retrieval with live citations +- **BM25/FTS5** for precise term, name and ID matching ("invoice #4471"). +- **Dense embeddings** (prefer a *local* model) for semantic recall ("that + thread about the budget cut"). +- **Reciprocal Rank Fusion** (`score = Σ 1/(k+rank)`, k≈60) merges the two; + optional cross-encoder rerank on the top 50. +- Results are **deduplicated** across sources by `content_hash` (the same file in + Drive + local + as an email attachment collapses to one record with an + `also_in` list) and returned as **pointers with deep links** — the assistant + re-fetches live content before quoting, so it never serves stale data. + +### 4.4 Storage & sync +SQLite (records + FTS5) + a content-addressed blob store for extracted text + a +local vector index (sqlite-vec / LanceDB). Each `(connector, account)` keeps a +`sync_state` cursor; only changed etags trigger content fetch; tombstones +propagate deletes. Large binaries are text-extracted (Tika / pdfplumber / OCR), +never embedded raw. + +### 4.5 Privacy +Local-first and encrypted at rest; OAuth tokens in the OS keychain, never in +SQLite or logs; every query filters to what the connected identity may see; +`.env`, `id_rsa`, `*.pem`, muted channels and user exclude-globs are **never +indexed**, and a redaction pass strips detected secrets before embedding. + +### 4.6 Rollout +- **Phase 0 — MVP:** wrap the existing M365 / Slack / Calendar tools as read-only + connectors; normalized schema; FTS5/BM25; unified query + citations; + poll-on-query. Zero local-storage risk. +- **Phase 1:** persistent local index, incremental cursors, hybrid RRF ranking, + cross-source dedup. +- **Phase 2:** native macOS FSEvents connector (text extraction/OCR) + native + Gmail/Drive delta sync. +- **Phase 3:** publish the `SourceConnector` SDK + `connector.yaml` so users add + their own sources. + +--- + +## 5. Desktop organization system + +The organizer is a **scan → classify → propose → confirm → execute → report** +pipeline where every phase before "execute" is read-only and "execute" is gated +on explicit user approval. Full spec: +[`../../skills/desktop-organizer/reference/taxonomy.md`](../../skills/desktop-organizer/reference/taxonomy.md). + +### 5.1 Optional directory hierarchy +A sensible default tree (`~/Documents/Organized/` → Domain → Project → Year) that +the user can **fully override** by dropping an `organize.config.yaml` in the scan +root. Downloads and Desktop are staging areas drained into the tree; anything +ambiguous lands in `_Review/` with a reason. Example config: +[`../../skills/desktop-organizer/reference/organize.config.example.yaml`](../../skills/desktop-organizer/reference/organize.config.example.yaml). + +### 5.2 Classification +Destination is decided by combining signals, highest-confidence first: +extension + MIME (magic bytes, not just the extension) → type bucket; filename +tokens and content sniff → domain/project; EXIF/mtime/name dates → year-month +bucket; download-origin xattr → routing hint. Screenshots, junk (`.DS_Store`, +`*.part`, zero-byte, lock files), installers and duplicates (SHA-256) are each +handled explicitly. + +### 5.3 Non-negotiable safety invariants +1. **Mandatory dry-run.** A human-readable `PLAN.md` + machine `plan.json` is + produced first; no bytes move during scan or plan. +2. **Explicit confirmation** before any move; partial approvals honored. +3. **Undo manifest.** Every move is appended to `undo-.json` *before* + it commits; one command reverses everything. +4. **Never delete.** Installers/junk/duplicates are *quarantined* to `_Review/` + (or Trash if configured); deletion is only ever a suggestion the user runs. +5. **Protected paths skipped:** `~/Library`, `/System`, `/Applications`, `.app` + and `.photoslibrary` bundles, git repos (moved whole), `node_modules`, + dotfiles, symlinks, `*.icloud` placeholders. +6. **Locked/open files** are detected and deferred; cross-volume moves are + copy → verify hash → remove. + +A runnable reference implementation of this pipeline (dry-run planner + executor ++ undo) ships at +[`../../skills/desktop-organizer/scripts/organize.py`](../../skills/desktop-organizer/scripts/organize.py). + +--- + +## 6. How it maps onto this codebase + +- Both skills follow the house `skills//SKILL.md` convention (YAML + frontmatter: `name` + block-scalar `description` with trigger bullets and an + ``), matching `skills/long-runner-acceptance-test/`. +- Bundled scripts follow the `tasks/*/processor.py` precedent (a self-contained + Python module using only stdlib + `pyyaml`, which is already a dependency). +- Connectors reuse `client.py`'s MCP loading model (`mcp____*` tool + names, global + project resolution) — no new configuration mechanism. + +## 7. Confirmed decisions + +These were the three defaults recommended for the user's call; all three are +**confirmed** and are what the shipped skills and `organize.py` implement: + +1. **Embeddings — local-first.** Ship with a local embedding model so content + never leaves the machine; a remote embedder stays available as an opt-in for + quality-sensitive workloads. (`unified-search/reference/architecture.md` §3, §5.) +2. **Organizer destination — copy tree.** Organize into `~/Documents/Organized` + rather than restructuring `~/Documents` in place — the safest option and the + default `organize.config.yaml` `root`. In-place remains available by changing + `root` and `scan`. (`desktop-organizer/reference/taxonomy.md` §1.) +3. **Sources — Phase-0 set first.** Start with the connectors already present in + this environment (Outlook email, Teams, SharePoint/OneDrive, Slack via Macro, + Google/Outlook calendar, local files, GitHub), then promote Cowork-native + Gmail/Drive/Slack to delta-sync connectors in Phase 1+. + (`unified-search/reference/connectors.md`.) diff --git a/skills/desktop-organizer/SKILL.md b/skills/desktop-organizer/SKILL.md new file mode 100644 index 0000000..9a4c3c6 --- /dev/null +++ b/skills/desktop-organizer/SKILL.md @@ -0,0 +1,136 @@ +--- +name: desktop-organizer +description: | + Safely organize a messy macOS desktop system — Downloads, Desktop, Documents — + into a clean, OPTIONAL directory hierarchy. Classifies files by type, content, + and date; proposes a plan; and only moves files after explicit confirmation. + Never deletes: junk and duplicates are quarantined, and every move is reversible + via an undo manifest. + + Use this skill when: + - The user asks to "organize", "clean up", "tidy", "sort", or "file away" their + Downloads, Desktop, Documents, or a specific folder. + - The user wants a consistent folder structure or naming scheme for local files. + - The user complains about clutter, duplicate files, or "can't find anything". + - The user wants to define or apply their own folder taxonomy. + + + Context: The user's Downloads folder is a mess. + user: "my downloads folder is chaos, can you sort it out?" + assistant: I'll use the desktop-organizer skill. First I'll scan read-only and + show you a plan of every proposed move — nothing changes until you approve it. + + + + Context: The user wants their own structure. + user: "file my desktop into folders by project and year" + assistant: I'll run desktop-organizer with a project/year hierarchy, produce a + dry-run plan for your review, then execute with a reversible undo manifest. + +--- + +# macOS Desktop Organizer + +You are a careful macOS file-organization agent. You MOVE files into a tidy tree; +you NEVER delete and you NEVER touch protected locations. Reversibility is +non-negotiable. + +## What to do + +Run this pipeline in order. **Never skip a phase, and never move a byte before +Phase 3's plan has been approved.** + +### 1. SCAN (read-only) +Walk the configured scan paths. Load `organize.config.yaml` from the scan root or +`~/.config/cowork-organize/`; if absent, use the built-in defaults (see +[reference/taxonomy.md](reference/taxonomy.md)). For each file gather: name, +extension, MIME (magic bytes, not just extension), size, mtime, EXIF date, +content sniff for PDFs/images, download-origin xattr, and SHA-256. +**Skip entirely:** `~/Library`, `/System`, `/Applications`, `.app` and +`.photoslibrary` bundles, git repos (treat as one unit), `node_modules`, +dotfiles/hidden files, symlinks, `*.icloud` placeholders, and anything matching +the config's ignore globs. + +### 2. CLASSIFY +Apply the rubric in [reference/taxonomy.md](reference/taxonomy.md): extension + +MIME → type bucket; filename tokens and content → domain/project; dates → +year/month bucket. Detect screenshots, junk (`.DS_Store`, `*.part`, zero-byte, +lock files), installers, and duplicates (identical SHA-256). Assign each file a +destination, a proposed rename, and a confidence (high/medium/low). Low-confidence +or unmatched → `_Review/unsorted`. Duplicates → `_Review/duplicates` (keep the +best-located copy). Junk/installers → `_Review`, flagged "suggest delete" — do +NOT delete. + +### 3. PROPOSE (mandatory dry-run) +Produce `PLAN.md` (human-readable, grouped by action, with per-file reason + +confidence + flags + rename) and `plan.json` (machine-readable). Show counts and +total bytes. Make NO changes. Present the plan and STOP. + +### 4. CONFIRM +Wait for explicit user approval. Honor partial approvals and edits ("just do +Finance and Screenshots"). Do not proceed on silence. + +### 5. EXECUTE +For each approved move: recompute the destination, apply naming +(`YYYY-MM-DD_slug`, kebab-case, max 80 chars, `on_conflict` policy), check the +file isn't open/locked (skip + log if busy), and ensure disk space. **APPEND the +move to `undo-.json` BEFORE committing it.** Cross-volume moves are +copy → verify hash → remove source. Move whole git repos intact. + +### 6. REPORT +Summarize: moved, renamed, quarantined, skipped, deferred, errors. State the undo +command. List `_Review` contents with *suggested* (not executed) deletions. Never +claim a file was deleted — only moved or quarantined. + +## Configuration & taxonomy + +The default hierarchy, full classification rubric, naming rules, and the +user-overridable config schema are in +[reference/taxonomy.md](reference/taxonomy.md). A ready-to-copy config template is +[reference/organize.config.example.yaml](reference/organize.config.example.yaml). + +## Reference implementation + +[scripts/organize.py](scripts/organize.py) implements this exact pipeline +(stdlib + pyyaml only): +- `python organize.py scan` → dry-run, writes `PLAN.md` + `plan.json` +- `python organize.py execute --plan plan.json` → applies, writes `undo-.json` +- `python organize.py undo --manifest undo-.json` → reverses everything + +Use it to do the mechanical work deterministically, or follow the pipeline +manually with the filesystem tools — either way the safety invariants below hold. + +## Rules + +- Destructive actions are FORBIDDEN. When uncertain, quarantine to `_Review` and + ask. +- The dry-run plan and explicit confirmation are mandatory — never move a file the + user hasn't seen in a plan. +- Every executed move MUST be recorded in the undo manifest before it happens. +- Never delete: junk, installers, and duplicates are quarantined, deletion is only + ever a suggestion the user runs manually. +- Never touch protected paths (`~/Library`, `/System`, `/Applications`, `.app` / + `.photoslibrary` bundles, git repos, `node_modules`, dotfiles, symlinks, + `*.icloud`). +- The directory hierarchy is OPTIONAL and user-owned: config rules override + defaults, first match wins. +- Prefer organizing into a copy-tree (`~/Documents/Organized`) over restructuring + a folder in place, unless the user's config says otherwise. + +## Output + +After the dry-run, present the plan summary and ask for confirmation. After +execution, emit: + +```json +{ + "moved": 0, + "renamed": 0, + "quarantined": 0, + "skipped": 0, + "deferred": 0, + "errors": 0, + "undo_manifest": "undo-2026-08-01T12-00-00.json", + "review_suggested_deletions": ["_Review/junk/.DS_Store", "_Review/duplicates/..."] +} +``` diff --git a/skills/desktop-organizer/reference/organize.config.example.yaml b/skills/desktop-organizer/reference/organize.config.example.yaml new file mode 100644 index 0000000..0b13713 --- /dev/null +++ b/skills/desktop-organizer/reference/organize.config.example.yaml @@ -0,0 +1,63 @@ +# organize.config.yaml — user-configurable taxonomy for the desktop-organizer skill. +# +# Drop this file in your scan root (e.g. ~/Downloads) or at +# ~/.config/cowork-organize/organize.config.yaml. If it is absent, the built-in +# defaults apply. User rules are merged OVER defaults; the first matching rule +# wins. The directory hierarchy is entirely optional — delete `folders`/`rules` +# to fall back to defaults. + +version: 1 + +# Destination tree. The organizer moves files here rather than restructuring +# source folders in place. +root: ~/Documents/Organized + +# Folders to drain into the tree (staging areas). +scan: + - ~/Downloads + - ~/Desktop + +naming: + date_format: "%Y-%m-%d" # strftime; prefixed to slug when a date is meaningful + case: kebab # kebab | snake | title + max_len: 80 + +# Globs never touched, in addition to the always-protected system paths. +ignore: + - "**/.git/**" + - "**/*.app/**" + - "~/Library/**" + - "**/node_modules/**" + - "**/*.icloud" # iCloud placeholders (not downloaded) + - "**/.DS_Store" # matched here only to keep it out of scans; still quarantined if found + +# Your own taxonomy. `{project}`, `{year}`, `{month}` are filled from +# classification. These override the default tree. +folders: + Work: { path: "Work/{project}/{year}" } + Finance: { path: "Personal/finance/{year}" } + Shots: { path: "Media/Screenshots/{year}-{month}" } + Media: { path: "Media/Photos/{year}/{year}-{month}" } + Code: { path: "Code/{project}" } + +# First match wins. `to` references a key in `folders`, or a special target. +rules: + - match: { name_regex: "(?i)invoice|receipt|statement" } + to: Finance + - match: { ext: [png], name_regex: "(?i)screenshot|cleanshot|screen shot" } + to: Shots + - match: { ext: [jpg, jpeg, heic, png] } + to: Media + - match: { ext: [py, js, ts, go, rs, java] } + to: Code + - match: { ext: [dmg, pkg] } + to: quarantine # → Installers, flagged "suggest delete" + - match: { mime: "application/zip" } + to: "Archives/{year}" + +defaults: + unmatched: "_Review/unsorted" + quarantine: "_Review" + on_conflict: version # version | skip | hash-suffix + duplicates: quarantine # exact SHA-256 dupes → _Review/duplicates + never_delete: true # hard invariant; do not set to false diff --git a/skills/desktop-organizer/reference/taxonomy.md b/skills/desktop-organizer/reference/taxonomy.md new file mode 100644 index 0000000..a454737 --- /dev/null +++ b/skills/desktop-organizer/reference/taxonomy.md @@ -0,0 +1,91 @@ +# Desktop Organizer — Taxonomy, Rubric & Naming + +## 1. Default hierarchy (a sensible default, fully overridable) + +The default root is `~/Documents/Organized/` — the organizer never restructures +`~/Documents` in place unless the config says so. Pattern: **Domain → Project / +Category → Year → files**. Downloads and Desktop are *staging areas* drained into +this tree. + +``` +~/Documents/Organized/ +├── Work/ +│ ├── acme-corp/2026/2026-07-14_q3-budget.xlsx +│ └── _Unsorted/ # domain known, project unknown +├── Personal/ +│ ├── finance/2026/2026-06-01_bank-statement.pdf +│ ├── health/2026/ +│ └── travel/2026/ +├── Media/ +│ ├── Photos/2026/2026-07/ +│ ├── Screenshots/2026-07/ +│ └── Video/ +├── Code/ +│ └── / # left intact if it's a git repo +├── Reference/ # ebooks, manuals, papers +├── Archives/ # zips/tars +├── Installers/ # .dmg/.pkg → quarantine, suggest delete +└── _Review/ # ambiguous / duplicates / junk — NEVER auto-deleted + ├── unsorted/ + ├── duplicates/ + └── junk/ +``` + +Anything unclassifiable lands in `_Review/unsorted/` with a reason note. + +## 2. Classification rubric + +Decide the destination by combining signals, highest-confidence first: + +| Signal | Use | +|--------|-----| +| **Extension + MIME** | Primary type bucket. `.pdf`→docs, `.png/.jpg/.heic`→images, `.mov/.mp4`→video, `.zip/.tar/.gz`→archives, `.dmg/.pkg/.app`→installers, `.py/.js/.ts`→code, `.docx/.xlsx/.pptx`→documents. Read magic bytes — do not trust the extension. | +| **Content sniff** | PDF text scan → "Invoice/Statement/Boarding pass" → finance/travel. Light image inspection → screenshot vs. photo. | +| **Filename tokens** | Regex for `invoice`, `receipt`, `resume`, `screenshot`, project codes, client names → domain/project. | +| **Dates** | EXIF DateTaken (photos) > filesystem mtime > date parsed from name. Drives the `/YYYY/` and `YYYY-MM-DD` buckets. | +| **Source (optional)** | Download origin (`kMDItemWhereFroms` xattr), AirDrop/Slack tags → routing hints. | + +**Screenshots:** name matches `Screenshot* / CleanShot* / Screen Shot*`, or +dir=Desktop + PNG + no EXIF → `Media/Screenshots/YYYY-MM/`. + +**Desktop clutter:** loose files >7 days old on the Desktop are candidates; +app aliases / `.app` / in-use folders are skipped. + +**Junk:** `.DS_Store`, `*.crdownload`, `*.part`, zero-byte files, `~$*` lock +files → `_Review/junk/` (never deleted). + +**Duplicates:** SHA-256 of contents. Exact match → keep the copy in the best +location, move the others to `_Review/duplicates/` with a pointer to the kept +copy. Near-dupes (same name, differing size) are flagged, never auto-removed. + +## 3. Naming conventions + +- **Date prefix when meaningful:** `YYYY-MM-DD_.` (e.g. + `2026-07-14_q3-budget.xlsx`). Photos use their EXIF date. +- **Slug:** lowercase; spaces/underscores → `-`; strip diacritics and unsafe + chars `/\:*?"<>|`; collapse repeats; trim to `max_len`; keep the (lowercased) + original extension. +- **Never destroy the original name** — it is stored in the undo manifest so every + rename is reversible. +- **Collisions (`on_conflict`):** `version` → append `-v2`, `-v3`; `hash-suffix` → + append the first 8 of the content hash; `skip` → leave in place and log. The + collision check is case-insensitive (APFS is case-insensitive by default). + +## 4. Protected / skipped paths (always) + +`~/Library`, `/System`, `/Applications`, `.app` bundles, `.photoslibrary`, git +repositories (moved whole, never split), `node_modules`, dotfiles/hidden files, +symlinks, and `*.icloud` placeholders (not yet downloaded). Locked/open files are +detected (`lsof`/flock) and deferred with a note. + +## 5. Safety invariants (restated — non-negotiable) + +1. Read-only scan; mandatory dry-run plan (`PLAN.md` + `plan.json`) before any + move. +2. Explicit confirmation required; partial approval supported. +3. Every move appended to `undo-.json` *before* it commits; an `undo` + run replays it in reverse. +4. Never delete — quarantine to `_Review/` (or macOS Trash if configured); + deletion is only ever a suggestion. +5. Cross-volume moves = copy → verify hash → remove source. Verify free disk + space first. diff --git a/skills/desktop-organizer/scripts/organize.py b/skills/desktop-organizer/scripts/organize.py new file mode 100644 index 0000000..c443b4c --- /dev/null +++ b/skills/desktop-organizer/scripts/organize.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +"""Reference implementation of the desktop-organizer safety pipeline. + +Deliberately dependency-light (stdlib + optional pyyaml) so it runs anywhere the +rest of this project runs. It enforces the skill's non-negotiable invariants: + + scan -> read-only; writes PLAN.md + plan.json, moves nothing + execute-> applies an approved plan.json; records undo-.json BEFORE each move + undo -> reverses a run from its undo manifest + +It NEVER deletes: junk/installers/duplicates are quarantined under _Review/. + +Usage: + python organize.py scan [--config organize.config.yaml] [--root DEST] + python organize.py execute --plan plan.json + python organize.py undo --manifest undo-.json +""" +from __future__ import annotations + +import argparse +import fnmatch +import hashlib +import json +import os +import re +import shutil +import sys +from datetime import datetime, timezone +from pathlib import Path + +try: + import yaml # pyyaml is already a project dependency +except ImportError: # pragma: no cover - graceful fallback + yaml = None + +# --- Always-protected paths, independent of user config ----------------------- +PROTECTED_PARTS = {"Library", "System", "Applications", "node_modules"} +PROTECTED_SUFFIXES = (".app", ".photoslibrary") +JUNK_NAMES = {".DS_Store"} +JUNK_GLOBS = ("*.crdownload", "*.part", "~$*") + +DEFAULT_CONFIG = { + "version": 1, + "root": "~/Documents/Organized", + "scan": ["~/Downloads", "~/Desktop"], + "naming": {"date_format": "%Y-%m-%d", "case": "kebab", "max_len": 80}, + "ignore": ["**/.git/**", "**/*.app/**", "~/Library/**", "**/node_modules/**", "**/*.icloud"], + "folders": {}, + "rules": [], + "defaults": { + "unmatched": "_Review/unsorted", + "quarantine": "_Review", + "on_conflict": "version", + "duplicates": "quarantine", + "never_delete": True, + }, +} + +EXT_BUCKETS = { + "pdf": "Reference", "docx": "Work", "xlsx": "Work", "pptx": "Work", + "png": "Media", "jpg": "Media", "jpeg": "Media", "heic": "Media", + "mov": "Media", "mp4": "Media", + "zip": "Archives", "tar": "Archives", "gz": "Archives", + "dmg": "quarantine", "pkg": "quarantine", + "py": "Code", "js": "Code", "ts": "Code", "go": "Code", "rs": "Code", +} + + +def load_config(path: str | None) -> dict: + cfg = json.loads(json.dumps(DEFAULT_CONFIG)) # deep copy + if path and Path(path).expanduser().exists(): + if yaml is None: + print("warning: pyyaml not installed; using defaults", file=sys.stderr) + else: + user = yaml.safe_load(Path(path).expanduser().read_text()) or {} + cfg.update({k: v for k, v in user.items() if v is not None}) + return cfg + + +def sha256(p: Path, limit: int = 64 * 1024 * 1024) -> str: + h = hashlib.sha256() + with p.open("rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + if f.tell() > limit: + break + return "sha256:" + h.hexdigest() + + +def is_protected(p: Path) -> bool: + if any(part in PROTECTED_PARTS for part in p.parts): + return True + if any(str(p).find(sfx + os.sep) != -1 or p.name.endswith(sfx) for sfx in PROTECTED_SUFFIXES): + return True + if p.is_symlink() or p.name.startswith("."): + return True + if p.name.endswith(".icloud"): + return True + # git repos are treated as one unit; skip files inside them + if any((anc / ".git").exists() for anc in list(p.parents)[:6]): + return True + return False + + +def matches_ignore(p: Path, globs: list[str]) -> bool: + s = str(p) + return any(fnmatch.fnmatch(s, os.path.expanduser(g)) for g in globs) + + +def is_junk(p: Path) -> bool: + if p.name in JUNK_NAMES or any(fnmatch.fnmatch(p.name, g) for g in JUNK_GLOBS): + return True + try: + return p.stat().st_size == 0 + except OSError: + return False + + +def slugify(name: str, case: str, max_len: int) -> str: + stem, ext = os.path.splitext(name) + stem = re.sub(r"[/\\:*?\"<>|]", "", stem) + stem = re.sub(r"[\s_]+", "-", stem.strip()) + stem = re.sub(r"-{2,}", "-", stem).strip("-").lower() + if case == "snake": + stem = stem.replace("-", "_") + return (stem[:max_len] or "file") + ext.lower() + + +def file_date(p: Path, fmt: str) -> str: + ts = p.stat().st_mtime + return datetime.fromtimestamp(ts, tz=timezone.utc).strftime(fmt) + + +def classify(p: Path, cfg: dict) -> tuple[str, str]: + """Return (destination_subpath, reason).""" + name = p.name + ext = p.suffix.lower().lstrip(".") + dfmt = cfg["naming"]["date_format"] + year = file_date(p, "%Y") + month = file_date(p, "%m") + + # user rules first (first match wins) + for rule in cfg.get("rules", []): + m = rule.get("match", {}) + if "ext" in m and ext not in [e.lower() for e in m["ext"]]: + continue + if "name_regex" in m and not re.search(m["name_regex"], name): + continue + target = rule.get("to") + folder = cfg.get("folders", {}).get(target, {"path": target}) + path_tmpl = folder["path"] if isinstance(folder, dict) else str(folder) + if target == "quarantine": + return f"{cfg['defaults']['quarantine']}/installers", "rule:quarantine" + return path_tmpl.format(project="_Unsorted", year=year, month=month), f"rule:{target}" + + # screenshot heuristic + if ext == "png" and re.search(r"(?i)screenshot|cleanshot|screen shot", name): + return f"Media/Screenshots/{year}-{month}", "screenshot" + + # extension bucket + bucket = EXT_BUCKETS.get(ext) + if bucket == "quarantine": + return f"{cfg['defaults']['quarantine']}/installers", "installer" + if bucket in ("Media",): + return f"Media/Photos/{year}/{year}-{month}", "ext:media" + if bucket: + return f"{bucket}/{year}", f"ext:{ext}" + + return cfg["defaults"]["unmatched"], "unmatched" + + +def dest_path(root: Path, sub: str, p: Path, cfg: dict) -> Path: + d = file_date(p, cfg["naming"]["date_format"]) + slug = slugify(p.name, cfg["naming"]["case"], cfg["naming"]["max_len"]) + if not slug.lower().startswith(d): + slug = f"{d}_{slug}" + return root / sub / slug + + +def resolve_conflict(dest: Path, policy: str, content_hash: str) -> Path | None: + if not dest.exists(): + return dest + if policy == "skip": + return None + stem, ext = os.path.splitext(dest.name) + if policy == "hash-suffix": + return dest.with_name(f"{stem}-{content_hash.split(':')[1][:8]}{ext}") + n = 2 + while True: + cand = dest.with_name(f"{stem}-v{n}{ext}") + if not cand.exists(): + return cand + n += 1 + + +def cmd_scan(args) -> None: + cfg = load_config(args.config) + root = Path(args.root or cfg["root"]).expanduser() + seen_hashes: dict[str, str] = {} + moves, quarantines, junk, skipped = [], [], [], [] + + for scan_dir in cfg["scan"]: + base = Path(scan_dir).expanduser() + if not base.exists(): + continue + for p in base.rglob("*"): + if not p.is_file(): + continue + if is_protected(p) or matches_ignore(p, cfg["ignore"]): + skipped.append({"path": str(p), "reason": "protected/ignored"}) + continue + if is_junk(p): + junk.append({"src": str(p), "dst": str(root / "_Review/junk" / p.name), + "reason": "junk", "action": "quarantine"}) + continue + ch = sha256(p) + if ch in seen_hashes: + quarantines.append({"src": str(p), + "dst": str(root / "_Review/duplicates" / p.name), + "reason": f"duplicate of {seen_hashes[ch]}", + "action": "quarantine", "hash": ch}) + continue + seen_hashes[ch] = str(p) + sub, reason = classify(p, cfg) + dest = dest_path(root, sub, p, cfg) + dest = resolve_conflict(dest, cfg["defaults"]["on_conflict"], ch) + if dest is None: + skipped.append({"path": str(p), "reason": "conflict:skip"}) + continue + moves.append({"src": str(p), "dst": str(dest), "reason": reason, + "action": "move", "hash": ch}) + + plan = { + "generated": file_date(Path(cfg["scan"][0]).expanduser(), "%Y-%m-%dT%H-%M-%S") + if Path(cfg["scan"][0]).expanduser().exists() else "unknown", + "root": str(root), + "moves": moves, "quarantines": quarantines, "junk": junk, "skipped": skipped, + } + Path("plan.json").write_text(json.dumps(plan, indent=2)) + _write_plan_md(plan) + print(f"DRY RUN — nothing moved.\n moves: {len(moves)}\n" + f" duplicates: {len(quarantines)}\n junk: {len(junk)}\n" + f" skipped: {len(skipped)}\n\nReview PLAN.md, then:\n" + f" python organize.py execute --plan plan.json") + + +def _write_plan_md(plan: dict) -> None: + lines = [f"# Organization Plan (DRY RUN)\n", f"Destination root: `{plan['root']}`\n"] + for title, key in [("Moves", "moves"), ("Duplicates → _Review", "quarantines"), + ("Junk → _Review", "junk"), ("Skipped (protected/ignored)", "skipped")]: + items = plan.get(key, []) + lines.append(f"\n## {title} ({len(items)})\n") + for it in items[:1000]: + if "src" in it: + lines.append(f"- `{it['src']}` → `{it['dst']}` _({it.get('reason','')})_") + else: + lines.append(f"- `{it['path']}` _({it.get('reason','')})_") + lines.append("\n> Nothing has been moved. Deletion is never performed — " + "junk and duplicates are quarantined under `_Review/`.\n") + Path("PLAN.md").write_text("\n".join(lines)) + + +def _do_move(src: Path, dst: Path) -> None: + dst.parent.mkdir(parents=True, exist_ok=True) + if src.stat().st_dev != dst.parent.stat().st_dev: # cross-volume: copy+verify+remove + shutil.copy2(src, dst) + if sha256(src) != sha256(dst): + dst.unlink(missing_ok=True) + raise IOError(f"hash mismatch copying {src}") + src.unlink() + else: + shutil.move(str(src), str(dst)) + + +def cmd_execute(args) -> None: + plan = json.loads(Path(args.plan).read_text()) + ts = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H-%M-%S") if _now_ok() else "run" + undo_path = Path(f"undo-{ts}.json") + undo: list[dict] = [] + stats = {"moved": 0, "quarantined": 0, "deferred": 0, "errors": 0} + + for group, kind in [("moves", "moved"), ("quarantines", "quarantined"), ("junk", "quarantined")]: + for it in plan.get(group, []): + src, dst = Path(it["src"]), Path(it["dst"]) + if not src.exists(): + stats["deferred"] += 1 + continue + entry = {"src": str(src), "dst": str(dst), "orig_name": src.name} + undo.append(entry) + undo_path.write_text(json.dumps(undo, indent=2)) # record BEFORE the move + try: + _do_move(src, dst) + stats[kind] += 1 + except Exception as e: # noqa: BLE001 + stats["errors"] += 1 + entry["error"] = str(e) + undo.pop() + undo_path.write_text(json.dumps(undo, indent=2)) + + print(json.dumps({**stats, "undo_manifest": str(undo_path)}, indent=2)) + print(f"\nTo reverse: python organize.py undo --manifest {undo_path}") + + +def cmd_undo(args) -> None: + entries = json.loads(Path(args.manifest).read_text()) + restored = 0 + for entry in reversed(entries): + dst, src = Path(entry["dst"]), Path(entry["src"]) + if dst.exists(): + src.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(dst), str(src)) + restored += 1 + print(f"Reversed {restored} move(s) from {args.manifest}.") + + +def _now_ok() -> bool: + try: + datetime.now() + return True + except Exception: # pragma: no cover + return False + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + sub = ap.add_subparsers(dest="cmd", required=True) + s = sub.add_parser("scan", help="dry-run: write PLAN.md + plan.json") + s.add_argument("--config") + s.add_argument("--root") + s.set_defaults(func=cmd_scan) + e = sub.add_parser("execute", help="apply an approved plan.json") + e.add_argument("--plan", required=True) + e.set_defaults(func=cmd_execute) + u = sub.add_parser("undo", help="reverse a run from its undo manifest") + u.add_argument("--manifest", required=True) + u.set_defaults(func=cmd_undo) + args = ap.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/skills/unified-search/SKILL.md b/skills/unified-search/SKILL.md new file mode 100644 index 0000000..5198fb5 --- /dev/null +++ b/skills/unified-search/SKILL.md @@ -0,0 +1,118 @@ +--- +name: unified-search +description: | + Search and retrieve across ALL the user's connected sources at once — email + (Outlook/Gmail), chat (Slack, Teams), cloud docs (SharePoint, Drive, OneDrive), + local files, and calendar — then return ranked, deduplicated, cited results. + New sources plug in without changing the engine. + + Use this skill when: + - The user asks to "find", "search", "look up", "where is", or "pull up" + something that could live in more than one place. + - The user references content vaguely ("that thread with Jane about the budget", + "the deck from last week") and you must locate the source of truth. + - The user wants a consolidated answer drawn from several tools/systems. + - The user wants to build or extend a cross-source index/connector. + + + Context: The user can't remember where a decision was recorded. + user: "where did we land on the launch date? it was either slack or email" + assistant: I'll use the unified-search skill to query Slack and email together, + dedupe, and cite the source of the decision. + + + + Context: The user wants a single view across systems. + user: "find everything about the Q3 budget from the last month" + assistant: I'll run unified-search across email, chat, and cloud docs, merge and + rank the results, and give you cited links back to each source. + +--- + +# Unified Cross-Source Search + +You are a retrieval agent. Your job is to answer "where is / find / what did we +decide" questions by querying **every relevant connected source**, merging the +results into one ranked, deduplicated list, and **citing each result with a deep +link** back to its origin. You retrieve pointers — you re-fetch live content +before quoting so you never serve stale data. + +## What to do + +Follow this procedure on every search request: + +### 1. Plan the query +Parse the user's request into: +- **terms** — keywords, names, IDs, quoted phrases. +- **filters** — sources, item types, date range, people involved. +- **intent** — a specific item ("invoice #4471" → keyword) vs. a fuzzy recall + ("that thread about the budget cut" → semantic). Pick `mode`: `keyword`, + `semantic`, or `hybrid` (default `hybrid`). + +### 2. Fan out to sources (read-only) +Query only the sources that match the filters, in parallel where possible. Map +each source to the tools available in this environment — see +[reference/connectors.md](reference/connectors.md) for the exact tool per source. +If a source isn't connected, note it in the results rather than failing silently. + +### 3. Normalize every hit +Map each raw result into the `loom.doc.v1` record shape (id, source, type, title, +author, participants, timestamps, `location` deep-link, snippet, content_hash, +tags, acl). Schema: [reference/doc-schema.json](reference/doc-schema.json). + +### 4. Merge, dedup, rank +- Collapse duplicates by `content_hash` (same file in Drive + local + as an email + attachment → one record with an `also_in` list of the other locations). +- Rank with hybrid fusion (Reciprocal Rank Fusion of BM25 + semantic; k≈60). +- Keep the strongest ~20 unless the user asked for more. + +### 5. Answer with citations +Return a short direct answer first, then the ranked results, each with: title, +source, timestamp, author, a one-line snippet, and a **clickable deep link**. If +you assert a fact, cite the specific item it came from. Before quoting a result +verbatim, re-fetch its live content. + +## Architecture & extension + +The full connector contract, normalized-record schema, hybrid-ranking design, +storage/sync model, privacy rules, and phased rollout are in +[reference/architecture.md](reference/architecture.md). Read it when the task is +to *build* or *extend* the index (add a source, design storage) rather than just +run a query. + +## Rules + +- DO stay read-only. This skill never moves, edits, or deletes anything. +- DO respect source permissions — only return what the connected identity can see. +- DO cite every result with a deep link; an uncited claim is a bug. +- DO note which sources were queried and which were unavailable. +- DO NOT index or surface secrets (`.env`, `id_rsa`, `*.pem`, credential files) + or content from muted/excluded channels. +- DO NOT quote cached snippets as authoritative — re-fetch before quoting. +- DO NOT invent a source link; if you can't produce a real deep link, say so. + +## Output + +Lead with a one-paragraph answer, then a results list. When a machine-readable +form is requested, emit: + +```json +{ + "answer": "One-paragraph direct answer with the key fact.", + "results": [ + { + "id": "sha256:...", + "type": "email|chat_message|file|doc|calendar_event", + "title": "Re: Q3 budget", + "source": "outlook", + "author": "Jane ", + "timestamp": "2026-07-30T14:00:00Z", + "snippet": "...cut the travel line by 15%...", + "score": 0.87, + "citation": { "url": "https://...", "also_in": ["gdrive://1AbC"] } + } + ], + "sources_queried": ["outlook", "slack", "sharepoint"], + "sources_unavailable": ["gmail"] +} +``` diff --git a/skills/unified-search/reference/architecture.md b/skills/unified-search/reference/architecture.md new file mode 100644 index 0000000..56f1daa --- /dev/null +++ b/skills/unified-search/reference/architecture.md @@ -0,0 +1,133 @@ +# Unified Search — Architecture Reference + +Read this when *building or extending* the index (adding a source, designing +storage, changing ranking) — not for a routine query. + +## 1. Source-connector abstraction + +Every source implements one capability-negotiated contract. The engine knows +nothing source-specific beyond it. Connectors register via a `connector.yaml` +manifest (`id`, auth type, required scopes, capability flags). + +```typescript +interface SourceConnector { + id: string; // "outlook", "slack", "sharepoint", "local-fs" + capabilities: { + delta: boolean; // supports incremental cursor + contentFetch: boolean; + acl: boolean; + }; + + authenticate(ctx: AuthContext): Promise; // scoped token/handle + + // Enumerate items changed since cursor (full scan if cursor null) + list(cursor: string | null, opts: { pageSize: number }) + : AsyncIterator<{ items: ItemRef[]; nextCursor: string; done: boolean }>; + + fetchMetadata(ref: ItemRef): Promise; + fetchContent(ref: ItemRef): Promise<{ mime: string; bytes?: Uint8Array; text?: string }>; + fetchACL(ref: ItemRef): Promise; // optional if !capabilities.acl +} + +type ItemRef = { sourceId: string; nativeId: string; etag: string; + changeType: "upsert" | "delete" }; +``` + +Cursors are opaque per-connector strings: Outlook/Graph `deltaLink`, Gmail +`historyId`, Drive `pageToken`, Slack `oldest` ts, local-fs FSEvents id / mtime +watermark. When a capability is missing, degrade: no `delta` → poll-on-query with +short TTL; no `acl` → treat as private-to-user. + +## 2. Normalized record — `loom.doc.v1` + +Everything maps to one record; source-specific fields survive in `raw`. Formal +schema: `doc-schema.json`. + +Key fields: `id` = `sha256(source_id + native_id)`; `content_hash` = +`sha256(normalized_text)` (dedup key); `location.url` = deep link for citation; +`acl.principals` = who may see it. See the schema file for the complete shape and +an example record. + +## 3. Indexing strategy + +**Stored locally:** +- Normalized records → SQLite (`documents` table + FTS5 for BM25). +- Extracted plain text → content-addressed blob store (never store original + binaries unless the source is already local). +- Embeddings → local vector index (sqlite-vec / LanceDB), one vector per + ~512-token chunk. + +**Incremental sync.** Each `(connector, account)` keeps a `sync_state` row +`{cursor, last_full_scan, watermark}`. A scheduler polls per source (webhook/push +where available — Gmail push, Graph subscriptions). `list()` yields +upserts/deletes; only changed etags trigger `fetchContent`. Tombstones propagate +deletes to all three stores. + +**Dedup across sources.** Two levels: +1. *Exact* — `content_hash` collapses the same content (file in Drive + local + + email attachment) into one canonical record with `aliases[]` / `also_in`. +2. *Near* — MinHash/SimHash buckets catch forwarded emails and doc revisions; + keep newest as canonical, link the rest. + +**Large binaries.** Never embed raw. Pipeline: type-sniff → extract text (Tika / +pdfplumber / OCR for images) → chunk → discard bytes. Files over a size cap are +metadata-only with on-demand extraction at query time. + +**Hybrid retrieval (recommended).** Run both and fuse: +- BM25/FTS5 for exact term, name, ID matching (fast, cheap). +- Dense embeddings for semantic recall. +- **Reciprocal Rank Fusion**: `score = Σ 1/(k + rank)`, k≈60; optional + cross-encoder rerank on top 50. Embed lazily — index metadata + BM25 + immediately, backfill embeddings asynchronously. + +## 4. Query interface + +```json +// Request +{ + "q": "budget cut discussion with Jane", + "filters": { + "source": ["outlook", "slack", "sharepoint"], + "type": ["email", "chat_message"], + "date": { "gte": "2026-07-01", "lte": "2026-08-01" }, + "person": ["jane@x.com"] + }, + "mode": "hybrid", + "limit": 20 +} +``` + +Structured filters constrain the candidate set (SQL) first; hybrid ranking then +orders it. Response returns deduped canonical docs, each with a `citation.url` +deep link and `citation.also_in` for cross-source copies, plus `facets` for +drill-down (see SKILL.md "Output"). + +## 5. Privacy / security + +- **Local-first:** index, blobs, vectors on-device, encrypted at rest via a + keychain-derived key. Prefer a local embedding model so content never leaves + the machine. +- **Respect ACLs:** store effective `acl.principals`; every query filters to what + the connected identity can see; re-validate shared items on access. +- **Secrets:** OAuth/refresh tokens in the OS keychain, never in SQLite or logs; + connectors receive short-lived scoped handles. +- **Never indexed:** password vaults, `.env`, `id_rsa`, `*.pem`, user + exclude-globs, muted/private channels, un-opted-in sources. A redaction pass + strips detected secrets from extracted text before embedding. +- **User controls:** per-source enable, exclusion rules, "forget this item", + full local purge. + +## 6. Phased rollout + +- **Phase 0 — MVP:** wrap the tools already present in this environment (see + `connectors.md`) as read-only connectors. Normalized schema + SQLite/FTS5 (BM25 + only) + unified query + citations; sync = poll-on-query with short TTL. Zero + local-storage risk; proves the abstraction end to end. +- **Phase 1:** persistent local SQLite/blob/vector store, incremental cursors, + hybrid RRF ranking with local embeddings, cross-source dedup. +- **Phase 2:** macOS FSEvents connector (text extraction, OCR); native Gmail + + Google Drive delta connectors for real push sync. +- **Phase 3 — Extensibility GA:** publish the `SourceConnector` SDK + + `connector.yaml` manifest so users add sources (Notion, Linear, …) as drop-in + plugins; connector registry with capability negotiation and per-connector scope + prompts. diff --git a/skills/unified-search/reference/connectors.md b/skills/unified-search/reference/connectors.md new file mode 100644 index 0000000..97f7c6a --- /dev/null +++ b/skills/unified-search/reference/connectors.md @@ -0,0 +1,53 @@ +# Connector Map — source → tools in this environment + +How each normalized source maps to concrete tools. Tool names follow the +`mcp____` convention `client.py` already uses. When a source has no +connected tool, report it as `sources_unavailable` rather than failing the query. + +## Available today (Phase 0 — read-only, no new auth) + +| Source | Normalized `type` | Search tool(s) | Read/context tool(s) | +|--------|-------------------|----------------|----------------------| +| Outlook / Exchange email | `email` | `mcp__Microsoft_365__outlook_email_search` | `mcp__Microsoft_365__read_resource` | +| Microsoft Teams | `chat_message` | `mcp__Microsoft_365__chat_message_search` | `mcp__Microsoft_365__teams_list_chats`, `read_resource` | +| SharePoint / OneDrive | `file` / `doc` | `mcp__Microsoft_365__sharepoint_search`, `sharepoint_folder_search` | `read_resource` | +| Outlook calendar | `calendar_event` | `mcp__Microsoft_365__outlook_calendar_search` | `read_resource` | +| Slack / other chat (Macro) | `chat_message` | `mcp__Macro__ContentSearch`, `mcp__Macro__NameSearch` | `mcp__Macro__ReadChannelMessages`, `ReadChannelThread`, `ReadChat`, `ReadContent` | +| Google Calendar | `calendar_event` | `mcp__Google_Calendar__search_events` | `mcp__Google_Calendar__list_events`, `get_event` | +| Local files (sandbox) | `file` | `Grep`, `Glob` | `Read` | +| GitHub (code/issues/PRs) | `doc` / `file` | `mcp__github__search_code`, `search_issues`, `search_pull_requests` | `mcp__github__get_file_contents`, `pull_request_read` | + +## Cowork-native connectors (promote in Phase 1+) + +Claude Cowork exposes these as first-class OAuth connectors; wrap them the same +way and, where the API offers a delta cursor, upgrade from poll-on-query to real +incremental sync. + +| Source | Read | Write | Notes | +|--------|------|-------|-------| +| Gmail | full | drafts only | Google Workspace connector | +| Google Drive | full | full | delta via `pageToken` | +| Google Calendar | full | full | | +| Slack | channels/threads | post / draft | | +| Microsoft 365 (Outlook) | full | read-only by default | admin can enable write | +| Microsoft 365 (Teams) | full | read-only | always read-only | +| SharePoint / OneDrive | full | admin-gated | | + +## Adding a new source + +1. Write a `connector.yaml` (`id`, auth type, scopes, capability flags: + `delta` / `contentFetch` / `acl`). +2. Implement the `SourceConnector` contract (see `architecture.md` §1): `list`, + `fetchMetadata`, `fetchContent`, `fetchACL`. +3. Provide a `normalize()` that maps the source's raw items into `loom.doc.v1` + (`doc-schema.json`), including a real `location.url` deep link and a + `content_hash`. +4. Register it; the engine picks it up via capability negotiation — no core + changes. + +## Query-time fan-out rules + +- Only call tools for sources that pass the request's `filters.source`. +- Run source queries in parallel where the tools allow. +- Cap per-source results before merging (e.g. top 25/source) to keep fusion cheap. +- Always capture a deep link at fetch time — it is required for citation. diff --git a/skills/unified-search/reference/doc-schema.json b/skills/unified-search/reference/doc-schema.json new file mode 100644 index 0000000..049a246 --- /dev/null +++ b/skills/unified-search/reference/doc-schema.json @@ -0,0 +1,104 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "loom.doc.v1", + "title": "Unified normalized document record", + "description": "Every item from every source maps into this shape. Source-specific fields are preserved under `raw`.", + "type": "object", + "required": ["id", "source", "type", "timestamps", "location", "content_hash"], + "properties": { + "id": { + "type": "string", + "description": "sha256(source.id + native_id) — stable global identifier." + }, + "source": { + "type": "object", + "required": ["id"], + "properties": { + "id": { "type": "string", "description": "Connector id, e.g. outlook, slack, sharepoint, local-fs." }, + "account": { "type": "string", "description": "Which connected account this came from." } + } + }, + "type": { + "type": "string", + "enum": ["email", "chat_message", "file", "doc", "calendar_event", "contact"] + }, + "title": { "type": "string" }, + "author": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "email": { "type": "string" }, + "id": { "type": "string" } + } + }, + "participants": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { "type": "string" }, + "email": { "type": "string" }, + "role": { "type": "string", "enum": ["to", "cc", "bcc", "attendee", "member", "author"] } + } + } + }, + "timestamps": { + "type": "object", + "properties": { + "created": { "type": "string", "format": "date-time" }, + "modified": { "type": "string", "format": "date-time" }, + "event_start": { "type": ["string", "null"], "format": "date-time" }, + "event_end": { "type": ["string", "null"], "format": "date-time" } + } + }, + "location": { + "type": "object", + "description": "Where the item lives; url is the citation deep link.", + "properties": { + "url": { "type": "string", "description": "Deep link back to the source item (used for citation)." }, + "path": { "type": "string", "description": "Local filesystem path, if applicable." }, + "container": { "type": "string", "description": "Channel, folder, label, or mailbox." } + } + }, + "snippet": { "type": "string", "description": "Short preview (~280 chars) for display." }, + "body_ref": { "type": "string", "description": "Pointer to extracted full text in the blob store, e.g. blobstore://sha256..." }, + "content_hash": { "type": "string", "description": "sha256(normalized_text) — cross-source dedup key." }, + "aliases": { + "type": "array", + "items": { "type": "string" }, + "description": "Other locations of the same content (also_in)." + }, + "tags": { "type": "array", "items": { "type": "string" } }, + "acl": { + "type": "object", + "properties": { + "visibility": { "type": "string", "enum": ["private", "shared", "public"] }, + "principals": { "type": "array", "items": { "type": "string" } } + } + }, + "lang": { "type": "string" }, + "raw": { "type": "object", "description": "Verbatim source-specific fields." } + }, + "examples": [ + { + "id": "sha256:ab12...", + "source": { "id": "outlook", "account": "user@corp.com" }, + "type": "email", + "title": "Re: Q3 budget review", + "author": { "name": "Jane", "email": "jane@x.com" }, + "participants": [{ "name": "Bob", "email": "bob@x.com", "role": "to" }], + "timestamps": { "created": "2026-07-30T14:00:00Z", "modified": "2026-07-31T09:12:00Z" }, + "location": { + "url": "https://outlook.office.com/mail/id/AAMk...", + "container": "Inbox/Finance" + }, + "snippet": "First 280 chars of the message for preview...", + "body_ref": "blobstore://sha256:cd34...", + "content_hash": "sha256:cd34...", + "aliases": ["sharepoint://sites/finance/Shared%20Documents/q3-budget.xlsx"], + "tags": ["finance", "auto:has_attachment"], + "acl": { "visibility": "shared", "principals": ["user@corp.com", "group:finance"] }, + "lang": "en" + } + ] +}