Jeeves Watcher 🎩

Filesystem watcher that keeps a Qdrant vector store in sync with document changes.

Overview

jeeves-watcher monitors a configured set of directories for file changes, extracts text content, generates embeddings, and maintains a synchronized Qdrant vector store for semantic search. It automatically:

• Watches directories for file additions, modifications, and deletions
• Extracts text from various formats (Markdown, PDF, DOCX, HTML, JSON, plain text)
• Chunks large documents for optimal embedding
• Embeds content using configurable providers (Google Gemini, mock for testing)
• Syncs to Qdrant for fast semantic search
• Enriches metadata via rules and API endpoints
• Indexes source documents (v0.20.0+): records that live in another system, retrieved by HTTP callback when the owner notifies the watcher (POST /documents)

Source Documents and the SDK

Besides files, the watcher can index records owned by another application (chat threads, tickets, database rows). The application notifies the watcher with a reference such as app://chat/thread/<id> and a designated inference rule; the watcher calls back to a URL configured under sources, renders the JSON record with the rule, embeds it and keeps its points in sync. The watcher keeps no copy of the record. See the Source Documents guide.

@karmaniverous/jeeves-watcher-sdk is a typed TypeScript client for the application-facing API (search, scan, render, metadata, reindex, issues, document notify/delete). When the API is exposed beyond localhost, protect it with api.apiKey (Authentication).

Architecture

For detailed architecture documentation, see packages/service/guides/architecture.md.

Prerequisites

• Node.js >=24.22.2
• Qdrant must be running and reachable before starting jeeves-watcher. The Qdrant collection is created automatically on first startup — no manual collection creation required.

Quick Start

Installation

npm install -g @karmaniverous/jeeves-watcher

Initialize Configuration

Create a new configuration file in your project:

jeeves-watcher init -o ./config

-o names the output directory; this writes ./config/config.json with sensible defaults (an existing file is left untouched). The template sets embedding.dimensions to 768; with Gemini (gemini-embedding-001 returns 3072-dimensional vectors) change it to 3072 before the first start.

Configure

Edit ./config/config.json to specify:

• Watch paths: Directories to monitor
• Embedding provider: Google Gemini or mock (for testing)
• Qdrant connection: URL and collection name
• Inference rules: Automatic metadata enrichment based on file patterns

Example minimal configuration:

{
  "watch": {
    "paths": ["./docs"],
    "ignored": ["**/node_modules/**", "**/.git/**"]
  },
  "embedding": {
    "provider": "gemini",
    "model": "gemini-embedding-001",
    "apiKey": "${GOOGLE_API_KEY}"
  },
  "vectorStore": {
    "url": "http://localhost:6333",
    "collectionName": "my_docs"
  }
}

Start Watching

jeeves-watcher start -c ./config/config.json

The watcher will:

1. Index all existing files in watched directories
2. Monitor for changes
3. Update Qdrant automatically

CLI Commands

Command	Description
jeeves-watcher start -c <path>	Start the filesystem watcher (foreground)
jeeves-watcher init -o <dir>	Write a default config.json into <dir> (pass -o: without it the command fails)
jeeves-watcher status	Probe service health and version (GET /status)
jeeves-watcher reindex	Trigger a scoped reindex (default rules)
jeeves-watcher rebuild-metadata	Rebuild the enrichment store from Qdrant payloads
jeeves-watcher search <query>	Search the vector store
jeeves-watcher enrich <path>	Enrich document metadata with key-value pairs
jeeves-watcher config validate -c <path>	Validate a config file against the schema
jeeves-watcher service <action>	Manage the watcher as a system service (install, uninstall, start, stop, restart, status)
jeeves-watcher scan	Scan the vector store with filter-only queries
jeeves-watcher config [jsonpath]	Query the effective config via JSONPath
jeeves-watcher issues	Show indexing issues and errors
jeeves-watcher helpers	Show loaded map and template helpers
jeeves-watcher config apply [-f <file>] [--replace]	Merge a config patch (file or stdin) into the running service's config, validate, write and reload

The API commands (search, enrich, scan, walk, reindex, rebuild-metadata, issues, helpers, vcs ..., config, config apply) call the API through the SDK with a 60 s timeout and accept --api-key (default: the JEEVES_WATCHER_API_KEY environment variable); their errors print <code>: <message> (a 401 adds key guidance). See the CLI Reference.

Configuration

Environment Variable Substitution

Config strings support ${VAR_NAME} syntax for environment variable injection:

{
  "embedding": {
    "apiKey": "${GOOGLE_API_KEY}"
  }
}

If GOOGLE_API_KEY is set in the environment, the value is substituted at config load time. Set templates in inference rules use Handlebars {{...}} syntax (e.g. {{frontmatter.title}}), which is distinct from the ${...} environment variable syntax used in config values like embedding.apiKey.

Watch Paths

{
  "watch": {
    "paths": ["./docs", "./notes"],
    "ignored": ["**/node_modules/**", "**/*.tmp"]
  }
}

• paths: Array of glob patterns or directories to watch
• ignored: Array of patterns to exclude
• respectGitignore: (default: true) Skip processing files ignored by .gitignore in git repositories. Nested .gitignore files are respected within their subtree.
• moveDetection: (optional) Correlate unlink+add events as file moves to avoid re-embedding. enabled (default: true), bufferMs (default: 2000) — how long to buffer unlink events before treating as deletes.

Embedding Provider

Google Gemini

{
  "embedding": {
    "provider": "gemini",
    "model": "gemini-embedding-001",
    "apiKey": "${GOOGLE_API_KEY}"
  }
}

Vector Store

{
  "vectorStore": {
    "url": "http://localhost:6333",
    "collectionName": "my_collection"
  }
}

Inference Rules

Automatically enrich metadata based on file patterns using declarative JSON Schemas:

{
  "schemas": {
    "base": {
      "type": "object",
      "properties": {
        "domain": {
          "type": "string",
          "description": "Content domain"
        }
      }
    }
  },
  "inferenceRules": [
    {
      "name": "meeting-classifier",
      "description": "Classify files under meetings directory",
      "match": {
        "properties": {
          "file": {
            "type": "object",
            "properties": {
              "path": { "type": "string", "glob": "**/meetings/**" }
            }
          }
        }
      },
      "schema": [
        "base",
        {
          "properties": {
            "domain": { "set": "meetings" },
            "category": { "type": "string", "set": "notes" }
          }
        }
      ]
    }
  ]
}

Inference rules use schema arrays that reference global named schemas. Type coercion automatically converts string interpolation results to declared types (integer, number, boolean, array, object). See Inference Rules Guide for details.

Chunking

Chunking settings are configured under embedding:

{
  "embedding": {
    "chunkSize": 1000,
    "chunkOverlap": 200
  }
}

Enrichment Store

Enrichment metadata (from POST /metadata or watcher_set_metadata) is stored in a SQLite database at <stateDir>/enrichments.sqlite, using Node's built-in node:sqlite module (no native addon). Enrichments survive full reindexes. Composable merge: scalar fields overwrite, array fields union+deduplicate with inference rule output.

{
  "stateDir": ".jeeves-metadata"
}

API Endpoints

The watcher provides a REST API (default port: 1936):

Endpoint	Method	Description
/status	GET	Health check, uptime, collection (name, point count, dimensions, system index states), reindex and initial-scan state, VCS and embedding health; degraded while embeddings fail or a VCS breaker is tripped
/openapi.json	GET	OpenAPI 3.1 document of the catalog routes (every route above and below); open like /status
/search	POST	Semantic search ({ query: string, limit?: number, offset?: number, filter?: object }; unknown keys are rejected)
/render	POST	Render a document through inference rules ({ path } or { source }) (v0.8.0+)
/search/facets	GET	Schema-derived search facet definitions with live values (v0.8.0+); ?rules=a,b scopes them to those rules (v0.21.0+)
/search/facets/counts	POST	Approximate value counts per indexed field over a filter, each document once, or { min, max } for number and date fields ({ fields, filter?, rules?, limit? }) (v0.21.0+)
/metadata	POST	Update document metadata with schema validation ({ path | source, metadata: object })
/documents	POST	Notify changed source documents ({ documents: [{ source, rule }] }); 202, or per-document results with ?wait=true (v0.20.0+)
/documents/delete	POST	Delete source documents ({ sources } (at most 1000) or { filter }; never deletes file points) (v0.20.0+)
/reindex	POST	Scoped reindex with blast area plan (issues, rules, full, path, prune + dryRun). path accepts string | string[]; scope path also accepts source references.
/rebuild-metadata	POST	Rebuild the enrichment store from Qdrant payloads (file points only; system fields are not copied)
/config	GET	Merged config document (public keys only: never api, sources, embedding or vectorStore) with rule values, helpers and issues; optional ?path=<jsonpath> filter. Rules include source attribution.
/config/schema	GET	JSON Schema of merged virtual document (v0.5.0+)
/walk	POST	Watched files matching globs ({ globs: string[] }), from the watcher's inventory: only files that pass its filters, sorted by path. Returns { paths, matchedCount, scannedRoots }.
/config/match	POST	Test paths against inference rules ({ paths: string[] }) (v0.5.0+)
/issues	GET	Current embedding failures and processing errors, including documents over a processing ceiling (too_large)
/rules/register	POST	Register virtual inference rules from an external source
/rules/unregister	DELETE	Remove all virtual rules from a source ({ source })
/rules/unregister/:source	DELETE	Remove all virtual rules from a named source
/scan	POST	Filter-only point query with cursor pagination ({ filter, limit?, cursor?, fields?, countOnly? })
/config/validate	POST	Validate a configuration without applying ({ config?, testPaths? })
/config/apply	POST	Merge a patch, validate, write, and reload configuration ({ patch, replace? }; config is a deprecated alias of patch). The echoed config is redacted like GET /config.
/rules/reapply	POST	Re-apply inference rules to files matching globs ({ globs })
/points/delete	POST	Delete points matching a Qdrant filter ({ filter })
/vcs/*	GET/POST	Version control: status, history, show, diff, check-exclusion, revert, exclude (see the Version Control guide)

Errors carry { error, code, message?, details? }; the HTTP status follows from code, with one exception: a prune reindex without a vector store answers 400 not_available (see Error Handling). Every route is an entry of the endpoint catalog (WATCHER_CATALOG in watcher-core), which validates its request (an invalid one is 400 invalid_request with one details entry per issue) and its response: a response that doesn't match its schema is a 500 that only a server bug produces.

Example: Search

curl -X POST http://localhost:1936/search \
  -H "Content-Type: application/json" \
  -d '{"query": "machine learning algorithms", "limit": 5}'

Example: Search With Filter

curl -X POST http://localhost:1936/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "error handling",
    "limit": 10,
    "filter": {
      "must": [{ "key": "domain", "match": { "value": "backend" } }]
    }
  }'

Example: Update Metadata

curl -X POST http://localhost:1936/metadata \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/path/to/document.md",
    "metadata": {
      "priority": "high",
      "category": "research"
    }
  }'

OpenClaw Plugin

This repo includes an OpenClaw plugin (packages/openclaw) that exposes the jeeves-watcher API as native agent tools:

Tool	Description
watcher_status	Service health, version and uptime
watcher_service	Manage the system service (install, uninstall, start, stop, restart, status)
watcher_search	Semantic search across indexed documents
watcher_set_metadata	Set or update document metadata
watcher_config	Query the effective runtime config via JSONPath
watcher_walk	List watched files matching globs
watcher_validate_config	Validate a candidate config, optionally testing file paths
watcher_config_apply	Apply a config patch (response config redacted)
watcher_reindex	Trigger a scoped reindex with blast area plan
watcher_scan	Filter-only point query with cursor pagination
watcher_list_issues	List processing issues (file paths and source references)
watcher_get_vcs_status	Version tracking state for every root
watcher_list_vcs_history	Change history by path or glob, optional date range
watcher_show_vcs_file	File content at a specific version
watcher_diff_vcs	Changes between two versions, or a version and current
watcher_revert_vcs	Restore files from a historical version
watcher_exclude_vcs_path	Exclude or re-include paths from version tracking
watcher_check_vcs_exclusion	Whether a path is excluded from version tracking, and why

The plugin is a standard OpenClaw plugin on @karmaniverous/jeeves 0.9 core, installed by jeeves install. Every tool is built from the watcher's endpoint catalog by core's toolset and named watcher_ plus its catalog entry in snake_case. It writes no workspace files; its always-in-context watcher rules are injected via the before_prompt_build hook. See the OpenClaw Integration Guide for details.

Plugin configuration supports apiUrl (defaults to http://127.0.0.1:1936), an optional apiKey (sent as a bearer token when the service sets api.apiKey; env fallback JEEVES_WATCHER_API_KEY) and configRoot (no default: set it in plugin config or via JEEVES_CONFIG_ROOT; resolved lazily, so the plugin registers before it is set).

Supported File Formats

• Markdown (.md, .markdown) — with YAML frontmatter support
• PDF (.pdf) — text extraction
• DOCX (.docx) — Microsoft Word documents
• HTML (.html, .htm) — content extraction (scripts/styles removed)
• JSON (.json) — with smart text field detection
• Plain Text (.txt, .text)

Any other extension is read as plain text.

Releasing

The packages are released one at a time with npm run release (or npm run release:pre) from each package directory. Release order matters: packages/core first, then its dependents (packages/sdk, packages/service, packages/openclaw).

Each dependent's release-it before:init hook runs scripts/check-core-range.mjs, which stops the release before any version bump, tag or publish when the package's declared range for @karmaniverous/jeeves-watcher-core does not admit the version in packages/core/package.json. The check reads the local core version, not the registry, so it passes as soon as the range is raised, even before core is published. If it fails, raise the dependent's range and release core first.

License

BSD-3-Clause

---

Built for you with ❤️ on Bali by Jason Williscroft & Jeeves.