A fast, local document search engine built in Go. docsearch indexes your documents and lets you search them using keyword (BM25), semantic (vector embeddings), or hybrid search — all running entirely on your machine.
- Three search modes — keyword (BM25), semantic (cosine similarity on vector embeddings), or hybrid (combined ranking)
- Incremental indexing — only re-indexes files that have changed since the last run
- Fuzzy search — automatic spelling correction and term suggestions
- AI-powered answers — optional Gemini integration for generative answers grounded in your documents
- Auto-discovery — automatically finds document directories without manual configuration
- Graceful cancellation — interrupt indexing with
Ctrl+Cand the partial index is safely persisted - Memory-aware indexing — configurable memory limits prevent OOM on large document collections
- Chunk-based indexing — documents are split into chunks for finer-grained search and relevance
- PDF support — parses PDF documents using the pdfcpu library
- Query explainer — understand why a document matched with a per-term BM25 score breakdown
- Evaluation — built-in Precision@K / Recall@K metrics for search quality assessment
- Benchmark suite — measure indexing and search latency against a 250-document dataset
# Install directly
go install github.com/Smitbafna/local-search-engine/cmd/docsearch@latest
# Or clone and build
git clone https://github.com/Smitbafna/local-search-engine.git
cd local-search-engine
go build -o docsearch ./cmd/docsearchRequires Go 1.21+.
# Index a directory (auto-discovers documents)
docsearch index ~/Documents
# Search using hybrid mode (default)
docsearch search "distributed consensus"
# Search with keyword-only mode
docsearch search "raft algorithm" --mode keyword
# Search with semantic (vector) mode
docsearch search "how do distributed systems work" --mode semantic
# Check index status
docsearch statusIndex supported documents from the given directory. If no directory is provided, docsearch auto-discovers the best candidate directory based on heuristic scoring.
| Flag | Default | Description |
|---|---|---|
--rebuild |
false |
Delete the existing index and rebuild from scratch |
--max-doc-size |
10 MB |
Maximum document size in bytes (larger files are skipped) |
--max-memory |
0 (unlimited) |
Maximum total memory in bytes for document content. Caps concurrent workers so that workers × max-doc-size stays within the limit |
--embedding |
auto |
Embedding provider: auto, local, or gemini |
If --embedding is set to auto, docsearch uses the Gemini embedding API when GEMINI_API_KEY is set in the environment, and falls back to the local embedding provider otherwise.
docsearch index ~/Documents --max-doc-size 5242880 # 5 MB per file
docsearch index ~/Documents --max-memory 52428800 # 50 MB total
docsearch index . --rebuild # Rebuild from scratch
docsearch index ~/notes --embedding gemini # Use Gemini embeddingsAuto-discovery uses heuristic scoring (directory name + document count) to pick the best root from candidates like ~/Documents, ~/Downloads, ~/Desktop, and subdirectories of the CWD.
Search the indexed documents.
| Flag | Default | Description |
|---|---|---|
--mode |
hybrid |
Search mode: keyword, semantic, or hybrid |
--fuzzy |
false |
Force fuzzy search even if exact matches exist |
--limit |
0 (unlimited) |
Maximum number of results to return |
--verbose |
false |
Show search latency metrics (parse, lookup, ranking times) |
Search modes:
keyword— BM25 scoring on the inverted index. Fast and precise for exact term matching. Automatically falls back to fuzzy suggestions when no exact matches are found.semantic— Cosine similarity search over vector embeddings. Understands conceptual relationships even when the query doesn't share exact terms with the document.hybrid— Combines BM25 and semantic scores using min-max normalization and configurable weights (default 0.5 BM25 / 0.5 semantic). Provides the most balanced results.
docsearch search "raft consensus algorithm" # Hybrid (default)
docsearch search "consensus" --mode keyword # BM25 only
docsearch search "how does paxos work" --mode semantic # Vector only
docsearch search "consensus" --fuzzy # Force fuzzy
docsearch search "distributed systems" --limit 5 # Top 5 results
docsearch search "raft" --verbose # With latency metricsAI-powered answers (Gemini):
When GEMINI_API_KEY is set and you use hybrid or semantic mode, the top result's chunk content is passed to Gemini to generate a concise answer grounded in your document. This requires no additional setup beyond setting the environment variable.
export GEMINI_API_KEY="your-api-key-here"
docsearch search "what is the raft consensus algorithm"
# Output includes an AI-generated answer based on your indexed documentsDisplay the current index status.
$ docsearch status
DocSearch Index
Location: /home/user/.docsearch/index.json
Documents: 5
Unique terms: 1234
Index size: 25.2 MB
Last indexed: 2026-07-25 21:00Explain how a query is scored. When used alone, it shows the query plan:
$ docsearch explain "raft consensus"Shows query structure: what terms are looked up, their document frequency, and their contribution to the final score.
Use --doc to see a per-term BM25 score breakdown for a specific document:
$ docsearch explain "consensus algorithm" --doc raft.md
Document: raft.md
Query terms:
──────────────────────────────
consensus (exact, distance=0)
Term frequency: 15
Document frequency: 3
IDF: 0.28
Contribution: 2.15
algorithm (exact, distance=0)
Term frequency: 8
Document frequency: 4
IDF: 0.22
Contribution: 0.97
──────────────────────────────
Total BM25 score: 3.12Clear the entire index. Prompts for confirmation.
docsearch clear
Continue? [y/N]| Format | Status | Notes |
|---|---|---|
.txt |
✅ Full support | Plain text |
.md |
✅ Full support | Markdown with YAML frontmatter support |
.pdf |
✅ Full support | Text extraction via pdfcpu |
All index data is stored at ~/.docsearch/:
| File | Description |
|---|---|
index.json |
Inverted index (terms → postings) |
documents.json |
Document metadata |
catalog/ |
Parsed document content (content-addressed) |
vectors/ |
Vector embeddings for semantic search |
embedding_cache/ |
Cached embeddings to skip unchanged documents |
┌─────────────┐
│ Scanner │ Walks directory, finds supported files
└──────┬──────┘
│
┌──────▼──────┐
│ Registry │ Routes files to parsers by extension
└──────┬──────┘
│
┌───────────┼───────────┐
│ │ │
┌────▼───┐ ┌────▼───┐ ┌────▼───┐
│ Text │ │ Markdown│ │ PDF │ Parsers extract text content
│ Parser │ │ Parser │ │ Parser │
└────────┘ └────────┘ └────────┘
│ │ │
└───────────┼───────────┘
│
┌──────▼──────┐
│ Chunker │ Splits documents into chunks
└──────┬──────┘
│
┌───────────┼───────────┐
│ │ │
┌────▼───┐ ┌────▼───┐ ┌────▼───┐
│ BM25 │ │ Vector │ │ Store │
│ Index │ │ Index │ │Content │
└────────┘ └────────┘ └────────┘
│ │
└──────┬────┘
│
┌──────▼──────┐
│ Query │ Keyword / Semantic / Hybrid
│ Execution │
└─────────────┘
Query ──► Analyzer ──► Query Parser ──► BM25 Evaluator ──┐
│ ├──► Hybrid Ranker ──► Results
│ │
└──► Embedding ──► Cosine Similarity ──┘
Uses a lightweight in-process embedding model (384-dimensional vectors). Fast and private — no external API calls. Suitable for most use cases.
Requires a GEMINI_API_KEY environment variable. Produces 768-dimensional vectors with potentially better semantic understanding. Also enables AI-powered answers in search results.
Set the key in your environment or add it to a .env file in the current working directory:
export GEMINI_API_KEY="your-api-key-here"Or create a .env file:
GEMINI_API_KEY=your-api-key-here
docsearch includes a built-in evaluation framework using Precision@K and Recall@K metrics.
- Precision@K: Proportion of relevant documents in the top K results
- Recall@K: Proportion of all relevant documents that appear in the top K results
Query: "How do distributed systems reach agreement?"
Expected relevant documents:
raft.mdpaxos.mdconsensus-notes.md
If the top 5 results contain 3 relevant documents:
- Precision@5 = 3/5 = 0.6
- Recall@5 = 3/3 = 1.0
go test ./internal/query -run TestPrecisionRecall -vA benchmark dataset and runner are included at cmd/benchmark/.
| Metric | Value |
|---|---|
| Documents | 250 |
| Total tokens | ~290,000 |
| Unique terms | 167,115 |
| Index size | 25.2 MB |
Breakdown:
- 100 Markdown files (~200 words each)
- 100 Text files (~200 words each)
- 50 Large documents (~5000 words each)
| Benchmark | Latency |
|---|---|
| Initial indexing | 522 ms |
| Incremental indexing (1 file modified) | 29 ms |
| Exact search (avg) | 286 µs |
| Fuzzy search (avg) | 301 µs |
| Index loading | 345 ms |
go run cmd/benchmark/main.goThis will:
- Generate 250 test documents in
test-documents/ - Build and index the full dataset
- Measure and report all performance metrics
- Display a summary table
Note: The test-documents/ directory is regenerated each time the benchmark runs and should not be committed to version control.
Files exceeding a configurable size limit are skipped during scanning:
Skipped: large-file.txt
Reason: Document exceeds maximum size (10485760 bytes)
The default limit is 10 MB. Adjust it with --max-doc-size:
docsearch index <directory> --max-doc-size 5242880 # 5 MBThe --max-memory flag caps the total memory used for document content during indexing. When set, the number of concurrent workers is reduced so that workers × max-doc-size stays within the limit:
docsearch index <directory> --max-memory 52428800 # 50 MBA value of 0 (the default) means unlimited — all available CPU cores are used as workers.
After indexing, docsearch reports peak and final heap memory usage:
Peak memory: 12.3 MB
Final memory: 3.4 MB
Large documents are processed one at a time, with content released from memory after each file is indexed:
File → Parse → Store Content → Analyze → Release Memory → Next File
The indexing operation is fully cancellable. Pressing Ctrl+C (SIGINT) or sending SIGTERM during docsearch index <directory> triggers a graceful shutdown:
- Stop accepting new files — the job feeder stops dispatching new parse jobs to workers.
- Finish current work — workers complete any file they have already started processing.
- Save consistent index — the partial index is persisted to disk so it can be resumed on the next run.
- Exit cleanly — the process exits with a message indicating that the index was interrupted and partially saved.
This is implemented using context.Context, which flows through the entire pipeline:
Context → Scanner → Workers → Parser → Indexer
Every long-running operation checks for context cancellation:
- Scanner — checks
ctx.Done()before processing each file during the directory walk. - Workers — check
ctx.Done()before picking up each new job; jobs already in progress are allowed to finish. - Parser — the
Parsemethod accepts acontext.Contextand returns early if the context is already cancelled. - Indexer — indexing is performed by a single writer goroutine that processes results sequentially, ensuring the index is always in a consistent state when saved.
| Variable | Description |
|---|---|
GEMINI_API_KEY |
API key for Gemini embedding provider and AI-powered answers |
Set these in your shell or add them to a .env file in the current working directory.
cmd/
docsearch/ # Main CLI binary (cobra commands)
benchmark/ # Benchmark dataset and runner
internal/
analyzer/ # Text analysis (tokenization, normalization, stemming)
document/ # Document types, chunking, search results
embedding/ # Embedding providers (local, gemini) + cache
index/ # Inverted index, vector index, catalog, store, fuzzy lookup, snippets
parser/ # Document parsers (text, markdown, PDF), registry
query/ # Query parsing, BM25 evaluation, hybrid ranking, explanation
scanner/ # Directory scanner with size checks
documents/ # Sample documents for testing
test-documents/ # Generated benchmark documents (gitignored)