binsg — a tiny local semantic grep

Binary-quantized embedding search over text files or whole directories.

Project layout

This is a Cargo workspace, split so the reusable engine and the CLI are separate things:

• crates/binsg-core — the actual engine: the .bsg file format, quantization, the incremental directory indexer, and the parallel Hamming-distance search. No dependency on candle, HTTP, or any specific model — it works against anything implementing the TextEmbedder trait.
• crates/binsg-candle — a TextEmbedder implementation backed by a local BERT/MiniLM model via candle, including the zero-setup auto-download.
• crates/binsg-cli — the binsg command-line tool, a thin wrapper gluing the two together with clap.

If you just want the CLI, none of this changes how you use it. If you want to embed binary-quantized search into your own Rust program — possibly with your own embedding backend instead of MiniLM — depend on binsg-core directly; see below.

Quick start (zero setup)

You don't need to fetch the model yourself — the first index or search call downloads it automatically into --model-dir (default ./model) if it's missing, then never touches the network again.

cargo build --release
cargo run --release -- index ./my-notes --output notes.bsg
cargo run --release -- search "how do I reset my password" notes.bsg

First run prints a one-time "fetching..." line per file while it grabs config.json, tokenizer.json, and model.safetensors (~90MB total, sentence-transformers/all-MiniLM-L6-v2) from HuggingFace. Every run after that is fully offline.

If you're behind a TLS-intercepting proxy (common on corporate/enterprise networks) and the download fails with a certificate error, that's what the native-certs feature on ureq (in binsg-candle) is for — it makes the downloader trust your OS's certificate store instead of only a bundled public root list, the same way curl already does. It's on by default in this repo.

Indexing a directory, not just a file

index accepts either a single file or a directory:

cargo run --release -- index ./my-project --output project.bsg

Directories are walked recursively using the same gitignore-aware walker ripgrep uses — anything your .gitignore excludes is skipped automatically, along with anything that looks binary (a NUL byte in the first few KB).

Incremental re-indexing

Re-running index against an output file that already exists is an update, not a rebuild: each file's content is hashed, and only files whose hash changed since the last run are actually re-embedded. If nothing changed at all, the embedder is never even constructed, so a no-op re-index costs milliseconds, not a model load.

cargo run --release -- index ./my-notes --output notes.bsg   # first run: embeds everything
echo "one more line" >> my-notes/todo.txt
cargo run --release -- index ./my-notes --output notes.bsg   # second run: re-embeds only todo.txt

Searching

cargo run --release -- search "how do I reset my password" notes.bsg --top-k 5

Output is grep-style: similarity path:line: text, most similar first. Similarity is 1 - hamming_distance / dim_bits, a proxy for cosine similarity between the original float embeddings — treat it as a ranking signal, not a calibrated probability.

Using it as a library

Depend on binsg-core directly to get the file format, indexer, and search without the CLI or candle:

[dependencies]
binsg-core = { path = "path/to/binsg/crates/binsg-core" }  # or a version, once published

use binsg_core::{TextEmbedder, build_index, search_top_k, format::BsgFile, quantize};

struct MyEmbedder; // wire up to whatever produces your embeddings
impl TextEmbedder for MyEmbedder {
    fn dim(&self) -> usize { 384 }
    fn embed(&self, text: &str) -> anyhow::Result<Vec<f32>> {
        todo!("call your own model or API here")
    }
}

let stats = build_index(&input_path, &output_path, || Ok(MyEmbedder))?;
let bsg = BsgFile::open(&output_path)?;
let query_bits = quantize(&MyEmbedder.embed("some query")?);
let results = search_top_k(&bsg, &query_bits, 10); // Vec<(entry_index, hamming_distance)>

binsg-candle is one TextEmbedder implementation, not the only one that could exist — swap in a remote API call, a different local model, whatever fits.

Installing a prebuilt binary

.github/workflows/release.yml builds binsg for Linux, macOS (Intel and Apple Silicon), and Windows on every pushed version tag (v*) and attaches each as a downloadable asset on the GitHub Release. Once a repo exists and a tag is pushed:

git tag v0.1.0
git push origin v0.1.0

...GitHub Actions builds and attaches the binaries automatically. Note: this workflow is written and its build steps verified locally, but the actual CI run (particularly the Windows build of tokenizers' onig C dependency) hasn't been exercised on real GitHub Actions runners yet — worth watching the first tagged run.

Homebrew

packaging/homebrew/binsg.rb is a formula stub that builds from source via cargo install. It has placeholder url/sha256 fields — fill those in once a real tagged release exists (the sha256 is the hash of that release's source tarball), then host it in a <you>/homebrew-binsg tap repo.

Build notes

First build compiles candle and its gemm kernels from scratch (a couple of minutes); incremental builds after that are fast. Both debug and --release builds were verified to complete successfully.

Cargo.lock pins half to 2.4.1 — at the time this was built, the newest published half had drifted to a rand_distr version incompatible with the one candle-core itself expects, surfacing as a Distribution<f16> trait error. If you ever regenerate the lockfile and hit that, re-pin with:

cargo update -p half --precise 2.4.1

Tested behavior

cargo test -p binsg-core covers, entirely offline (no model, no network): the binary format (quantization, packing, multi-file round-trip, corruption detection), content hashing, the parallel search ranking, and the incremental indexer's actual decisions (first-run embeds everything, an unchanged re-run reuses everything and never constructs an embedder, a partial change re-embeds only the changed file, deleted files drop out, .gitignore-excluded and binary-looking files are skipped) — using a fake embedder so this runs in milliseconds.

The full pipeline — directory walk, .gitignore handling, incremental reuse, and the auto-download (from both an empty and a partially-populated model directory) — was additionally verified manually against the real MiniLM model, both before and after the workspace split into binsg-core/binsg-candle/binsg-cli.