(o_o) filegoblin (filegoblin)

"Ingesting the messy world, spitting out clean context."

filegoblin is a single-binary CLI that "gobbles" almost any file or website — PDF, Word, Excel, PowerPoint, email, EPUB, HTML, whole repos — and converts it into clean, LLM-ready markdown.

Philosophy

It just works. filegoblin document.pdf > context.md needs no manual. Power flags exist, but the default path is the product.
Format invisibility. You shouldn't have to think about extensions; detection is extension-first with magic-byte fallback, so renamed and extension-less files route correctly.
Fail loudly, never silently. An empty result exits non-zero, parse failures are reported in-band, and invalid flags are hard errors. An agent can trust every exit code.
Local and private. Parsing and PII scrubbing happen entirely on your machine. No cloud APIs, no runtimes, no pip install.
Not just a concatenator. cat * > out.txt exists; filegoblin adds structure (real markdown from Word, one consistent table dialect, readability-grade web extraction), token budgeting, and redaction.

📦 Installation

Every release comes in two flavours:

with the PDF vision models for layout and reading order, formulas as LaTeX, tables, OCR on scanned pages, and a page reader for handwritten and math scans (~1.3 GB; ~510 MB without the page reader);
lite, without them: the fast geometric converter, plus Apple Vision OCR on macOS.

Everything else is the same.

Channel	Lite	With the models
GitHub release (latest)	`filegoblin-<target>` archive	`filegoblin-full-<target>` archive: one binary with the models compiled in
PyPI	`pip install filegoblin`	`pip install filegoblin`, then `filegoblin --install-models` (the `filegoblin[models]` packages aren't on PyPI yet)
npm	`npm i -g filegoblin`	`npm i -g filegoblin @filegoblin/models`
Cargo	`cargo install --git https://github.com/funkyfunc/filegoblin filegoblin`	then `filegoblin --install-models`

Builds are published for macOS (Apple Silicon and Intel), Linux (x86_64 and aarch64), and Windows (x86_64).

Getting the models later.

filegoblin --install-models downloads this version's model archive from the GitHub release, checks its SHA-256 and unpacks it into ~/.filegoblin/models. You can pass a directory instead, and set FILEGOBLIN_MODELS_URL to point at an internal mirror. filegoblin only downloads when you ask it to.
filegoblin --models-status shows where each model was found.

Behind a corporate network. The npm model packages (and the PyPI ones, once they're published) carry the weights themselves, one package per model, with the two largest models in halves to stay under the registries' 100 MB per-file limit. They install from a company mirror (Artifactory, Nexus, CodeArtifact) like any other package, with no access to github.com.

Air-gapped machines. Copy the full download. Alternatively, unpack the filegoblin-vision-models.tar.gz archive on a shared read-only path and name that directory in FILEGOBLIN_MODELS (the archive itself is not read). That variable takes a PATH-style list. Licenses are in LICENSES.md inside the archive.

From source — needs a Rust toolchain (1.87+):

cargo install --git https://github.com/funkyfunc/filegoblin filegoblin

That gives you everything in the "always on" list below, including web fetching and crawling. The optional engines are opt-in because each one pulls in a heavy dependency:

Want	Add	Build needs
OCR on Linux/Windows	`--features ocr`	`just get-brains` for the weights (macOS uses Apple Vision, no flag)
PDF vision models (layout, reading order, formulas → LaTeX, tables, scan OCR)	`--features vision` (on in release builds)	The weights: `filegoblin --install-models` (the release's `filegoblin-vision-models.tar.gz`, SHA-256 checked) or `just build-vision-models`. filegoblin finds them in `models/` next to the binary, in `~/.filegoblin/models`, or in `$FILEGOBLIN_MODELS`. Add `bundled-models` with `FILEGOBLIN_MODELS_DIR` set to compile them into one binary
Audio/video transcription	`--features transcribe`	`cmake` + a C/C++ compiler; downloads a ~574 MB model on first use
Metal-accelerated transcription (macOS only)	`--features transcribe-metal`	Xcode Command Line Tools. Does not build on Linux/Windows
WASM plugins	`--features wasm`	—
BM25 search (`--search`)	`--features rag`	—

# Example: a macOS build with GPU-accelerated transcription
cargo install --git https://github.com/funkyfunc/filegoblin filegoblin --features transcribe-metal

Without the models, --vision smart (the default) behaves exactly like the fast converter, with one note per run on stderr; --vision off silences it. With the models, it uses them only on the pages that need them. Anyone building from source can make a single self-contained binary with --features vision,bundled-models and FILEGOBLIN_MODELS_DIR set.

Measuring quality. just gauntlet runs the 29-case adversarial gauntlet (just gauntlet off for the fast path). Each case gets pass/fail canaries and a 0–100 quality score graded against hand-checked gold excerpts in test_assets/gauntlet/gold/, plus checks that --format plain and --chunk-size lose nothing. just gauntlet-compare A B shows what a change did case by case. Results are in GAUNTLET_SCORECARD.md.

🏗 Current Project Structure

Note to Agent: Maintain this section as you add modules.

.
├── Cargo.toml        (Virtual Workspace Root)
├── crates/
│   ├── core/         (filegoblin-core: The Engine & Parsers)
│   │   ├── Cargo.toml
│   │   ├── build.rs  (Man page & shell completions)
│   │   └── src/
│   │       ├── lib.rs        (Core Pipeline Logic)
│   │       ├── cli.rs        (Clap Flag Definitions)
│   │       ├── formatters.rs (Output Assembly & Splitting)
│   │       ├── formatters/   (Plain text & specialized formatters)
│   │       ├── chunking/     (Layout-aware semantic chunking engine)
│   │       ├── curation/     (BM25 Search & Auto-Pruning)
│   │       ├── privacy_shield.rs (PII Redaction Engine)
│   │       ├── compressor/   (Token Compression Pipeline)
│   │       ├── parsers/      (File parsing engines; pdf_vision.rs = layout-guided PDF assembly,
│   │       │                  pdf_furniture.rs + pdf_edges.rs = page furniture, pdf_words.rs = glyph-level
│   │       │                  words, pdf_math.rs = inline-math clean-up, pdf_worker.rs = PDFs converted
│   │       │                  in child processes)
│   │       ├── vision/       (Bundled ONNX models on rten: layout, formula, table, OCR, page reader; hayro rendering)
│   │       └── pipeline/     (Pipeline execution stages)
│   └── cli/          (filegoblin: The Headless Binary)
│       ├── Cargo.toml
│       └── src/main.rs       (CLI Entry & Mascot)
│           └── walk.rs       (--walk interactive picker)
├── scripts/          (Dev tooling: run_gauntlet.py = graded gauntlet, olmocr_bench.py = olmOCR-bench
│                      on a fixed dev/test split, engines.py = other tools graded the same way
│                      (--engine; docs/COMPARISON.md), model packaging and release helpers)
├── npm/              (npx filegoblin Zero-Install Launcher)
│   ├── package.json
│   └── bin/filegoblin.js
└── .cargo/
    └── config.toml   (Aliases)

⚡ Current Capabilities (v1.17.0)

Some capabilities are gated behind opt-in Cargo features (built off by default to keep the binary lean). Feature-gated items are marked below.

✨ Key Features

Parsers: pdf_oxide, docx-rs, tree-sitter, quick-xml, calamine, scraper (Statically Linked).
OCR built in on macOS: Apple Vision (objc2) with no weights or flags — scanned PDFs and images just work. Linux gets pure-Rust ocrs/rten inference behind --features ocr.
WASM Extensibility: wasmtime for pure-rust statically-linked component execution.
Privacy First: 100% local PII/secret scrubbing — deterministic patterns (SSN, email, cloud/token prefixes) plus a Shannon-entropy tier that redacts high-entropy secrets. No model weights, no data egress.
Structural Minification: Tree-sitter powered --extract symbols mode for Rust source.

Source	Path Example	Parser Used	Output Format
PDF	`~/receipt.pdf`	`PdfGobbler`	Markdown (headings, pipe tables, OCR for scans)
Word	`~/notes.docx`	`OfficeGobbler`	Markdown (headings, lists, emphasis, links, equations, charts)
Excel/CSV	`~/data.xlsx`	`SheetGobbler`	Markdown pipe tables (header row named)
PowerPoint	`~/deck.pptx`	`PptxGobbler`	Per-slide markdown + speaker notes + embedded charts
OpenDocument	`~/doc.odt`, `~/deck.odp`	`OpenDocGobbler`	Markdown (headings, lists, tables)
Source Code	`~/project/src`	`CodeGobbler`	Full content (or `--extract symbols`)
Email	`~/thread.eml`, `~/inbox.mbox`	`EmailGobbler`	Header block + markdown body + attachment list
RTF	`~/notes.rtf`	`RtfGobbler`	Plain text
Images	`~/screenshot.png`, `~/IMG_0042.HEIC`	`OcrGobbler` (built-in on macOS)	OCR Text
Subtitles	`~/movie.srt`, `~/talk.vtt`	`SubtitleGobbler`	Timestamped transcript
RSS/Atom	`https://hnrss.org/frontpage`	`FeedGobbler`	Entry list (title, link, date, body)
Webpages	`https://news.ycombinator.com`	`HtmlGobbler` (default build)	DOM extraction + front-matter

Semantic Search: BM25 relevance scoring via tantivy with zero model weights.
Manifest TOC: Auto-generated table of contents with per-file token counts.

⚙️ Development & Asset Pipeline

filegoblin maintains a strict zero-dependency profile for the end user. However, building the project requires assembling several logic assets.

build.rs generates the manpage and shell completions into target/generated/. The .rten tensor weights for pure-Rust OCR fallback on Linux/Windows are not downloaded automatically — fetch them with the provided justfile when building with --features ocr:

just get-brains

(On macOS, OCR uses Apple's Vision framework natively; no weights needed.)

🚀 Quick Start

Basic Gobble (Files):

# Output a single file to stdout
filegoblin my_notes.pdf > context.md

# Merge multiple files natively
filegoblin src/main.rs src/lib.rs README.md --write context.md

URL Ingestion (Web):

filegoblin https://example.com/api-docs > context.md

Web Horde (Split into mapping directory):

filegoblin https://bettercli.org/ --horde --split
# Translates to -> ./bettercli.org_gobbled/

Privacy Shield (Redact PII & Secrets locally):

# Typed redaction (default): values become [EMAIL], [AWS_KEY], [CREDIT_CARD], …
filegoblin ./src/api_keys.ts --scrub > safe_context.md

# Pipe a log through it as a pre-flight gate before any cloud LLM
cat prod.log | filegoblin - --scrub --scrub-style plain

# CI / pre-commit guard: non-zero exit if anything sensitive is found
git ls-files -z | xargs -0 cat | filegoblin - --scrub --scrub-fail -q || echo "secrets detected!"

# Exempt a known-public value, or add your own pattern via ./.filegoblin-scrub.json
filegoblin ./config --horde --scrub --scrub-allow "support@example.com"

# Consistent pseudonyms: the same value maps to [EMAIL_1] everywhere, so an LLM can
# still tell that an entity recurs — without ever seeing it.
cat ticket.txt | filegoblin - --scrub --scrub-style pseudonym

Detection is 100% local with zero model weights: validated rules (Luhn for cards, range checks for SSN, mod-97 for IBAN) plus cloud/token prefixes (AWS, GitHub, Stripe, Slack, Google, OpenAI, JWT, private-key blocks, connection strings) and a Shannon-entropy catch-all for unknown secrets. A value-free audit tally is printed to stderr (🛡️ Privacy Shield redacted 4 item(s): …). An optional .filegoblin-scrub.json supplies an allowlist and custom patterns.

What filegoblin does NOT do: maintain a secrets blocklist. It converts exactly what you point it at — including a .env file if you name one. Deciding what's sensitive is yours: the directory walker respects .gitignore (where secrets usually already live), --exclude filters by glob, and --scrub redacts what does get through.

Machine Contract (--json) — extract any file to clean text for agents/scripts:

# One weird binary format → clean text + metadata, in a single call
filegoblin report.pdf --json | jq '.[0] | {format, tokens}'
# → { "format": "pdf", "tokens": 1234 }

filegoblin ./src/ --horde -q --json | jq '.[].path'

--json emits a deterministic, alphabetically-sorted array. A successful record is { "path", "format", "tokens", "content" } (format is a coarse type like pdf, spreadsheet, code, image, text; tokens is an LLM token estimate). A file that fails to parse appears in-band as { "path", "format", "error" } rather than silently vanishing — so an agent can trust that every input is accounted for. Exit codes: 0 success, 4 when nothing could be extracted (empty result vs. total failure), 3 when --scrub --scrub-fail redacted something, 5 when the run went over its memory ceiling (--memory).

Multi-Format Output (--format markdown|plain|json / -f):

# Default GFM markdown with tables, math, and code fences
filegoblin document.pdf -f markdown > doc.md

# Strip markdown punctuation for dense vector embeddings & BM25 search
# (Pipe tables become space-aligned text columns; headings & math stripped of tokens)
filegoblin document.pdf -f plain > embedding_input.txt

# Structured JSON document array for agent pipelines
filegoblin document.pdf -f json > document.json

Layout-Aware Semantic Chunking (--chunk-size, --chunk-overlap, --chunk-strategy):

# Split along structural boundaries (H1/H2/H3, tables, code blocks) with 512 token budget
# Replicates table headers across chunk boundaries and embeds hierarchical breadcrumbs
filegoblin document.pdf --chunk-size 512 > chunks.md

# Structured JSON chunks with breadcrumbs and token estimates for RAG vectors
filegoblin document.pdf -f json --chunk-size 512 --chunk-overlap 64 > rag_chunks.json

Token Compression (Reduce LLM context sizes):

# Strip code comments, minify JSON/HTML, collapse whitespace
filegoblin ./src/main.rs --compress contextual > context.md

Multi-Part Splitting (Chunk output by tokens):

# Automatically break massive repositories into `partN` files at a 50k token threshold
filegoblin ./src/ --horde --chunk 50k

Pipe from stdin:

curl -s "https://api.github.com/users/octocat" | filegoblin -q --json

Clipboard Ingestion:

# Read contents directly from your OS clipboard
filegoblin --clipboard > context.md

Development & Intelligence:

# Get a heuristic overview of the project with token counts
filegoblin ./src/ --summary --tokens

Filtering & Exclusion:

# Only Rust files, excluding tests and generated code
filegoblin ./src/ --horde --include "*.rs" --exclude "*test*" --exclude "*generated*"

# Limit crawl depth to top-level files only
filegoblin ./src/ --horde --depth 1

Manifest / Table of Contents:

# Prepend a TOC with file paths and token counts
filegoblin ./src/ --horde --include "*.rs" --manifest

Token-Only Mode (Scripting):

# Print just the token count, no content — ideal for CI/scripts
filegoblin ./src/ --horde --tokens-only

Semantic Search with Relevance Scores:

# Search across a codebase and see BM25 relevance scores
filegoblin ./src/ --horde --search "authentication"

Git-Diff Mode (Only Changed Files):

# Ingest only files changed since last commit
filegoblin . --horde --git-diff HEAD~1

# Show unified diffs instead of full file contents
filegoblin . --horde --git-diff HEAD~1 --diff-format

Scoping a crawl (--depth, --include, --exclude):

These apply to both halves of --horde — local file paths, and the URL path when crawling. Out-of-scope pages are never fetched, so scoping saves requests rather than just trimming the output.

# Only the guide section, and never the changelog
filegoblin https://docs.example.com --horde --include "/guide/*" --exclude "*changelog*"

# The seed page and whatever it links to directly — nothing further
filegoblin https://docs.example.com --horde --depth 1

# Shorthand: a bare word matches anywhere in the path
filegoblin https://docs.example.com --horde --include guide

The seed URL is always fetched, so --include "/guide/*" starting at / still works. --depth 0 is the seed alone.

Walk and pick (--walk) — for when you don't want the whole site:

Pointing --horde at a website converts everything it can reach: the changelog, the 2019 blog archive, every language variant. --walk opens a picker instead — read each page, follow its links, mark what you want, convert only that.

filegoblin --walk https://docs.example.com > context.md
filegoblin --walk ./my-repo > context.md

 https://docs.example.com/guide/     8.4k tok  ↳1
 # Getting Started

 Install the CLI, then point it at a file. Every
 conversion happens locally on your machine.

 ─ pick ──────────────────────────────────────────
 ❯ ◉ this page — Getting Started
   ○ Next: Configuration
   ○ API Reference

  1 selected   ↑↓ move · space keep · ⏎ follow · b back · d done · q quit

The preview is the actual converted markdown, so you're judging what you'd get, not a guess. Links are scoped to the same origin (host and port), and selecting a directory includes everything beneath it. Pages you revisit come from the conversion cache, so backing up is instant.

The picker draws on stderr and the markdown goes to stdout, so redirecting works normally. It needs a terminal — piping without one is a clear error rather than a hang, and every other invocation is unchanged, so scripts, agents and the MCP server are unaffected.

MCP Server (Claude Desktop / Claude Code):

filegoblin --mcp speaks the Model Context Protocol over stdio, exposing a single convert tool that takes a file path, directory, or URL.

For Claude Code:

claude mcp add filegoblin -- filegoblin --mcp

For Claude Desktop, add this to claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\):

{
  "mcpServers": {
    "filegoblin": {
      "command": "filegoblin",
      "args": ["--mcp"]
    }
  }
}

Use an absolute path to the binary if it isn't on the PATH your client sees. To let the tool reach services on your own machine, add "env": { "FILEGOBLIN_ALLOW_LOCAL_URLS": "1" } — see the security note below for why that is off by default.

🗺️ Roadmap

Model packages on PyPI: pip install "filegoblin[models]", once the model projects exist there (see docs/RELEASING.md).
Crates.io (cargo install filegoblin): publish the CLI binary and the filegoblin-core library crate.
Homebrew tap (brew install funkyfunc/tap/filegoblin).

🔒 Security notes

URLs pointing at private addresses are refused. Loopback, private ranges, link-local (including the cloud metadata endpoint 169.254.169.254), and any hostname that resolves to one are blocked — on the initial URL, on every redirect hop, and inside the DNS resolver itself, so a name that resolves differently between the check and the connection (DNS rebinding) is refused too. IPv6 spellings of those addresses ([::ffff:127.0.0.1], NAT64, 6to4) count as the address they carry. Set FILEGOBLIN_ALLOW_LOCAL_URLS=1 (or true) to allow them — reasonable for converting your own dev server, and worth thinking about before enabling it for the MCP server, where the URL comes from a model.
--browser goes through the same guard. Chrome is pointed at a small proxy inside filegoblin, so every connection the page makes — redirects, JavaScript navigations, images and scripts — is checked like a direct fetch. A render that ends on a refused address fails; refused subresources are listed on stderr. The profile is a throwaway temp dir, and the mode is opt-in.
Archives and Office files have decompression caps. Zip, tar, tar.gz, and the zip-container formats (.docx/.pptx/.odt/.epub) bound what they inflate and refuse entries whose paths escape the extraction directory.
A damaged file fails on its own. A parser that panics, a zip document with malformed XML (which some readers loop on forever) and HTML nested thousands of levels deep are each that file's error; the rest of the run goes on.
Symlinks stay inside the tree. In a directory walk (--horde, an extracted archive, a cloned repository) a symlink is followed only if it resolves inside the walked directory; links that point elsewhere are skipped and listed on stderr. A file you name directly is read wherever it points.
A hostile PDF fails on its own. The CLI converts every PDF in a worker process (its own binary, re-run), so nothing a PDF does to the parser can end or exhaust the run. A worker that crashes (a stack overflow from nesting too deep), uses more than its share of memory (a decompression bomb) or finishes no page for 120 seconds (pathological, effectively endless work) is stopped; that file fails with an error saying which (exit 4 if nothing else converted), and the next PDF gets a fresh worker. Workers are reused across files (at most two at once, FILEGOBLIN_PDF_WORKERS), so the vision models load once per worker. FILEGOBLIN_PDF_MEMORY_MB and FILEGOBLIN_PDF_TIMEOUT (seconds) set the limits; FILEGOBLIN_PDF_WORKER=0 converts in-process, for debugging only, since a hostile PDF can then take the whole process down again. Nesting the pre-scan can see (the file's syntax, its unfiltered and Flate streams) is still refused before parsing, and a parser panic is still an ordinary error. Library users of filegoblin-core convert in-process unless they opt in (parsers::pdf_worker).
No run can take the machine's memory. A run's budget is the machine's RAM, or what --memory says (--memory 2G on a small instance, --memory 64G on a big one; FILEGOBLIN_MEMORY is the same setting for wrappers that cannot pass flags). The CLI process has the same self-measured ceiling as a PDF worker: past half the budget (FILEGOBLIN_MEMORY_MB sets it directly) it stops with a message and exit 5. The PDF workers share the other half (a quarter of the budget each with the default two, or one worker when two would get under 1 GB each; FILEGOBLIN_PDF_MEMORY_MB sets one worker's limit instead), so together the run's processes stay within the budget. Input it reads as a stream is bounded too: an MCP message over 1 MB is answered with an error and dropped as it arrives, and piped stdin is read up to 1 GB.
Many runs at once cannot take it either. A budget bounds one run, so conversions that use the vision models also share the machine: one at a time per 8 GB of memory (FILEGOBLIN_VISION_SLOTS sets the number), across every filegoblin process of the user. Others wait, saying so once, and start as slots free up.
A damaged legacy Office file fails on its own. .doc, .ppt and .xls are read on a separate thread with its own stack, so a reader panic is an error for that file. 1,600 truncated and corrupted variants of the fixtures (700 of them from files Word and PowerPoint saved) all exited 0 or 4, using at most 29 MB.

🛠 Contributing

All code must pass the ritual before being merged:

cargo horde-check   # clippy -D warnings + fmt --check + test, via .cargo/config.toml

Golden-file tests live in crates/cli/tests/ and use insta snapshots; run cargo insta review to inspect a diff before accepting it.

🎨 Vibe & Aesthetics

filegoblin embraces the Mischievous Librarian archetype. We don't "process" data; we feast on it.

Navigation Menu

Repository navigation

About

Repository files navigation

(o_o) filegoblin (filegoblin)

Philosophy

📦 Installation

🏗 Current Project Structure

⚡ Current Capabilities (v1.17.0)

✨ Key Features

⚙️ Development & Asset Pipeline

🚀 Quick Start

🗺️ Roadmap

🔒 Security notes

🛠 Contributing

🎨 Vibe & Aesthetics

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.cargo		.cargo
.githooks		.githooks
.github/workflows		.github/workflows
crates		crates
docs		docs
examples		examples
npm		npm
packaging		packaging
scripts		scripts
test_assets		test_assets
.gitignore		.gitignore
BENCHMARK_REPORT.md		BENCHMARK_REPORT.md
CLAUDE.md		CLAUDE.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
GAUNTLET_SCORECARD.md		GAUNTLET_SCORECARD.md
GEMINI.md		GEMINI.md
LICENSE		LICENSE
README.md		README.md
dist-workspace.toml		dist-workspace.toml
justfile		justfile

Latest commit

History

Folders and files

Repository files navigation

(o_o) filegoblin (filegoblin)

Philosophy

📦 Installation

🏗 Current Project Structure

⚡ Current Capabilities (v1.17.0)

✨ Key Features

⚙️ Development & Asset Pipeline

🚀 Quick Start

🗺️ Roadmap

🔒 Security notes

🛠 Contributing

🎨 Vibe & Aesthetics

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages