(o_o) filegoblin (filegoblin)

"Ingesting the messy world, spitting out clean context."

filegoblin is a single-binary CLI that "gobbles" almost any file or website — PDF, Word, Excel, PowerPoint, email, EPUB, HTML, whole repos — and converts it into clean, LLM-ready markdown.

Philosophy

It just works. filegoblin document.pdf > context.md needs no manual. Power flags exist, but the default path is the product.
Format invisibility. You shouldn't have to think about extensions; detection is extension-first with magic-byte fallback, so renamed and extension-less files route correctly.
Fail loudly, never silently. An empty result exits non-zero, parse failures are reported in-band, and invalid flags are hard errors. An agent can trust every exit code.
Local and private. Parsing and PII scrubbing happen entirely on your machine. No cloud APIs, no runtimes, no pip install.
Not just a concatenator. cat * > out.txt exists; filegoblin adds structure (real markdown from Word, one consistent table dialect, readability-grade web extraction), token budgeting, and redaction.

📦 Installation

Every release comes in two flavours:

with the PDF vision models for layout and reading order, formulas as LaTeX, tables, and OCR on scanned pages (~510 MB);
lite, without them: the fast geometric converter, plus Apple Vision OCR on macOS.

Everything else is the same.

Channel	Lite	With the models
GitHub release (latest)	`filegoblin-<target>` archive	`filegoblin-full-<target>` archive: one binary with the models compiled in
PyPI	`pip install filegoblin`	`pip install filegoblin`, then `filegoblin --install-models` (the `filegoblin[models]` packages aren't on PyPI yet)
npm	`npm i -g filegoblin`	`npm i -g filegoblin @filegoblin/models`
Cargo	`cargo install --git https://github.com/funkyfunc/filegoblin filegoblin`	then `filegoblin --install-models`

Builds are published for macOS (Apple Silicon and Intel), Linux (x86_64 and aarch64), and Windows (x86_64).

Getting the models later.

filegoblin --install-models downloads this version's model archive from the GitHub release, checks its SHA-256 and unpacks it into ~/.filegoblin/models. You can pass a directory instead, and set FILEGOBLIN_MODELS_URL to point at an internal mirror. filegoblin only downloads when you ask it to.
filegoblin --models-status shows where each model was found.

Behind a corporate network. The npm model packages (and the PyPI ones, once they're published) carry the weights themselves, one package per model, with the two largest models in halves to stay under the registries' 100 MB per-file limit. They install from a company mirror (Artifactory, Nexus, CodeArtifact) like any other package, with no access to github.com.

Air-gapped machines. Copy the full download. Alternatively, put the filegoblin-vision-models.tar.gz archive on a shared read-only path and name it in FILEGOBLIN_MODELS. That variable takes a PATH-style list. Licenses are in LICENSES.md inside the archive.

From source — needs a Rust toolchain (1.87+):

cargo install --git https://github.com/funkyfunc/filegoblin filegoblin

That gives you everything in the "always on" list below, including web fetching and crawling. The optional engines are opt-in because each one pulls in a heavy dependency:

Want	Add	Build needs
OCR on Linux/Windows	`--features ocr`	`just get-brains` for the weights (macOS uses Apple Vision, no flag)
PDF vision models (layout, reading order, formulas → LaTeX, tables, scan OCR)	`--features vision` (on in release builds)	The weights: `filegoblin --install-models` (the release's `filegoblin-vision-models.tar.gz`, SHA-256 checked) or `just build-vision-models`. filegoblin finds them in `models/` next to the binary, in `~/.filegoblin/models`, or in `$FILEGOBLIN_MODELS`. Add `bundled-models` with `FILEGOBLIN_MODELS_DIR` set to compile them into one binary
Audio/video transcription	`--features transcribe`	`cmake` + a C/C++ compiler; downloads a ~574 MB model on first use
Metal-accelerated transcription (macOS only)	`--features transcribe-metal`	Xcode Command Line Tools. Does not build on Linux/Windows
WASM plugins	`--features wasm`	—
BM25 search (`--search`)	`--features rag`	—

# Example: a macOS build with GPU-accelerated transcription
cargo install --git https://github.com/funkyfunc/filegoblin filegoblin --features transcribe-metal

Without the models, --vision smart (the default) behaves exactly like the fast converter, with one note per run on stderr; --vision off silences it. With the models, it uses them only on the pages that need them. Anyone building from source can make a single self-contained binary with --features vision,bundled-models and FILEGOBLIN_MODELS_DIR set.

Measuring quality. just gauntlet runs the 29-case adversarial gauntlet (just gauntlet off for the fast path). Each case gets pass/fail canaries and a 0–100 quality score graded against hand-checked gold excerpts in test_assets/gauntlet/gold/, plus checks that --format plain and --chunk-size lose nothing. just gauntlet-compare A B shows what a change did case by case. Results are in GAUNTLET_SCORECARD.md.

🏗 Current Project Structure

Note to Agent: Maintain this section as you add modules.

.
├── Cargo.toml        (Virtual Workspace Root)
├── crates/
│   ├── core/         (filegoblin-core: The Engine & Parsers)
│   │   ├── Cargo.toml
│   │   ├── build.rs  (Man page & shell completions)
│   │   └── src/
│   │       ├── lib.rs        (Core Pipeline Logic)
│   │       ├── cli.rs        (Clap Flag Definitions)
│   │       ├── formatters.rs (Output Assembly & Splitting)
│   │       ├── formatters/   (Plain text & specialized formatters)
│   │       ├── chunking/     (Layout-aware semantic chunking engine)
│   │       ├── curation/     (BM25 Search & Auto-Pruning)
│   │       ├── privacy_shield.rs (PII Redaction Engine)
│   │       ├── compressor/   (Token Compression Pipeline)
│   │       ├── parsers/      (File parsing engines; pdf_vision.rs = layout-guided PDF assembly)
│   │       ├── vision/       (Bundled ONNX models on rten: layout, formula, table, OCR; hayro rendering)
│   │       └── pipeline/     (Pipeline execution stages)
│   └── cli/          (filegoblin: The Headless Binary)
│       ├── Cargo.toml
│       └── src/main.rs       (CLI Entry & Mascot)
│           └── walk.rs       (--walk interactive picker)
├── npm/              (npx filegoblin Zero-Install Launcher)
│   ├── package.json
│   └── bin/filegoblin.js
└── .cargo/
    └── config.toml   (Aliases)

⚡ Current Capabilities (v1.15.0)

Some capabilities are gated behind opt-in Cargo features (built off by default to keep the binary lean). Feature-gated items are marked below.

✨ Key Features

Parsers: pdf_oxide, docx-rs, tree-sitter, quick-xml, calamine, scraper (Statically Linked).
OCR built in on macOS: Apple Vision (objc2) with no weights or flags — scanned PDFs and images just work. Linux gets pure-Rust ocrs/rten inference behind --features ocr.
WASM Extensibility: wasmtime for pure-rust statically-linked component execution.
Privacy First: 100% local PII/secret scrubbing — deterministic patterns (SSN, email, cloud/token prefixes) plus a Shannon-entropy tier that redacts high-entropy secrets. No model weights, no data egress.
Structural Minification: Tree-sitter powered --extract symbols mode for Rust source.

Source	Path Example	Parser Used	Output Format
PDF	`~/receipt.pdf`	`PdfGobbler`	Markdown (headings, pipe tables, OCR for scans)
Word	`~/notes.docx`	`OfficeGobbler`	Markdown (headings, lists, emphasis, links, equations, charts)
Excel/CSV	`~/data.xlsx`	`SheetGobbler`	Markdown pipe tables (header row named)
PowerPoint	`~/deck.pptx`	`PptxGobbler`	Per-slide markdown + speaker notes + embedded charts
OpenDocument	`~/doc.odt`, `~/deck.odp`	`OpenDocGobbler`	Markdown (headings, lists, tables)
Source Code	`~/project/src`	`CodeGobbler`	Full content (or `--extract symbols`)
Email	`~/thread.eml`, `~/inbox.mbox`	`EmailGobbler`	Header block + markdown body + attachment list
RTF	`~/notes.rtf`	`RtfGobbler`	Plain text
Images	`~/screenshot.png`, `~/IMG_0042.HEIC`	`OcrGobbler` (built-in on macOS)	OCR Text
Subtitles	`~/movie.srt`, `~/talk.vtt`	`SubtitleGobbler`	Timestamped transcript
RSS/Atom	`https://hnrss.org/frontpage`	`FeedGobbler`	Entry list (title, link, date, body)
Webpages	`https://news.ycombinator.com`	`HtmlGobbler` (default build)	DOM extraction + front-matter

Semantic Search: BM25 relevance scoring via tantivy with zero model weights.
Manifest TOC: Auto-generated table of contents with per-file token counts.

⚙️ Development & Asset Pipeline

filegoblin maintains a strict zero-dependency profile for the end user. However, building the project requires assembling several logic assets.

build.rs generates the manpage and shell completions into target/generated/. The .rten tensor weights for pure-Rust OCR fallback on Linux/Windows are not downloaded automatically — fetch them with the provided justfile when building with --features ocr:

just get-brains

(On macOS, OCR uses Apple's Vision framework natively; no weights needed.)

🚀 Quick Start

Basic Gobble (Files):

# Output a single file to stdout
filegoblin my_notes.pdf > context.md

# Merge multiple files natively
filegoblin src/main.rs src/lib.rs README.md --write context.md

URL Ingestion (Web):

filegoblin https://example.com/api-docs > context.md

Web Horde (Split into mapping directory):

filegoblin https://bettercli.org/ --horde --split
# Translates to -> ./bettercli.org_gobbled/

Privacy Shield (Redact PII & Secrets locally):

# Typed redaction (default): values become [EMAIL], [AWS_KEY], [CREDIT_CARD], …
filegoblin ./src/api_keys.ts --scrub > safe_context.md

# Pipe a log through it as a pre-flight gate before any cloud LLM
cat prod.log | filegoblin - --scrub --scrub-style plain

# CI / pre-commit guard: non-zero exit if anything sensitive is found
git ls-files -z | xargs -0 cat | filegoblin - --scrub --scrub-fail -q || echo "secrets detected!"

# Exempt a known-public value, or add your own pattern via ./.filegoblin-scrub.json
filegoblin ./config --horde --scrub --scrub-allow "support@example.com"

# Consistent pseudonyms: the same value maps to [EMAIL_1] everywhere, so an LLM can
# still tell that an entity recurs — without ever seeing it.
cat ticket.txt | filegoblin - --scrub --scrub-style pseudonym

Detection is 100% local with zero model weights: validated rules (Luhn for cards, range checks for SSN, mod-97 for IBAN) plus cloud/token prefixes (AWS, GitHub, Stripe, Slack, Google, OpenAI, JWT, private-key blocks, connection strings) and a Shannon-entropy catch-all for unknown secrets. A value-free audit tally is printed to stderr (🛡️ Privacy Shield redacted 4 item(s): …). An optional .filegoblin-scrub.json supplies an allowlist and custom patterns.

What filegoblin does NOT do: maintain a secrets blocklist. It converts exactly what you point it at — including a .env file if you name one. Deciding what's sensitive is yours: the directory walker respects .gitignore (where secrets usually already live), --exclude filters by glob, and --scrub redacts what does get through.

Machine Contract (--json) — extract any file to clean text for agents/scripts:

# One weird binary format → clean text + metadata, in a single call
filegoblin report.pdf --json | jq '.[0] | {format, tokens}'
# → { "format": "pdf", "tokens": 1234 }

filegoblin ./src/ --horde -q --json | jq '.[].path'

--json emits a deterministic, alphabetically-sorted array. A successful record is { "path", "format", "tokens", "content" } (format is a coarse type like pdf, spreadsheet, code, image, text; tokens is an LLM token estimate). A file that fails to parse appears in-band as { "path", "format", "error" } rather than silently vanishing — so an agent can trust that every input is accounted for. Exit codes: 0 success, 4 when nothing could be extracted (empty result vs. total failure), 3 when --scrub --scrub-fail redacted something.

Multi-Format Output (--format markdown|plain|json / -f):

# Default GFM markdown with tables, math, and code fences
filegoblin document.pdf -f markdown > doc.md

# Strip markdown punctuation for dense vector embeddings & BM25 search
# (Pipe tables become space-aligned text columns; headings & math stripped of tokens)
filegoblin document.pdf -f plain > embedding_input.txt

# Structured JSON document array for agent pipelines
filegoblin document.pdf -f json > document.json

Layout-Aware Semantic Chunking (--chunk-size, --chunk-overlap, --chunk-strategy):

# Split along structural boundaries (H1/H2/H3, tables, code blocks) with 512 token budget
# Replicates table headers across chunk boundaries and embeds hierarchical breadcrumbs
filegoblin document.pdf --chunk-size 512 > chunks.md

# Structured JSON chunks with breadcrumbs and token estimates for RAG vectors
filegoblin document.pdf -f json --chunk-size 512 --chunk-overlap 64 > rag_chunks.json

Token Compression (Reduce LLM context sizes):

# Strip code comments, minify JSON/HTML, collapse whitespace
filegoblin ./src/main.rs --compress contextual > context.md

Multi-Part Splitting (Chunk output by tokens):

# Automatically break massive repositories into `partN` files at a 50k token threshold
filegoblin ./src/ --horde --chunk 50k

Pipe from stdin:

curl -s "https://api.github.com/users/octocat" | filegoblin -q --json

Clipboard Ingestion:

# Read contents directly from your OS clipboard
filegoblin --clipboard > context.md

Development & Intelligence:

# Get a heuristic overview of the project with token counts
filegoblin ./src/ --summary --tokens

Filtering & Exclusion:

# Only Rust files, excluding tests and generated code
filegoblin ./src/ --horde --include "*.rs" --exclude "*test*" --exclude "*generated*"

# Limit crawl depth to top-level files only
filegoblin ./src/ --horde --depth 1

Manifest / Table of Contents:

# Prepend a TOC with file paths and token counts
filegoblin ./src/ --horde --include "*.rs" --manifest

Token-Only Mode (Scripting):

# Print just the token count, no content — ideal for CI/scripts
filegoblin ./src/ --horde --tokens-only

Semantic Search with Relevance Scores:

# Search across a codebase and see BM25 relevance scores
filegoblin ./src/ --horde --search "authentication"

Git-Diff Mode (Only Changed Files):

# Ingest only files changed since last commit
filegoblin . --horde --git-diff HEAD~1

# Show unified diffs instead of full file contents
filegoblin . --horde --git-diff HEAD~1 --diff-format

Scoping a crawl (--depth, --include, --exclude):

These apply to both halves of --horde — local file paths, and the URL path when crawling. Out-of-scope pages are never fetched, so scoping saves requests rather than just trimming the output.

# Only the guide section, and never the changelog
filegoblin https://docs.example.com --horde --include "/guide/*" --exclude "*changelog*"

# The seed page and whatever it links to directly — nothing further
filegoblin https://docs.example.com --horde --depth 1

# Shorthand: a bare word matches anywhere in the path
filegoblin https://docs.example.com --horde --include guide

The seed URL is always fetched, so --include "/guide/*" starting at / still works. --depth 0 is the seed alone.

Walk and pick (--walk) — for when you don't want the whole site:

Pointing --horde at a website converts everything it can reach: the changelog, the 2019 blog archive, every language variant. --walk opens a picker instead — read each page, follow its links, mark what you want, convert only that.

filegoblin --walk https://docs.example.com > context.md
filegoblin --walk ./my-repo > context.md

 https://docs.example.com/guide/     8.4k tok  ↳1
 # Getting Started

 Install the CLI, then point it at a file. Every
 conversion happens locally on your machine.

 ─ pick ──────────────────────────────────────────
 ❯ ◉ this page — Getting Started
   ○ Next: Configuration
   ○ API Reference

  1 selected   ↑↓ move · space keep · ⏎ follow · b back · d done · q quit

The preview is the actual converted markdown, so you're judging what you'd get, not a guess. Links are scoped to the same origin (host and port), and selecting a directory includes everything beneath it. Pages you revisit come from the conversion cache, so backing up is instant.

The picker draws on stderr and the markdown goes to stdout, so redirecting works normally. It needs a terminal — piping without one is a clear error rather than a hang, and every other invocation is unchanged, so scripts, agents and the MCP server are unaffected.

MCP Server (Claude Desktop / Claude Code):

filegoblin --mcp speaks the Model Context Protocol over stdio, exposing a single convert tool that takes a file path, directory, or URL.

For Claude Code:

claude mcp add filegoblin -- filegoblin --mcp

For Claude Desktop, add this to claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\):

{
  "mcpServers": {
    "filegoblin": {
      "command": "filegoblin",
      "args": ["--mcp"]
    }
  }
}

Use an absolute path to the binary if it isn't on the PATH your client sees. To let the tool reach services on your own machine, add "env": { "FILEGOBLIN_ALLOW_LOCAL_URLS": "1" } — see the security note below for why that is off by default.

🗺️ Roadmap

Unified Output Formats (--format):
- --format markdown / --format md (default): Clean, token-optimized Markdown for LLM context windows.
- --format plain (or --plain): Syntax-stripped prose for text-embedding models (OpenAI, Cohere, Voyage) and token counters.
- --format json (or --json): Structured AST array of typed semantic blocks (heading, paragraph, table, code, math, callout) with bounding box and page metadata for RAG ingestors.
Semantic Section Chunking (--chunk-tokens, --chunk-by):
- Chunk documents into LLM context-ready token budgets (--chunk-tokens 512) without severing tables, code blocks, or mathematical equations across chunk boundaries.
- Boundary-aware splitting (--chunk-by section|chapter|page).
Multi-Channel Distribution:
- PyPI (pip install filegoblin): Python wheels built via Maturin for instant drop-in use in LangChain, LlamaIndex, and PyTorch pipelines.
- Crates.io (cargo install filegoblin): Publish CLI binary and filegoblin-core library crate.
- Homebrew Tap (brew install funkyfunc/tap/filegoblin): Zero-dependency native macOS installation.
- Precompiled Binary Releases: Automated cross-platform release archives (Linux x64/arm64, macOS Apple Silicon/Intel, Windows x64) via cargo-dist.

🔒 Security notes

URLs pointing at private addresses are refused. Loopback, private ranges, link-local (including the cloud metadata endpoint 169.254.169.254), and any hostname that resolves to one are blocked — on the initial URL, on every redirect hop, and inside the DNS resolver itself, so a name that resolves differently between the check and the connection (DNS rebinding) is refused too. Set FILEGOBLIN_ALLOW_LOCAL_URLS=1 to allow them — reasonable for converting your own dev server, and worth thinking about before enabling it for the MCP server, where the URL comes from a model.
--browser is a wider surface. Chrome does not honour those redirect guards, so a page it renders can redirect somewhere filegoblin would refuse to fetch directly. The initial URL is still validated, the profile is a throwaway temp dir, and the mode is strictly opt-in.
Archives and Office files have decompression caps. Zip, tar.gz, and the zip-container formats (.docx/.pptx/.odt/.epub) bound what they inflate and refuse entries whose paths escape the extraction directory.

🛠 Contributing

All code must pass the ritual before being merged:

cargo horde-check   # clippy -D warnings + fmt --check + test, via .cargo/config.toml

Golden-file tests live in crates/cli/tests/ and use insta snapshots; run cargo insta review to inspect a diff before accepting it.

🎨 Vibe & Aesthetics

filegoblin embraces the Mischievous Librarian archetype. We don't "process" data; we feast on it.

Navigation Menu

Repository navigation

About

Repository files navigation

(o_o) filegoblin (filegoblin)

Philosophy

📦 Installation

🏗 Current Project Structure

⚡ Current Capabilities (v1.15.0)

✨ Key Features

⚙️ Development & Asset Pipeline

🚀 Quick Start

🗺️ Roadmap

🔒 Security notes

🛠 Contributing

🎨 Vibe & Aesthetics

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.cargo		.cargo
.githooks		.githooks
.github/workflows		.github/workflows
crates		crates
docs		docs
examples		examples
npm		npm
packaging		packaging
scripts		scripts
test_assets		test_assets
.gitignore		.gitignore
BENCHMARK_REPORT.md		BENCHMARK_REPORT.md
CLAUDE.md		CLAUDE.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
GAUNTLET_SCORECARD.md		GAUNTLET_SCORECARD.md
GEMINI.md		GEMINI.md
LICENSE		LICENSE
README.md		README.md
dist-workspace.toml		dist-workspace.toml
justfile		justfile

Latest commit

History

Folders and files

Repository files navigation

(o_o) filegoblin (filegoblin)

Philosophy

📦 Installation

🏗 Current Project Structure

⚡ Current Capabilities (v1.15.0)

✨ Key Features

⚙️ Development & Asset Pipeline

🚀 Quick Start

🗺️ Roadmap

🔒 Security notes

🛠 Contributing

🎨 Vibe & Aesthetics

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages