Episteme

Episteme is a privacy-preserving, local-first document ingestion pipeline for an Obsidian vault. It uses deterministic extraction tools, typed BAML functions backed by explicitly configured local model endpoints, atomic note persistence, DuckDB provenance, and zk indexing.

Install the episteme CLI from crates.io with:

cargo install episteme-local

The package is named episteme-local; the installed executable and library crate remain episteme.

Current slice

Supported inputs:

• Text and scanned PDFs through Poppler and Tesseract
• HTML through Pandoc
• PNG, JPEG, TIFF, and WebP images through Tesseract

The pipeline creates status: unprocessed research notes and copies immutable originals into the archive only after a note has been persisted successfully. The inbox copy remains until explicit cleanup. Episteme does not publish, delete user-owned source files, or use cloud model fallbacks.

Current security boundary

This slice accepts locally trusted documents only. It uses immutable staging copies, shell-free bounded subprocesses, source quote verification, inert Markdown rendering, and symlink-aware vault paths. Poppler, Tesseract, and Pandoc are not yet executed inside a networkless OS sandbox, so automatic public downloads remain disabled.

Configure

Install Poppler, Tesseract, Pandoc, and zk, then ensure a local OpenAI-compatible model endpoint is running on an explicit loopback address. On macOS with Homebrew:

brew install poppler tesseract pandoc zk

Create episteme.toml from the packaged example or repository example. Replace every /path/to value and tool path with an absolute local path. Set both model sections to existing OpenAI-compatible endpoints bound to explicit loopback IP addresses. Episteme requires those services to be managed externally. The example uses dedicated port 18181 and intentionally avoids Ollama's conventional 11434 port.

Use

episteme --config episteme.toml doctor
episteme --config episteme.toml init
episteme --config episteme.toml classify "/path/inside/configured/inbox/document.pdf"
episteme --config episteme.toml ingest "/path/inside/configured/inbox/document.pdf"
episteme --config episteme.toml watch

classify extracts one inbox document, persists typed metadata in rebuildable DuckDB state, and prints the stored record as JSON. It does not create a note, archive the source, or refresh zk. Repeated classification of unchanged content reuses the digest-matched record.

watch requires two unchanged observations before processing a file. Manual and watched ingestion use the same recoverable application service.

Research distillation automatically uses typed map/reduce for extracted documents over 12,000 characters. Rust assigns stable IDs to exact source spans; model calls select IDs rather than generating quotes, and Rust reconstructs verbatim evidence and character locations. The final aggregate must pass the same source-grounding validation before Episteme writes to the vault.

Verify

just ci

The ignored live-test guard can be run explicitly without touching the real vault:

apfel --serve --host 127.0.0.1 --port 18181
just test-live-apfel

Apfel is managed separately and the recipe never uses or stops Ollama on port 11434.

Deferred roadmap

1. Private search across zk and msgvault
2. Decisions, commitments, claims, assumptions, and forecasts in the derived DuckDB index
3. Profile-scoped briefings and knowledge assurance
4. Governed automatic public research
5. Manifest-controlled Pandoc publication bundles
6. Dashboard, backup, restore, and declutter workflows