Lantern

Lantern is a local-first memory engine for agent activity.

It ingests text an agent has touched, chunks it deterministically, and keeps a full provenance trail — source URI, content hash, byte ranges, ingest time — alongside a BM25 keyword index. Everything lives in a single SQLite file under ./.lantern/ so it is easy to inspect, back up, or wipe by hand.

Thesis

Most memory tools are either chat-memory products, document search tools, or heavyweight agent frameworks. Lantern is narrower and more durable:

a local memory engine for agent activity with provenance-aware search

Provenance comes first. Every stored chunk can answer where it came from, when it was ingested, what exact byte range it covers, and why a search result surfaced it.

Install

Prebuilt Binaries

Static binaries are available from the GitLab Releases:

# Download and install (Linux amd64 example)
curl -L -o lantern https://git.skylantix.com/diogenes/lantern/-/releases/v0.3.0/downloads/lantern-linux-amd64
chmod +x lantern
sudo mv lantern /usr/local/bin/

All Linux binaries are fully static (musl) — no libc or OpenSSL dependency. SHA256 checksums are attached to each release.

Build from Source

Requires a recent Rust toolchain (2024 edition):

cargo build --release
./target/release/lantern --help

Commands

Every command that produces structured output accepts --format text or --format json; search additionally defaults to a compact summary mode.

Examples

Index a notes tree and search it

lantern init
lantern ingest notes/
lantern search "lantern bug tracker"

Capture an agent session transcript from stdin

cat session.jsonl | lantern ingest --stdin \
    --uri "session://2026-04-18-foo" --kind application/jsonl
lantern search haystack --kind application/jsonl

Stream session transcripts through a named pipe

mkfifo /tmp/lantern.pipe
# In one shell: the agent writes its transcript to the pipe between turns.
# In another: Lantern reads to EOF, ingests the batch, and is ready for the next.
lantern ingest /tmp/lantern.pipe

Lantern auto-detects the FIFO, reads until the writer closes, and routes the bytes through the stdin-append path. Each reader session lands as its own source under a fifo://<abs_path>#<suffix> URI, so repeated batches accumulate instead of overwriting. A .jsonl FIFO name still triggers the transcript extractor, preserving role / session / turn / tool metadata.

Watch a transcript directory for new sessions

lantern ingest ~/agent-sessions/ --follow --follow-interval-secs 5

Polling-based: Lantern re-scans the directory every interval and ingests any file whose content hash has changed. Unchanged files are a no-op, so this is cheap to leave running. Stop with Ctrl-C.

Drill into a single source

lantern inspect                 # copy a source id from the recent list
lantern show fd7e8e             # short prefix is enough

See what drifted since the last ingest

lantern diff notes/             # missing / changed / unchanged / unindexed

Snapshot the store before a risky change

lantern stash                   # writes .lantern/stashes/lantern-<ts>.tar.gz

.lantern-ignore

Lantern respects .lantern-ignore files for excluding paths from ingestion, similar to .gitignore. Place a .lantern-ignore file in the directory being ingested:

# Ignore build artifacts
target/
dist/
build/

# Ignore dependencies
node_modules/
.venv/
vendor/

# Ignore but keep one file
!important-logs/

# Ignore specific extensions
*.log
*.tmp

Pattern syntax:

• # — comments
• *, ?, ** — glob wildcards
• / suffix — match directories only
• ! prefix — negate (un-ignore)

Default ignores (applied when no .lantern-ignore exists): .git/, target/, node_modules/, .hermes/, __pycache__/, .venv/, vendor/

Use --no-ignore to bypass all ignore rules:

lantern ingest . --no-ignore

Data model

Two tables carry the indexed state; both are visible from sqlite3:

• sources — one row per ingested artifact. Keeps uri, optional filesystem path, kind (text/markdown, text/plain, application/jsonl), total bytes, content_sha256, and timestamps.
• chunks — one row per deterministic slice of a source. Keeps the parent source_id, ordinal, byte_start/byte_end, char_count, chunk text, and chunk sha256.

A shadow FTS5 virtual table (chunks_fts) is kept in sync by triggers and supplies BM25 ranking and snippet highlighting to search.

Development

cargo fmt
cargo check --all-targets
cargo test --all-targets
cargo run -- --help

Status

Early but usable. The CLI is stable, the schema versions its migrations (now through v18), and every command has integration test coverage. Beyond the core retrieval surfaces — keyword search (FTS5 BM25), semantic search (Ollama embeddings with cosine similarity, auto-accelerated with sqlite-vec for the default unfiltered path and backfilled on upgrade), hybrid search, an opt-in --vec-semantic path, and an MCP server — Lantern now exposes a typed entity graph (URLs, repos, domains, emails, file paths, @mentions, #hashtags) with neighbor and session-neighbor traversal, session-scoped retrieval with related and temporal session views, first-class memory records, confidence signals (feedback, query-success, access decay) surfaced in show / inspect / export, and JSONL ingest that preserves tool-call lineage across Anthropic, OpenAI, and Responses transcripts. Ingestion supports .lantern-ignore for excluding build artifacts and dependencies.

License

Lantern is licensed under the GNU Affero General Public License v3.0 only (AGPL-3.0-only).

Copyright (C) 2026 Raphael Bitton

See LICENSE.