basemind 0.2.1

Full AI context layer over MCP — tree-sitter code-map, document RAG (PDF/Office/HTML/email + OCR + reranker), shared agent memory, on-demand web crawl, git history + blame + per-symbol diff. 300+ languages, 8 coding-agent harnesses, content-addressed Fjall + LanceDB.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

//! On-demand web crawl tier: kreuzcrawl → markdown → kreuzberg chunk + embed → LanceDB.
//!
//! Gated on `feature = "crawl"`. Lives under its own module so the network
//! surface area (kreuzcrawl, robots handling, scope tagging) stays inspectable
//! in one place. Mirrors the layout of `scanner_docs.rs` for the scanner doc
//! tier — `engine.rs` builds the shared crawler handle, `ingest.rs` is the
//! shared chunk + embed + LanceDB write path used by `web_scrape` / `web_crawl`.

#![cfg(feature = "crawl")]

pub mod engine;
pub mod ingest;

pub use engine::build_engine;
pub use ingest::{IndexedPage, index_page};