pub struct Ingestor { /* private fields */ }Expand description
Chunks, embeds (in batches) and stores documents.
Chunk ids are content-addressed per document (dataset:document:hash), so
re-ingesting a changed document embeds only new text, keeps unchanged chunks
(refreshing their position and metadata), and — with
Ingestor::replace_existing — removes chunks the new version no longer
contains. The swap is ordered so a failure never loses the previous version:
everything is embedded first, then upserted, and only then are stale chunks
deleted. Identical text in two different documents is stored once per
document, so deleting one document never removes content from another.
Implementations§
Source§impl Ingestor
impl Ingestor
pub fn new(store: Arc<dyn VectorStore>, embedder: Arc<dyn Embedder>) -> Self
pub fn chunker(self, config: ChunkerConfig) -> Self
Sourcepub fn batch_size(self, size: usize) -> Self
pub fn batch_size(self, size: usize) -> Self
Texts per embedding request (default 32).
Sourcepub fn replace_existing(self, replace: bool) -> Self
pub fn replace_existing(self, replace: bool) -> Self
Remove chunks of the previous version that the new version no longer
contains (default true). With false, ingestion only adds chunks.