infino 0.1.0

A fast retrieval engine that stores data on object storage and runs SQL, full-text search, and vector search over it from a single system — search-on-Parquet.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The Infino Authors

//! Superfile module — the production-grade implementation of the embedded
//! BM25 + vector format.
//!
//! See `docs/architecture/superfile.md` for the live design reference.
//!
//! ## In-tree caller invariant
//!
//! The only in-tree caller of `SuperfileBuilder` /
//! `SuperfileReader` is the `supertable` layer. The
//! supertable owns the multi-superfile + manifest + storage
//! policy; each rayon shard worker uses `SuperfileBuilder`
//! one-shot (one `add_batch` loop → one `finish()`), and
//! each cached / opened superfile runs through
//! `SuperfileReader::open` once per cache hydration. The
//! builder is consume-on-`finish()`; a session that wants N
//! superfiles instantiates N builders.

pub mod builder;
pub mod error;
pub mod format;
pub mod fts;
pub mod lazy_source;
pub mod reader;
pub mod stats;
pub mod vector;

pub use error::{BuildError, FtsError, ReadError, VectorError};
pub use lazy_source::{BytesLazyByteSource, LazyByteSource, LazyByteSourceError};
pub(crate) use lazy_source::{LazySubSource, PrefetchedSource};
pub use reader::{OpenOptions, SuperfileReader, VectorSearchOptions};