Expand description
Retrieval in one file.
Your RAG prototype works. Now ship it — and the corpus stops being a thing you have and becomes a thing you operate: a vector-database container, an index directory that must travel with the documents and stay consistent with them, a rebuild step in CI. The most valuable artifact you built is the one piece you cannot hand to anyone.
SQLite solved this for relational data. Valise does it for retrieval —
documents, a BM25 index, compressed vectors, and the schema describing
them, in one .vls file you copy, version, and query in place.
use valise::prelude::*;
let store = Store::open(&path)?; // open-or-create
store.collection("notes", Schema::new().text("body"))?;
let mut w = store.writer();
w.put("notes", "doc-1", Record::new().text("body", "portable retrieval capsules"))?;
w.commit()?; // the durability point
let hits = store.search("notes", Search::new().text("body", "retrieval").top_k(10))?;
assert_eq!(hits.len(), 1);The schema is stored in the file. A later run, another process, or
another machine calls db::Store::open and searches immediately —
nothing to re-declare and nothing to migrate.
§There is no index to build
Every ANN library builds a graph before it answers anything, and pays again whenever the corpus changes. Valise builds nothing. Vectors are stored only as compact quantized codes, and the candidate-search structure — a one-bit-per-dimension sign sketch — is derived from those same codes when the file is opened. It lives in memory; nothing beyond the codes is ever written.
Measured at 100k vectors × 768d against recall-matched baselines: 0.50 s to ingest and commit, versus 50 s for FAISS HNSW, 91 s for USearch, and 139 s for hnswlib — 90–310× faster to build, in 5–6× fewer bytes, with no graph on disk to rebuild, invalidate, or corrupt independently of the data.
The bill is that the candidate scan is linear in corpus size, so a mature graph answers individual queries faster. Prefer Valise when the corpus fits your latency budget through a scan — roughly a million vectors per millisecond at d=768 — or at any scale when it is copied, shipped, or rebuilt more often than about once per ten thousand queries. Below ~256 dimensions a tuned index wins outright.
It is not a hosted vector database or a distributed search cluster, and it does not generate embeddings — bring your own model.
§Two API levels
prelude/db— the application layer. Start here.db::Storegives schema-by-name, keyed records, text / vector / hybrid search, partitions, and compaction. It is the surface the Python bindings consume.ValiseFile/Database— the engine. Explicit catalog registration (codecs, embedding spaces, analyzers, text spaces), frame and vector primitives, raw introspection. Reach for it when extending the format or embedding Valise inside another storage system, not when writing applications.
§Hybrid search
Add a vector field and query both channels; with two or more channels they are fused with reciprocal-rank fusion by default.
use valise::prelude::*;
let store = Store::open("kb.vls")?;
store.collection(
"notes",
Schema::new()
.text("body") // English BM25 by default
.vector("dense", Vector::dim(768)), // cosine + auto-calibrated QAM
)?;
let mut w = store.writer();
w.put(
"notes",
"doc-1",
Record::new()
.text("body", "portable retrieval capsules")
.vector("dense", &embedding),
)?;
w.commit()?;
let hits = store.search(
"notes",
Search::new()
.text("body", "retrieval capsule")
.vector("dense", &query)
.fuse(Fusion::rrf(60))
.top_k(10),
)?;Valise owns its retrieval primitives rather than embedding an opaque search-engine blob: BM25, TF-IDF cosine, count cosine, approximate cosine variants, and Dice / overlap / containment set scorers over canonical term dictionaries and postings; QAM Lloyd-Max and UPQ polar vector codecs with NEON and AVX2 kernels. There is no persisted HNSW graph or IVF index — so nothing to rebuild, and nothing on disk but the capsule.
§Durability
db::Writer::commit is the only durability point. Commits are
footer-rooted and atomic: after any crash a reader sees either the
previous committed state or the new one, never a mixture. Segment
payloads carry BLAKE3 checksums and the table of contents is
self-checksummed; if the active footer is unusable, recovery scans the
file and reinstates the newest footer that fully validates.
Concurrency is N readers and one writer, coordinated across processes
through a region inside the file header. Database::writer blocks
for the single writer slot; Database::try_writer returns None
rather than waiting.
§Platform support
Linux and macOS, on x86-64 and aarch64. Windows is not supported yet — the commit protocol relies on positional file IO and fcntl OFD advisory locks, and the Windows equivalents are not implemented. Building there fails with an explicit message.
§Command line
cargo install valise also gives you a CLI for inspecting a capsule
without writing any code:
valise info kb.vls # size, collections, record counts
valise search kb.vls notes "query" --top-k 5 # text search
valise get kb.vls notes doc-1 # one record
valise export kb.vls > kb.jsonl # every record, as JSON Lines§Further reading
The on-disk format, the commit and recovery protocol, the vector-search design, and the benchmark methodology are documented in the repository.
Re-exports§
pub use error::Error;pub use error::Result;pub use codec::CodecParams;pub use codec::QamLloydMaxParams;pub use codec::UpqDesignSource;pub use codec::UpqParams;pub use concurrency::connection::ReadConnection;pub use concurrency::connection::WriteConnection;pub use concurrency::database::Database;pub use concurrency::snapshot::Snapshot;pub use file::AutoPromote;pub use file::CommitOutcome;pub use file::CommitProfile;pub use file::CreateOptions;pub use file::EmbeddingSpaceSpec;pub use file::HybridHit;pub use file::HybridQuery;pub use file::HybridTextChannel;pub use file::HybridVectorChannel;pub use file::IngestProfile;pub use file::OpenMode;pub use file::PutFrame;pub use file::PutVector;pub use file::QueryAlgorithm;pub use file::QueryProfile;pub use file::ReadVectorResult;pub use file::Reconstruct;pub use file::TextIndexBuildProfile;pub use file::TextMode;pub use file::TextQuery;pub use file::TimeQuery;pub use file::ValiseFile;pub use file::VectorContract;pub use file::VectorFidelity;pub use file::VectorHit;pub use file::VectorOpenProfile;pub use file::VectorSearchQuery;pub use file::VoteSearchTrace;pub use file::last_query_profile;pub use file::last_vector_open_profile;pub use format::create_contract::CreateContractV1;pub use format::dtype::Dtype;pub use format::dtype::DtypeSet;pub use format::catalog::AnalyzerDesc;pub use format::catalog::CodecDesc;pub use format::catalog::CodecFamily;pub use format::catalog::EmbeddingSpaceDesc;pub use format::catalog::FieldDesc;pub use format::catalog::FieldSchemaDesc;pub use format::catalog::FieldSource;pub use format::catalog::FusionNormalization;pub use format::catalog::FusionProfileDesc;pub use format::catalog::IdfMode;pub use format::catalog::IdfVariant;pub use format::catalog::NormMode;pub use format::catalog::PunctuationPolicy;pub use format::catalog::RetrievalProfileDesc;pub use format::catalog::RetrievalProfileParams;pub use format::catalog::RetrievalProfileType;pub use format::catalog::Stemming;pub use format::catalog::StopwordsPolicy;pub use format::catalog::TextSpaceDesc;pub use format::catalog::TfMode;pub use format::catalog::TokenSource;pub use format::catalog::Tokenizer;pub use format::catalog::UnicodeNormalization;pub use format::catalog::VectorDesc;pub use format::catalog::VectorMetric;pub use format::catalog::VectorStatus;pub use format::catalog::WeightSource;pub use format::AnalyzerId;pub use format::CodecId;pub use format::CollectionId;pub use format::EmbeddingSpaceId;pub use format::FieldSchemaId;pub use format::FrameId;pub use format::FusionProfileId;pub use format::RetrievalProfileId;pub use format::SegmentId;pub use format::TextSpaceId;pub use format::VectorId;
Modules§
- codec
- Vector codecs (spec §14). Two production families:
QamLloydMax(Lloyd-Max-trained polar codebook;QamSlidingis its fast NEON scoring kernel) andUpq(unrestricted polar quantization, v2.4). - concurrency
- Concurrency primitives for Valise.
- db
- Application-facing CRUD layer over the Valise engine.
- error
- file
- High-level single-file Valise lifecycle.
- format
- On-disk Valise structures.
- io
- Low-level file I/O helpers.
- prelude
- Application-facing API in one import:
use valise::prelude::*;. - retrieval
- Query-time retrieval modules. Text retrieval (BM25, TF-IDF, exact
term-Jaccard) over the canonical primitives produced by the indexing
pipeline, plus the QAM sign-sketch scan + QAM-sliding rerank for
vectors (see
sketch). - text
- Text retrieval layer (spec §10–§13).