Skip to main content

Crate valise

Crate valise 

Source
Expand description

Retrieval in one file.

Your RAG prototype works. Now ship it — and the corpus stops being a thing you have and becomes a thing you operate: a vector-database container, an index directory that must travel with the documents and stay consistent with them, a rebuild step in CI. The most valuable artifact you built is the one piece you cannot hand to anyone.

SQLite solved this for relational data. Valise does it for retrieval — documents, a BM25 index, compressed vectors, and the schema describing them, in one .vls file you copy, version, and query in place.

use valise::prelude::*;
let store = Store::open(&path)?;              // open-or-create
store.collection("notes", Schema::new().text("body"))?;

let mut w = store.writer();
w.put("notes", "doc-1", Record::new().text("body", "portable retrieval capsules"))?;
w.commit()?;                                  // the durability point

let hits = store.search("notes", Search::new().text("body", "retrieval").top_k(10))?;
assert_eq!(hits.len(), 1);

The schema is stored in the file. A later run, another process, or another machine calls db::Store::open and searches immediately — nothing to re-declare and nothing to migrate.

§There is no index to build

Every ANN library builds a graph before it answers anything, and pays again whenever the corpus changes. Valise builds nothing. Vectors are stored only as compact quantized codes, and the candidate-search structure — a one-bit-per-dimension sign sketch — is derived from those same codes when the file is opened. It lives in memory; nothing beyond the codes is ever written.

Measured at 100k vectors × 768d against recall-matched baselines: 0.50 s to ingest and commit, versus 50 s for FAISS HNSW, 91 s for USearch, and 139 s for hnswlib — 90–310× faster to build, in 5–6× fewer bytes, with no graph on disk to rebuild, invalidate, or corrupt independently of the data.

The bill is that the candidate scan is linear in corpus size, so a mature graph answers individual queries faster. Prefer Valise when the corpus fits your latency budget through a scan — roughly a million vectors per millisecond at d=768 — or at any scale when it is copied, shipped, or rebuilt more often than about once per ten thousand queries. Below ~256 dimensions a tuned index wins outright.

It is not a hosted vector database or a distributed search cluster, and it does not generate embeddings — bring your own model.

§Two API levels

  • prelude / db — the application layer. Start here. db::Store gives schema-by-name, keyed records, text / vector / hybrid search, partitions, and compaction. It is the surface the Python bindings consume.
  • ValiseFile / Database — the engine. Explicit catalog registration (codecs, embedding spaces, analyzers, text spaces), frame and vector primitives, raw introspection. Reach for it when extending the format or embedding Valise inside another storage system, not when writing applications.

Add a vector field and query both channels; with two or more channels they are fused with reciprocal-rank fusion by default.

use valise::prelude::*;
let store = Store::open("kb.vls")?;
store.collection(
    "notes",
    Schema::new()
        .text("body")                            // English BM25 by default
        .vector("dense", Vector::dim(768)),      // cosine + auto-calibrated QAM
)?;

let mut w = store.writer();
w.put(
    "notes",
    "doc-1",
    Record::new()
        .text("body", "portable retrieval capsules")
        .vector("dense", &embedding),
)?;
w.commit()?;

let hits = store.search(
    "notes",
    Search::new()
        .text("body", "retrieval capsule")
        .vector("dense", &query)
        .fuse(Fusion::rrf(60))
        .top_k(10),
)?;

Valise owns its retrieval primitives rather than embedding an opaque search-engine blob: BM25, TF-IDF cosine, count cosine, approximate cosine variants, and Dice / overlap / containment set scorers over canonical term dictionaries and postings; QAM Lloyd-Max and UPQ polar vector codecs with NEON and AVX2 kernels. There is no persisted HNSW graph or IVF index — so nothing to rebuild, and nothing on disk but the capsule.

§Durability

db::Writer::commit is the only durability point. Commits are footer-rooted and atomic: after any crash a reader sees either the previous committed state or the new one, never a mixture. Segment payloads carry BLAKE3 checksums and the table of contents is self-checksummed; if the active footer is unusable, recovery scans the file and reinstates the newest footer that fully validates.

Concurrency is N readers and one writer, coordinated across processes through a region inside the file header. Database::writer blocks for the single writer slot; Database::try_writer returns None rather than waiting.

§Platform support

Linux and macOS, on x86-64 and aarch64. Windows is not supported yet — the commit protocol relies on positional file IO and fcntl OFD advisory locks, and the Windows equivalents are not implemented. Building there fails with an explicit message.

§Command line

cargo install valise also gives you a CLI for inspecting a capsule without writing any code:

valise info kb.vls                        # size, collections, record counts
valise search kb.vls notes "query" --top-k 5   # text search
valise get kb.vls notes doc-1             # one record
valise export kb.vls > kb.jsonl           # every record, as JSON Lines

§Further reading

The on-disk format, the commit and recovery protocol, the vector-search design, and the benchmark methodology are documented in the repository.

Re-exports§

pub use error::Error;
pub use error::Result;
pub use codec::CodecParams;
pub use codec::QamLloydMaxParams;
pub use codec::UpqDesignSource;
pub use codec::UpqParams;
pub use concurrency::connection::ReadConnection;
pub use concurrency::connection::WriteConnection;
pub use concurrency::database::Database;
pub use concurrency::snapshot::Snapshot;
pub use file::AutoPromote;
pub use file::CommitOutcome;
pub use file::CommitProfile;
pub use file::CreateOptions;
pub use file::EmbeddingSpaceSpec;
pub use file::HybridHit;
pub use file::HybridQuery;
pub use file::HybridTextChannel;
pub use file::HybridVectorChannel;
pub use file::IngestProfile;
pub use file::OpenMode;
pub use file::PutFrame;
pub use file::PutVector;
pub use file::QueryAlgorithm;
pub use file::QueryProfile;
pub use file::ReadVectorResult;
pub use file::Reconstruct;
pub use file::TextIndexBuildProfile;
pub use file::TextMode;
pub use file::TextQuery;
pub use file::TimeQuery;
pub use file::ValiseFile;
pub use file::VectorContract;
pub use file::VectorFidelity;
pub use file::VectorHit;
pub use file::VectorOpenProfile;
pub use file::VectorSearchQuery;
pub use file::VoteSearchTrace;
pub use file::last_query_profile;
pub use file::last_vector_open_profile;
pub use format::create_contract::CreateContractV1;
pub use format::dtype::Dtype;
pub use format::dtype::DtypeSet;
pub use format::catalog::AnalyzerDesc;
pub use format::catalog::CodecDesc;
pub use format::catalog::CodecFamily;
pub use format::catalog::EmbeddingSpaceDesc;
pub use format::catalog::FieldDesc;
pub use format::catalog::FieldSchemaDesc;
pub use format::catalog::FieldSource;
pub use format::catalog::FusionNormalization;
pub use format::catalog::FusionProfileDesc;
pub use format::catalog::IdfMode;
pub use format::catalog::IdfVariant;
pub use format::catalog::NormMode;
pub use format::catalog::PunctuationPolicy;
pub use format::catalog::RetrievalProfileDesc;
pub use format::catalog::RetrievalProfileParams;
pub use format::catalog::RetrievalProfileType;
pub use format::catalog::Stemming;
pub use format::catalog::StopwordsPolicy;
pub use format::catalog::TextSpaceDesc;
pub use format::catalog::TfMode;
pub use format::catalog::TokenSource;
pub use format::catalog::Tokenizer;
pub use format::catalog::UnicodeNormalization;
pub use format::catalog::VectorDesc;
pub use format::catalog::VectorMetric;
pub use format::catalog::VectorStatus;
pub use format::catalog::WeightSource;
pub use format::AnalyzerId;
pub use format::CodecId;
pub use format::CollectionId;
pub use format::EmbeddingSpaceId;
pub use format::FieldSchemaId;
pub use format::FrameId;
pub use format::FusionProfileId;
pub use format::RetrievalProfileId;
pub use format::SegmentId;
pub use format::TextSpaceId;
pub use format::VectorId;

Modules§

codec
Vector codecs (spec §14). Two production families: QamLloydMax (Lloyd-Max-trained polar codebook; QamSliding is its fast NEON scoring kernel) and Upq (unrestricted polar quantization, v2.4).
concurrency
Concurrency primitives for Valise.
db
Application-facing CRUD layer over the Valise engine.
error
file
High-level single-file Valise lifecycle.
format
On-disk Valise structures.
io
Low-level file I/O helpers.
prelude
Application-facing API in one import: use valise::prelude::*;.
retrieval
Query-time retrieval modules. Text retrieval (BM25, TF-IDF, exact term-Jaccard) over the canonical primitives produced by the indexing pipeline, plus the QAM sign-sketch scan + QAM-sliding rerank for vectors (see sketch).
text
Text retrieval layer (spec §10–§13).

Constants§

VERSION