1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
//! **Retrieval in one file.**
//!
//! Your RAG prototype works. Now ship it — and the corpus stops being a
//! thing you have and becomes a thing you *operate*: a vector-database
//! container, an index directory that must travel with the documents and
//! stay consistent with them, a rebuild step in CI. The most valuable
//! artifact you built is the one piece you cannot hand to anyone.
//!
//! SQLite solved this for relational data. Valise does it for retrieval —
//! documents, a BM25 index, compressed vectors, and the schema describing
//! them, in one `.vls` file you copy, version, and query in place.
//!
//! ```
//! use valise::prelude::*;
//! # fn main() -> Result<()> {
//! # let dir = std::env::temp_dir().join("valise_doc_intro");
//! # std::fs::create_dir_all(&dir)?;
//! # let path = dir.join("kb.vls");
//! # let _ = std::fs::remove_file(&path);
//! let store = Store::open(&path)?; // open-or-create
//! store.collection("notes", Schema::new().text("body"))?;
//!
//! let mut w = store.writer();
//! w.put("notes", "doc-1", Record::new().text("body", "portable retrieval capsules"))?;
//! w.commit()?; // the durability point
//!
//! let hits = store.search("notes", Search::new().text("body", "retrieval").top_k(10))?;
//! assert_eq!(hits.len(), 1);
//! # Ok(())
//! # }
//! # main().unwrap();
//! ```
//!
//! The schema is stored **in the file**. A later run, another process, or
//! another machine calls [`db::Store::open`] and searches immediately —
//! nothing to re-declare and nothing to migrate.
//!
//! # There is no index to build
//!
//! Every ANN library builds a graph before it answers anything, and pays
//! again whenever the corpus changes. Valise builds **nothing**. Vectors
//! are stored only as compact quantized codes, and the candidate-search
//! structure — a one-bit-per-dimension sign sketch — is derived from those
//! same codes when the file is opened. It lives in memory; nothing beyond
//! the codes is ever written.
//!
//! Measured at 100k vectors × 768d against recall-matched baselines:
//! **0.50 s** to ingest and commit, versus 50 s for FAISS HNSW, 91 s for
//! USearch, and 139 s for hnswlib — **90–310× faster to build**, in
//! **5–6× fewer bytes**, with no graph on disk to rebuild, invalidate, or
//! corrupt independently of the data.
//!
//! The bill is that the candidate scan is linear in corpus size, so a
//! mature graph answers individual queries faster. Prefer Valise when the
//! corpus fits your latency budget through a scan — roughly a million
//! vectors per millisecond at d=768 — or at any scale when it is copied,
//! shipped, or rebuilt more often than about once per ten thousand
//! queries. Below ~256 dimensions a tuned index wins outright.
//!
//! It is **not** a hosted vector database or a distributed search cluster,
//! and **it does not generate embeddings** — bring your own model.
//!
//! # Two API levels
//!
//! - **[`prelude`] / [`db`] — the application layer. Start here.**
//! [`db::Store`] gives schema-by-name, keyed records, text / vector /
//! hybrid search, partitions, and compaction. It is the surface the
//! Python bindings consume.
//! - **[`ValiseFile`] / [`Database`] — the engine.** Explicit catalog
//! registration (codecs, embedding spaces, analyzers, text spaces),
//! frame and vector primitives, raw introspection. Reach for it when
//! extending the format or embedding Valise inside another storage
//! system, not when writing applications.
//!
//! # Hybrid search
//!
//! Add a vector field and query both channels; with two or more channels
//! they are fused with reciprocal-rank fusion by default.
//!
//! ```no_run
//! use valise::prelude::*;
//! # fn main() -> Result<()> {
//! # let embedding: Vec<f32> = vec![0.0; 768];
//! # let query: Vec<f32> = vec![0.0; 768];
//! let store = Store::open("kb.vls")?;
//! store.collection(
//! "notes",
//! Schema::new()
//! .text("body") // English BM25 by default
//! .vector("dense", Vector::dim(768)), // cosine + auto-calibrated QAM
//! )?;
//!
//! let mut w = store.writer();
//! w.put(
//! "notes",
//! "doc-1",
//! Record::new()
//! .text("body", "portable retrieval capsules")
//! .vector("dense", &embedding),
//! )?;
//! w.commit()?;
//!
//! let hits = store.search(
//! "notes",
//! Search::new()
//! .text("body", "retrieval capsule")
//! .vector("dense", &query)
//! .fuse(Fusion::rrf(60))
//! .top_k(10),
//! )?;
//! # Ok(())
//! # }
//! ```
//!
//! Valise owns its retrieval primitives rather than embedding an opaque
//! search-engine blob: BM25, TF-IDF cosine, count cosine, approximate
//! cosine variants, and Dice / overlap / containment set scorers over
//! canonical term dictionaries and postings; QAM Lloyd-Max and UPQ polar
//! vector codecs with NEON and AVX2 kernels. There is no persisted HNSW
//! graph or IVF index — so nothing to rebuild, and nothing on disk but the
//! capsule.
//!
//! # Durability
//!
//! [`db::Writer::commit`] is the only durability point. Commits are
//! footer-rooted and atomic: after any crash a reader sees either the
//! previous committed state or the new one, never a mixture. Segment
//! payloads carry BLAKE3 checksums and the table of contents is
//! self-checksummed; if the active footer is unusable, recovery scans the
//! file and reinstates the newest footer that fully validates.
//!
//! Concurrency is N readers and one writer, coordinated across processes
//! through a region inside the file header. [`Database::writer`] blocks
//! for the single writer slot; [`Database::try_writer`] returns `None`
//! rather than waiting.
//!
//! # Platform support
//!
//! Linux and macOS, on x86-64 and aarch64. **Windows is not supported
//! yet** — the commit protocol relies on positional file IO and fcntl OFD
//! advisory locks, and the Windows equivalents are not implemented.
//! Building there fails with an explicit message.
//!
//! # Command line
//!
//! `cargo install valise` also gives you a CLI for inspecting a capsule
//! without writing any code:
//!
//! ```text
//! valise info kb.vls # size, collections, record counts
//! valise search kb.vls notes "query" --top-k 5 # text search
//! valise get kb.vls notes doc-1 # one record
//! valise export kb.vls > kb.jsonl # every record, as JSON Lines
//! ```
//!
//! # Further reading
//!
//! The on-disk format, the commit and recovery protocol, the vector-search
//! design, and the benchmark methodology are documented in the
//! [repository](https://github.com/white07S/valise).
// Fail with something a human can act on rather than a dozen "cannot find
// `unix` in `os`" errors from deep inside the IO layer.
compile_error!;
/// Opt-in profiling output for the `VALISE_*_PROFILE` env vars.
///
/// Writes straight to stderr on purpose: these are developer diagnostics
/// enabled by an env var, and routing them through `tracing` would make
/// them invisible unless the caller had installed a subscriber. The
/// crate otherwise denies `print_stderr`, so this macro is the single
/// sanctioned exception.
pub use prof_eprintln;
/// Application-facing API in one import: `use valise::prelude::*;`.
///
/// Re-exports the [`db`] Store layer plus the shared error types —
/// exactly the surface an application (or the Python bindings) needs to
/// open a store, declare collections, ingest records, and search.
// ---- shared error / version ----
pub use ;
pub const VERSION: &str = env!;
// ---- codec parameters (power-user surface; see docs/EXTENDING.md) ----
pub use CodecParams;
pub use QamLloydMaxParams;
pub use ;
// ---- engine: process-shared handle + snapshot reads ----
pub use ;
pub use Database;
pub use Snapshot;
// ---- engine: single-writer file API ----
pub use ;
pub use CreateContractV1;
pub use ;
// ---- engine: catalog descriptors + raw ids (schema registration) ----
pub use ;
pub use ;
// ---- bench-only adapters (cargo feature `bench`; not public API) ----
pub use QamLloydMaxBench;
pub use simd_bench;
pub use ;
pub use ;
pub use sketch_bench;