cyberbrain_embed/lib.rs
1//! Cyberbrain embed: static token embeddings in the model2vec format, implemented in-tree
2//! (SPEC §6).
3//!
4//! Original work, copyright 2026 Krynex Labs, licensed FSL-1.1-ALv2. Written from
5//! `docs/SPEC.md` alone under the clean-room boundary in §0.
6//!
7//! # What this crate does
8//!
9//! A model2vec-format model is two files: a `tokenizer.json` (HuggingFace tokenizers
10//! format) and a `model.safetensors` holding one embedding matrix of shape
11//! `[vocab, dim]`. The whole forward pass is
12//!
13//! 1. tokenise the text (no special tokens),
14//! 2. look up one row per token id, skipping the unknown token,
15//! 3. mean-pool the rows,
16//! 4. L2-normalise.
17//!
18//! There is no transformer. Producing a vector costs a few microseconds.
19//!
20//! # What this crate never does
21//!
22//! **No network I/O of any kind.** Model artefacts are loaded from a local path handed in by
23//! the caller (the policy crate's registered `ModelDownload` egress path, SPEC §12.1). Every
24//! load verifies a blake3 hash of both files against a manifest and refuses on mismatch.
25//! There is no "verify later", no "warn and continue" and no default download location.
26//!
27//! # Degenerate inputs
28//!
29//! An empty string, or a string whose every token is unknown to the model, has no rows to
30//! pool. Dividing by zero there would yield a NaN vector, and a NaN that reaches a cosine
31//! comparison silently corrupts every ranking it touches. Instead such inputs yield the
32//! **all-zero vector** and [`Embedding::tokens_known`] reports `0`, so a caller can tell the
33//! case apart and log it (SPEC §14.5: a branch that declines to act says why). The zero
34//! vector has cosine 0 against everything, which ranks it last rather than randomly.
35//! [`is_zero`] is the check callers should use before running a semantic search with such a
36//! query.
37
38mod artefact;
39mod model;
40mod pool;
41mod weights;
42
43#[cfg(any(test, feature = "synthetic"))]
44pub mod synthetic;
45
46#[cfg(test)]
47mod tests;
48
49pub use artefact::{ArtefactManifest, ModelPaths, VERIFIED_FILE, hash_bytes, hash_file};
50pub use model::{Description, Embedding, LoadOptions, ModelInfo, StaticEmbedder};
51pub use pool::is_zero;
52
53/// The only pooling this crate implements. Part of every profile id, so a future pooling
54/// change cannot be confused with vectors produced by this one.
55pub const POOLING: &str = "mean";
56
57/// The tensor name a model2vec artefact uses for its embedding matrix.
58pub const EMBEDDINGS_TENSOR: &str = "embeddings";