Skip to main content

Crate cyberbrain_embed

Crate cyberbrain_embed 

Source
Expand description

Cyberbrain embed: static token embeddings in the model2vec format, implemented in-tree (SPEC §6).

Original work, copyright 2026 Krynex Labs, licensed FSL-1.1-ALv2. Written from docs/SPEC.md alone under the clean-room boundary in §0.

§What this crate does

A model2vec-format model is two files: a tokenizer.json (HuggingFace tokenizers format) and a model.safetensors holding one embedding matrix of shape [vocab, dim]. The whole forward pass is

  1. tokenise the text (no special tokens),
  2. look up one row per token id, skipping the unknown token,
  3. mean-pool the rows,
  4. L2-normalise.

There is no transformer. Producing a vector costs a few microseconds.

§What this crate never does

No network I/O of any kind. Model artefacts are loaded from a local path handed in by the caller (the policy crate’s registered ModelDownload egress path, SPEC §12.1). Every load verifies a blake3 hash of both files against a manifest and refuses on mismatch. There is no “verify later”, no “warn and continue” and no default download location.

§Degenerate inputs

An empty string, or a string whose every token is unknown to the model, has no rows to pool. Dividing by zero there would yield a NaN vector, and a NaN that reaches a cosine comparison silently corrupts every ranking it touches. Instead such inputs yield the all-zero vector and Embedding::tokens_known reports 0, so a caller can tell the case apart and log it (SPEC §14.5: a branch that declines to act says why). The zero vector has cosine 0 against everything, which ranks it last rather than randomly. is_zero is the check callers should use before running a semantic search with such a query.

Structs§

ArtefactManifest
The expected blake3 digests of both files, lowercase hex. Produced once by whoever obtained the artefact (the policy crate, after its registered download) and stored in configuration; checked on every load.
Description
A model’s identity without the model: see StaticEmbedder::describe.
Embedding
One embedded input. The vector is L2-normalised, or all zero when nothing was pooled.
LoadOptions
Knobs for loading. Default is what production uses.
ModelInfo
Technical facts about the loaded model, for status and the model card (SPEC §12.7). Source and licence are not known here; the policy crate’s registry carries those.
ModelPaths
Where the two files of a model2vec-format model live.
StaticEmbedder
A loaded model2vec-format model. Cheap to share behind an Arc; embed takes &self.

Constants§

EMBEDDINGS_TENSOR
The tensor name a model2vec artefact uses for its embedding matrix.
POOLING
The only pooling this crate implements. Part of every profile id, so a future pooling change cannot be confused with vectors produced by this one.
VERIFIED_FILE
Name of the verification record next to the model files.

Functions§

hash_bytes
blake3 of a byte slice, lowercase hex.
hash_file
blake3 of a file’s contents, lowercase hex. This is what the policy crate calls after a download to fill an ArtefactManifest.
is_zero
True when every component is exactly zero: the vector this crate returns for an empty or all-unknown input. Callers should skip semantic search for such a query instead of ranking on cosines that are all 0.