Crate cyberbrain_embed 

Cyberbrain embed: static token embeddings in the model2vec format, implemented in-tree (SPEC §6).

Original work, copyright 2026 Krynex Labs, licensed FSL-1.1-ALv2. Written from docs/SPEC.md alone under the clean-room boundary in §0.

What this crate does

A model2vec-format model is two files: a tokenizer.json (HuggingFace tokenizers format) and a model.safetensors holding one embedding matrix of shape [vocab, dim]. The whole forward pass is

1. tokenise the text (no special tokens),
2. look up one row per token id, skipping the unknown token,
3. mean-pool the rows,
4. L2-normalise.

There is no transformer. Producing a vector costs a few microseconds.

What this crate never does

No network I/O of any kind. Model artefacts are loaded from a local path handed in by the caller (the policy crate’s registered ModelDownload egress path, SPEC §12.1). Every load verifies a blake3 hash of both files against a manifest and refuses on mismatch. There is no “verify later”, no “warn and continue” and no default download location.

Degenerate inputs

An empty string, or a string whose every token is unknown to the model, has no rows to pool. Dividing by zero there would yield a NaN vector, and a NaN that reaches a cosine comparison silently corrupts every ranking it touches. Instead such inputs yield the all-zero vector and Embedding::tokens_known reports 0, so a caller can tell the case apart and log it (SPEC §14.5: a branch that declines to act says why). The zero vector has cosine 0 against everything, which ranks it last rather than randomly. is_zero is the check callers should use before running a semantic search with such a query.

Structs

ArtefactManifest

The expected blake3 digests of both files, lowercase hex. Produced once by whoever obtained the artefact (the policy crate, after its registered download) and stored in configuration; checked on every load.

Embedding

One embedded input. The vector is L2-normalised, or all zero when nothing was pooled.

LoadOptions

Knobs for loading. Default is what production uses.

ModelInfo

Technical facts about the loaded model, for status and the model card (SPEC §12.7). Source and licence are not known here; the policy crate’s registry carries those.

ModelPaths

Where the two files of a model2vec-format model live.

StaticEmbedder

A loaded model2vec-format model. Cheap to share behind an Arc; embed takes &self.

Constants

EMBEDDINGS_TENSOR

The tensor name a model2vec artefact uses for its embedding matrix.

POOLING

The only pooling this crate implements. Part of every profile id, so a future pooling change cannot be confused with vectors produced by this one.

Functions

hash_bytes

blake3 of a byte slice, lowercase hex.

hash_file

blake3 of a file’s contents, lowercase hex. This is what the policy crate calls after a download to fill an ArtefactManifest.

is_zero

True when every component is exactly zero: the vector this crate returns for an empty or all-unknown input. Callers should skip semantic search for such a query instead of ranking on cosines that are all 0.