Struct Encoder 

pub struct Encoder { /* private fields */ }

Turns a set of token vectors into one vector.

Built from a token dimension, a Shape and a seed, and nothing else, so two processes that were told the same three things build the same encoder and the hyperplanes never have to be written down. Same rule as crate::Rotation, for the same reason.

Implementations

impl Encoder

pub fn new(dim: usize, shape: Shape, seed: u64) -> Encoder

An encoder for dim dimensional tokens.

Panics

If dim is zero, if ksim is not between 1 and 16, or if dproj or reps is zero.

pub fn dim(&self) -> usize

The token dimension this was built for.

pub fn shape(&self) -> Shape

The shape this was built with.

pub fn fde_dim(&self) -> usize

How long an encoding is, which is what to build the index at.

pub fn document(&self, tokens: &[f32]) -> Vec<f32>

Encode a document’s tokens, laid out end to end.

A bucket holds the average of the tokens that fell in it, and a bucket no token reached is filled from the tokens whose own bucket is nearest, so that every bucket has something to say. That filling is what lets a short document compete: a query token in a bucket the document never reached still has a best match in the document, and without the fill it would contribute nothing at all.

Panics

If tokens is empty or is not a whole number of Encoder::dim vectors.

pub fn query(&self, tokens: &[f32]) -> Vec<f32>

Encode a query’s tokens, laid out end to end.

A bucket holds the sum of the tokens that fell in it, not the average, because every query token is supposed to contribute its own best match to the score rather than share one. A bucket no query token reached is left at zero, because there is nothing there to ask for.

Panics

If tokens is empty or is not a whole number of Encoder::dim vectors.