Struct EngramHash 

pub struct EngramHash {
    pub layer_ids: Vec<usize>,
    pub max_ngram: usize,
    pub n_heads: usize,
    pub compressed_vocab: usize,
    pub pad_id: i64,
    pub primes: Vec<Vec<Vec<u64>>>,
    pub offsets: Vec<Vec<u64>>,
    pub token_map: Vec<u32>,
    pub multipliers: Vec<[u64; 4]>,
    /* private fields */
}

The exact source layout of the V4.1 n-gram tables. The two configured Engram layers have disjoint prime ranges, even though both tables use the same nominal 16M vocabulary.

Fields

layer_ids: Vec<usize>

max_ngram: usize

n_heads: usize

compressed_vocab: usize

pad_id: i64

primes: Vec<Vec<Vec<u64>>>

offsets: Vec<Vec<u64>>

token_map: Vec<u32>

multipliers: Vec<[u64; 4]>

Implementations

impl EngramHash

pub fn new( layer_ids: Vec<usize>, max_ngram: usize, n_heads: usize, table_vocab: usize, compressed_vocab: usize, pad_token: usize, token_map: Vec<u32>, ) -> Result<Self, String>

pub fn reset(&mut self)

pub fn push(&mut self, token: u32, participates: bool) -> Vec<Vec<usize>>

Append one token and return one flattened hash column vector per configured layer. participates=false is used for image span tokens; dead tokens break every n-gram lookback just like the reference.

Trait Implementations

impl Clone for EngramHash

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.