Struct tongrams::SimpleTrieArray

pub struct SimpleTrieArray { /* private fields */ }

Simple implementation of TrieArray with Vec<usize>.

Trait Implementations

impl Debug for SimpleTrieArray

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SimpleTrieArray

fn default() -> SimpleTrieArray

Returns the “default value” for a type.

impl TrieArray for SimpleTrieArray

fn token_id(&self, i: usize) -> usize

Gets the token id with a given index.

fn build(token_ids: Vec<usize>, pointers: Vec<usize>) -> Box<Self>

Builds a TrieArray from sequences of token ids and pointers.

fn serialize_into<W>(&self, writer: W) -> Result<usize> where
    W: Write, 

Serializes the data structure into the writer.

fn deserialize_from<R>(reader: R) -> Result<Box<Self>> where
    R: Read, 

Deserializes the data structure from the reader.

fn size_in_bytes(&self) -> usize

Gets the number of bytes to serialize the data structure.

fn memory_statistics(&self) -> Value

Gets breakdowns of memory usages for components.

fn range(&self, pos: usize) -> (usize, usize)

Gets the range pointers[pos]..pointers[pos+1].

fn find_token(&self, pos: usize, id: usize) -> Option<usize>

Finds the position i such that token_id(i) = id and i in range(pos).

fn num_tokens(&self) -> usize

Gets the number of tokens stored.

fn num_pointers(&self) -> usize

Gets the number of pointers stored.