Struct SliceVocab 

pub struct SliceVocab { /* private fields */ }

The obvious TokenVocab over an owned id -> bytes table. Handy for tests and for any caller that already has the vocabulary as a Vec.

Implementations

impl SliceVocab

pub fn new(tokens: Vec<Vec<u8>>) -> Self

Wrap a Vec where index i is the byte string of token id i.

Trait Implementations

impl Clone for SliceVocab

fn clone(&self) -> SliceVocab

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SliceVocab

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TokenVocab for SliceVocab

fn token_count(&self) -> usize

How many token ids exist (0..token_count() is the id range).

fn token_bytes(&self, id: u32) -> Option<&[u8]>

The bytes token id decodes to, or None for an out-of-range / byte-less control id (e.g. EOS). A None token contributes no bytes to the grammar and is never structurally allowed by JsonStructure on its own — permit it explicitly via JsonStructure::with_always_allowed if it should be samplable (EOS usually should, so the model can actually stop).