Struct AddedTokenSet 

pub struct AddedTokenSet { /* private fields */ }

An owned content → AddedToken set: what a backend is handed to build its matcher from.

A bare FxHashMap<String, u32> cannot express the strip flags, and every caller that has none (GGUF, the bundled vocabularies, tests) still gets to pass one — the From impls below turn it into a set of plain tokens — so only the tokenizer.json loader pays the cost of spelling the flags out.

Implementations

impl AddedTokenSet

pub fn new() -> Self

An empty set: no added token is recognized in the input.

pub fn insert(&mut self, content: impl Into<String>, token: AddedToken)

Declare content as an added token with explicit flags.

pub fn insert_plain(&mut self, content: impl Into<String>, id: u32)

Declare content as an added token with both strip flags off.

pub fn get(&self, content: &str) -> Option<AddedToken>

The token declared for content, if any.

pub fn len(&self) -> usize

Number of declared added tokens.

pub fn is_empty(&self) -> bool

Whether nothing is declared — matching is legitimately off.

pub fn iter(&self) -> impl Iterator<Item = (&str, AddedToken)> + '_

Iterate the declared tokens as (content, token) pairs.

pub fn into_id_map(self) -> FxHashMap<String, u32>

Consume the set into the plain name→id map the rest of the crate speaks (decode tables, SpecialPolicy’s named lookups). Consuming rather than borrowing moves the token strings instead of cloning every one of them.

Trait Implementations

impl Clone for AddedTokenSet

fn clone(&self) -> AddedTokenSet

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AddedTokenSet

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for AddedTokenSet

fn default() -> AddedTokenSet

Returns the “default value” for a type.

impl From<&HashMap<String, u32, FxBuildHasher>> for AddedTokenSet

fn from(map: &FxHashMap<String, u32>) -> Self

Converts to this type from the input type.

impl From<HashMap<String, u32, FxBuildHasher>> for AddedTokenSet

fn from(map: FxHashMap<String, u32>) -> Self

Converts to this type from the input type.

impl FromIterator<(String, AddedToken)> for AddedTokenSet

fn from_iter<T: IntoIterator<Item = (String, AddedToken)>>(iter: T) -> Self

Creates a value from an iterator.