Struct TokenMapping 

pub struct TokenMapping { /* private fields */ }

Maps TØR-G tokens to LLM vocabulary token IDs.

TØR-G has 9 fixed tokens plus a range of Id tokens. This struct provides bidirectional mapping between TØR-G tokens and the corresponding token IDs in an LLM’s vocabulary.

Implementations

impl TokenMapping

pub fn builder() -> TokenMappingBuilder

Create a builder for custom token mappings.

pub fn sequential(id_count: u16) -> Self

Example mapping using sequential IDs starting from 0.

Warning: This is for testing only. In production, you must map to actual unused token IDs in your LLM’s vocabulary.

Layout:

• 0: Or
• 1: Nor
• 2: Xor
• 3: NodeStart
• 4: NodeEnd
• 5: InputDecl
• 6: OutputDecl
• 7: True
• 8: False
• 9..265: Id(0)..Id(255)

pub fn ministral() -> Self

Mapping for Mistral/Ministral models using reserved <SPECIAL_N> tokens.

Ministral tokenizers reserve token IDs 36-565 as <SPECIAL_N> placeholders. This mapping uses IDs 36-300 for TØR-G tokens:

TØR-G Token	Ministral ID
Or	36
Nor	37
Xor	38
NodeStart	39
NodeEnd	40
InputDecl	41
OutputDecl	42
True	43
False	44
Id(0)	45
Id(255)	300

Compatible with: Ministral-3B, Ministral-8B, Mistral-7B v0.3+

pub fn get(&self, token: Token) -> Option<u32>

Map a TØR-G token to its LLM vocabulary ID.

Returns None if the token cannot be mapped (e.g., Id out of range).

pub fn reverse(&self, id: u32) -> Option<Token>

Map an LLM vocabulary ID back to a TØR-G token.

Returns None if the ID doesn’t correspond to any mapped token.

pub fn id_count(&self) -> u16

Get the number of Id tokens mapped.

pub fn total_tokens(&self) -> usize

Get the total number of mapped tokens (9 fixed + id_count).

Trait Implementations

impl Clone for TokenMapping

fn clone(&self) -> TokenMapping

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenMapping

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenMapping

fn default() -> Self

Returns the “default value” for a type.

impl PartialEq for TokenMapping

fn eq(&self, other: &TokenMapping) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenMapping

impl StructuralPartialEq for TokenMapping