Skip to main content

static_translation_table

codec_rs::translator

Function static_translation_table 

Source 
pub fn static_translation_table(
    from_map: &TokenizerMap,
    to_map: &TokenizerMap,
) -> HashMap<u32, Vec<u32>>
Expand description

Build a static V_A → V_B[] translation table by rendering each V_A vocab entry to text and re-tokenizing through V_B.

Context-free: the result for a given source ID may differ from what Translator::translate produces when the same ID appears mid-sentence (BPE merges depend on context). Useful for analysis (vocab overlap, cost estimation) and as a fast lookup when context-free translation is acceptable.