Skip to main content

detect_format_from_tokens

aprender::text::chat_template

Function detect_format_from_tokens 

Source 
pub fn detect_format_from_tokens(
    special_tokens: &SpecialTokens,
) -> TemplateFormat
Expand description

Auto-detect template format from special tokens

§Arguments

  • special_tokens - Special tokens from tokenizer

§Returns

Detected TemplateFormat