step_decode_stream

tokenizers::tokenizer

Function step_decode_stream 

Source 
pub fn step_decode_stream<M, N, PT, PP, D>(
    tokenizer: &TokenizerImpl<M, N, PT, PP, D>,
    id: u32,
    skip_special_tokens: bool,
    ids: &mut Vec<u32>,
    prefix: &mut String,
    prefix_index: &mut usize,
    read_index: &mut usize,
) -> Result<Option<String>>
where
    M: Model,
    N: Normalizer,
    PT: PreTokenizer,
    PP: PostProcessor,
    D: Decoder,
Expand description

Internal function exposed only to bypass python limitations