Trait mako::tokenization::hf_tokenizers::tokenizer::Decoder

pub trait Decoder {
    fn decode(&self, tokens: Vec<String>) -> Result<String>;
}

A Decoder has the responsibility to merge the given Vec<String> in a String.

Required methods

fn decode(&self, tokens: Vec<String>) -> Result<String>

Implementors

impl Decoder for DecoderWrapper

impl Decoder for BPEDecoder

impl Decoder for WordPiece

impl Decoder for ByteLevel

As a Decoder, ByteLevel is in charge of converting any byte-level characters to their unicode counterpart, before merging everything back into a single String.

impl Decoder for Metaspace