Struct edge_transformers::tokenizer::AutoTokenizer
source · pub struct AutoTokenizer {
pub tokenizer: Tokenizer,
pub eos_token: String,
pub pad_token: String,
pub eos_token_id: u32,
pub pad_token_id: u32,
}
Expand description
A simple tokenizer wrapper that reads special token map for further use in the pipeline.
Fields§
§tokenizer: Tokenizer
§eos_token: String
§pad_token: String
§eos_token_id: u32
§pad_token_id: u32