Skip to main content

load_from_hub

ferrum_tokenizer

Function load_from_hub 

Source 
pub async fn load_from_hub(
    repo_id: &str,
    revision: Option<&str>,
) -> Result<Box<dyn Tokenizer>>
Expand description

Load tokenizer from HuggingFace Hub