Struct BasicTokenizer 

pub struct BasicTokenizer { /* private fields */ }

A basic tokenizer that runs basic tokenization (punctuation splitting, lower casing, etc.). By default, it does not lower case the input.

Example

use bert_tokenizer::{BasicTokenizer, Tokenizer};
let tokenizer = BasicTokenizer::default();
let tokens = tokenizer.tokenize("Hello, World!");
assert_eq!(tokens, vec!["Hello", ",", "World", "!"]);

If you want to provide lower casing, you can use the do_lower_case method:

use bert_tokenizer::{BasicTokenizer, Tokenizer};
let tokenizer = BasicTokenizer::do_lower_case(true).build();
let tokens = tokenizer.tokenize("Hello, World!");
assert_eq!(tokens, vec!["hello", ",", "world", "!"]);

Implementations

impl BasicTokenizer

pub fn do_lower_case(do_lower_case: bool) -> BasicTokenizerBuilder

Trait Implementations

impl Default for BasicTokenizer

fn default() -> BasicTokenizer

Returns the “default value” for a type.

impl Tokenizer for BasicTokenizer

fn tokenize(&self, text: &str) -> Vec<String>

Apply basic tokenization (punctuation splitting, lower casing, etc.) to a piece of text.

Arguments

• text - Text to tokenize

Returns

• Vec<String> - Vector of tokens

Example

use bert_tokenizer::{BasicTokenizer, Tokenizer};

let tokenizer = BasicTokenizer::default();
let tokens = tokenizer.tokenize("Hello, World!");
assert_eq!(tokens, vec!["Hello", ",", "World", "!"]);