Crate polyglot_tokenizer

Source

Re-exports§

pub use tokenizer::Token;
pub use tokenizer::Tokenizer;

Modules§

tokenizer

Functions§

get_key_tokens
Tokenize the content and return only the identifiers and symbols from the langauge