Crate polyglot_tokenizer

Re-exports

pub use tokenizer::Token;

pub use tokenizer::Tokenizer;

Modules

tokenizer

Functions

get_key_tokens

Tokenize the content and return only the identifiers and symbols from the langauge