Crate code_tokenizer

Source
Expand description

§Code Tokenizer

A string/code tokenizer. Transforms a string with a vector of operators into a vector of keywords.

Functions§

get_tokens
Creates a vector of tokens from a String and a vector of operators.