Crate code_tokenizer

Code Tokenizer

A string/code tokenizer. Transforms a string with a vector of operators into a vector of keywords.

Functions

get_tokens

Creates a vector of tokens from a String and a vector of operators.