Function tokenize_bytes

Source
pub fn tokenize_bytes(bytes: &[u8]) -> Result<Tokens<'_>, TokenizationError>
Expand description

Tokenize bytes to Tokens