Crate panko 

Enums

Token

Token represents a single lexical item.

Functions

fit_message

Split text into chunks of at most chunk_size UTF-8 characters.

normalize_text

split_tokens

Split tokens by any of the provided separator tokens. Each separator starts a new chunk; surrounding tokens are concatenated.

tokenize

tokens_to_string

Join tokens into a string.