Crate blarse

Modules

Token
A representation of a token. The body field points to the underlying string from which the characters in this token come. The indices field contains a range representing which contiguous characters are used. The tags field is a vector of borrowed strings with user-defined values.

TokenStructure
Represents the possible shapes of a vector of tokens. A vector of tokens could contain multiple tokens, a single token (a common special case), or none at all (a rare corner case).

empty_token
A token with no contents and a static body.
format_tags
Formats the given vector of strings in the fashion (a; b; c), where a, b, and c are debug-formatted tags (in order to see escape characters).
print_tokens
Prints a vector of tokens using their default Display method.
process_rule
Processes a rule across a vector of tokens. Starting from the first token, iteratively applies the rule on a single token. If that application returns None, continues applying the rule on the token and the next, then the next, and so on until the function returns Some or there are no tokens left to process. This process is explained more thoroughly with examples in the readme.
process_rule_maybe_verbose
See process_rule. May or may not print a message signifying when each replacement is made.
process_rules
Processes multiple rules on a vector of tokens. See process_rule.
str_to_tokens
Chops up a string slice into a vector of owned tokens. Also appends an empty token to the tail of the vector to enable certain lexing functions like scanning for words.
token_from_string
Consolidates a single string slice and a vector of tags into one token. Mostly used for debugging.
tokens_structure
Goes with the TokenStructure enum. Specifies whether a borrowed vector of tokens consists of multiple tokens, a single token, or none at all.
wrap
Wraps up a vector of consecutive tokens into one token and applies the specified tags.