Crate blarse

Modules§

Token: A representation of a token. The body field points to the underlying string from which the characters in this token come. The indices field contains a range representing which contiguous characters are used. The tags field is a vector of borrowed strings with user-defined values.

TokenStructure: Represents the possible shapes of a vector of tokens. A vector of tokens could contain multiple tokens, a single token (a common special case), or none at all (a rare corner case).

empty_token: A token with no contents and a static body.
format_tags: Formats the given vector of strings in the fashion (a; b; c), where a, b, and c are debug-formatted tags (in order to see escape characters).
print_tokens: Prints a vector of tokens using their default Display method.
process_rule: Processes a rule across a vector of tokens. Starting from the first token, iteratively applies the rule on a single token. If that application returns None, continues applying the rule on the token and the next, then the next, and so on until the function returns Some or there are no tokens left to process. This process is explained more thoroughly with examples in the readme.
process_rule_maybe_verbose: See process_rule. May or may not print a message signifying when each replacement is made.
process_rules: Processes multiple rules on a vector of tokens. See process_rule.
str_to_tokens: Chops up a string slice into a vector of owned tokens. Also appends an empty token to the tail of the vector to enable certain lexing functions like scanning for words.
token_from_string: Consolidates a single string slice and a vector of tags into one token. Mostly used for debugging.
tokens_structure: Goes with the TokenStructure enum. Specifies whether a borrowed vector of tokens consists of multiple tokens, a single token, or none at all.
wrap: Wraps up a vector of consecutive tokens into one token and applies the specified tags.