Module token 

Structs

Token

A representation of a token. The body field points to the underlying string from which the characters in this token come. The indices field contains a range representing which contiguous characters are used. The tags field is a vector of borrowed strings with user-defined values.

Enums

TokenStructure

Represents the possible shapes of a vector of tokens. A vector of tokens could contain multiple tokens, a single token (a common special case), or none at all (a rare corner case).

Functions

empty_token

A token with no contents and a static body.

print_tokens

Prints a vector of tokens using their default Display method.

str_to_tokens

Chops up a string slice into a vector of owned tokens. Also appends an empty token to the tail of the vector to enable certain lexing functions like scanning for words.

token_from_string

Consolidates a single string slice and a vector of tags into one token. Mostly used for debugging.

tokens_structure

Goes with the TokenStructure enum. Specifies whether a borrowed vector of tokens consists of multiple tokens, a single token, or none at all.

wrap

Wraps up a vector of consecutive tokens into one token and applies the specified tags.