Module blarse::token

Structs

• Token
  A representation of a token. The body field points to the underlying string from which the characters in this token come. The indices field contains a range representing which contiguous characters are used. The tags field is a vector of borrowed strings with user-defined values.

Enums

• TokenStructure
  Represents the possible shapes of a vector of tokens. A vector of tokens could contain multiple tokens, a single token (a common special case), or none at all (a rare corner case).

Functions

• empty_token
  A token with no contents and a static body.
• print_tokens
  Prints a vector of tokens using their default Display method.
• str_to_tokens
  Chops up a string slice into a vector of owned tokens. Also appends an empty token to the tail of the vector to enable certain lexing functions like scanning for words.
• token_from_string
  Consolidates a single string slice and a vector of tags into one token. Mostly used for debugging.
• tokens_structure
  Goes with the TokenStructure enum. Specifies whether a borrowed vector of tokens consists of multiple tokens, a single token, or none at all.
• wrap
  Wraps up a vector of consecutive tokens into one token and applies the specified tags.