Module markovish::token

At the heart of a Chain is a Token. In fact, this is just a String. But we make a distinction here: A Token is any atomic piece of text.

When using ChainBuilder::feed_str(), it is the output of unicode_segmentation::UnicodeSegmentation::split_word_bounds(); that is, it can be a word, a symbol like ", or something else. See that crate for more information.

If you want more control of what you want a token to be, you can use ChainBuilder::feed_tokens().

Structs

• TokenPair
  An owned pair of Tokens.

Type Aliases

• Token
  Representation of a string segment.
• TokenPairRef
  A borrowed version of TokenPair that does not own its pair. Like TokenRef to Token.
• TokenRef
  A borrowed version of Token; if Token is String, then TokenRef is &str.