Crate n_gram

Modules

• default

Structs

• Config
  Config for Model.
• Model
  N-gram language model.

Constants

• EOS
• SOS

Functions

• eos
  Adds EOS token at the end.
• n_grams
  Creates list of n-grams from tokens.
• sos
  Adds SOS token at the start.
• tokenize
  Splits text by whitespaces.