Crate n_gram

Modules§

default

Structs§

Config: Config for Model.
Model: N-gram language model.

Constants§

EOS
SOS

Functions§

eos: Adds EOS token at the end.
n_grams: Creates list of n-grams from tokens.
sos: Adds SOS token at the start.
tokenize: Splits text by whitespaces.