Structs§
- Dictionary
- Dictionary holding vocabulary, tokenization logic, and open-addressing hash table.
- Entry
- A vocabulary entry.
Enums§
- Entry
Type - Word entry type.
Constants§
- BOW
- Beginning-of-word marker for subword computation.
- EOS
- EOS (end-of-sentence) token string.
- EOW
- End-of-word marker for subword computation.
- MAX_
LINE_ SIZE - Maximum tokens per line.
- MAX_
VOCAB_ SIZE - Maximum vocabulary size for the open-addressing hash table.