Expand description
blingfire is a thin Rust wrapper for the BlingFire tokenization library.
Enums§
- Error
- Error enum encoding tokenization errors.
Constants§
- MAX_
TEXT_ LENGTH - The maximum valid size of the input text for the tokenizer functions. Re-exported from the C++ library.
Functions§
- text_
to_ sentences - Tokenizes a piece of text into sentences separated by whitespace.
- text_
to_ words - Tokenizes a piece of text into words separated by whitespace.
Type Aliases§
- Result
- Result of calling the tokenizer functions.