[−][src]Crate blingfire
blingfire is a thin Rust wrapper for the BlingFire tokenization library.
Enums
| Error | Error enum encoding tokenization errors. |
Constants
| MAX_TEXT_LENGTH | The maximum valid size of the input text for the tokenizer functions. Re-exported from the C++ library. |
Functions
| text_to_sentences | Tokenizes a piece of text into sentences separated by whitespace. |
| text_to_words | Tokenizes a piece of text into words separated by whitespace. |
Type Definitions
| Result | Result of calling the tokenizer functions. |