text-tokenizer-0.6.4
- text-tokenizer 0.6.4
- Docs.rs crate page
- MIT
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- merl-twin
- Dependencies
- lazy_static ^1.5 normal
  text-parsing ^0.6 normal
  unicode-properties ^0.1 normal
  unicode-segmentation ^1.2 normal
- Versions
- 0% of the crate is documented
Go to latest version
Platform
Feature flags

docs.rs

Rust

text_tokenizer0.6.4

Crate Items

Structs
Enums
Traits

List of all items

Structs

EMOJIMAP
Text
TextLocality
TextStr
TextToken
TextTokens
TokenizerParams
Tokens
UnicodeSentenceBreaker

Enums

Bound
Error
Formatter
Number
Numerical
OriginalError
Separator
Special
Struct
Token
Token2
TokenizerOptions
Unicode
Word

Traits

IntoTokenizer
SentenceBreaker