text-tokenizer-0.5.10
- text-tokenizer 0.5.10
- Permalink
- Docs.rs crate page
- MIT
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- merl-twin
- Dependencies
- lazy_static ^1.5 normal
  
  text-parsing ^0.6 normal
  
  unicode-properties ^0.1 normal
  
  unicode-segmentation ^1.2 normal
- Versions
- 0% of the crate is documented
Platform
Feature flags

Rust

text_tokenizer0.5.10

All Items

Crate Items

Re-exports
Structs
Enums
Traits

Crate text_tokenizer

Re-exports§

pub use emoji::EMOJIMAP;
pub use emoji::EMOJIMAP;

Structs§

EMOJIMAP
Text
TextStr
TextToken
TextTokens
TokenizerParams
Tokens
UnicodeSentenceBreaker

Enums§

Bound
Error
Formatter
Number
Numerical
OriginalError
Separator
Special
Struct
Token
Token2
TokenizerOptions
Unicode
Word

Traits§

IntoTokenizer
SentenceBreaker