text-tokenizer-0.4.1
- text-tokenizer 0.4.1
- Docs.rs crate page
- MIT
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- merl-twin
- Dependencies
- lazy_static ^1.1 normal
  
  regex ^1 normal
  
  unicode-segmentation ^1.2 normal
  
  unicode_categories ^0.1 normal
- Versions
- 0% of the crate is documented
Go to latest version
Platform
Feature flags

docs.rs

Rust

text_tokenizer0.4.1

All Items

Re-exports
Structs
Enums
Traits

Crate text_tokenizer

Re-exports

pub use emoji::EMOJIMAP;
pub use emoji::EMOJIMAP;

Structs

Breaker
CharBoundError
CharToken
EMOJIMAP
PositionalToken
Tokens

Enums

BasicToken
Formatter
Number
Numerical
Separator
Token
TokenizerOptions

Traits

IntoTokenizer
Tokenizer
UnicodeCategories