language-tokenizer-0.1.0
- language-tokenizer 0.1.0
- Docs.rs crate page
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- savannstm
- Dependencies
- icu_segmenter ^2.1.2 normal optional
  itertools ^0.14.0 normal optional
  lindera ^1.5.1 normal optional
  num_enum ^0.7.5 normal
  serde ^1.0.228 normal optional
  strsim ^0.11.1 normal
  strum ^0.27.2 normal
  strum_macros ^0.27.2 normal
  thiserror ^2.0.17 normal
  unicode-normalization ^0.1.25 normal optional
  unicode-segmentation ^1.12.0 normal optional
  waken_snowball ^0.1.0 normal optional
- Versions
- 11.29% of the crate is documented
Platform
Feature flags

docs.rs

Rust

All

language_tokenizer0.1.0

Crate Items

Structs
Enums
Functions

List of all items

Structs

Token

Enums

Algorithm
Error
MatchMode
MatchResult

Functions

find_all_matches
find_match
tokenize