textprep-0.1.4
- textprep 0.1.4
- Docs.rs crate page
- MIT OR Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- arclabs561
- Dependencies
- aho-corasick ^1.1 normal
  serde ^1.0 normal optional
  unicode-casefold ^0.2 normal optional
  unicode-normalization ^0.1 normal
  unicode-segmentation ^1.10 normal
  criterion ^0.7 dev
  gramdex ^0.1 dev
  proptest ^1.9 dev
  serde_json ^1.0 dev
- Versions
- 62.92% of the crate is documented
Platform
Feature flags

docs.rs

Rust

Skip to main content

tokenize_refs_with_offsets

textprep0.1.4

In textprep::tokenize

textprep::tokenize

Function tokenize_refs_with_offsets

pub fn tokenize_refs_with_offsets(text: &str) -> Vec<TokenRef<'_>>