moritzbrantner-text-core-0.1.0
- moritzbrantner-text-core 0.1.0
- Docs.rs crate page
- MIT OR Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- moritzbrantner
- Dependencies
- moritzbrantner-jobs-core ^0.1.1 normal
  moritzbrantner-runtime-core ^0.1.1 normal
  serde ^1.0.228 normal
  serde_json ^1.0.150 normal
  unicode-normalization ^0.1.24 normal
  unicode-segmentation ^1.12.0 normal
  moritzbrantner-video-analysis-core ^0.1.3 normal
  criterion ^0.5.1 dev
- Versions
- 69.6% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

tokenize

text_core0.1.0

Function tokenize

pub fn tokenize(text: &str, options: &TextProcessingOptions) -> Vec<Token>

Expand description

Returns tokenize.