use-token-0.1.0
- use-token 0.1.0
- Permalink
- Docs.rs crate page
- MIT OR Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- CloudBranch
- Dependencies
- Versions
- 100% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

tokenize_words

use_token0.1.0

Function tokenize_words

pub fn tokenize_words(input: &str) -> Vec<Token>

Expand description

Extracts conservative word tokens.