brack-tokenizer-0.1.0
- brack-tokenizer 0.1.0
- Docs.rs crate page
- MIT OR Apache-2.0
- Links
- crates.io
- Source
- Owners
- momeemt
- Dependencies
- anyhow ^1.0.79 normal
  unicode-segmentation ^1.10 normal
- Versions
- 0% of the crate is documented
Platform
Feature flags

docs.rs

Rust

tokenize

brack_tokenizer0.1.0

In brack_tokenizer::tokenize

brack_tokenizer::tokenize

Function tokenize

pub fn tokenize<P: AsRef<Path>>(path: P) -> Result<Vec<Token>>