sentience-tokenize-0.1.1
- sentience-tokenize 0.1.1
- Docs.rs crate page
- MIT
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- nbursa
- Dependencies
- criterion ^0.5 dev
  proptest ^1.5 dev
- Versions
- 21.57% of the crate is documented
Go to latest version
Platform
Feature flags

docs.rs

Rust

sentience_tokenize0.1.1

sentience_tokenize

Function tokenize_iter

pub fn tokenize_iter(src: &str) -> Tokens ⓘ

Expand description

Iterator-based API over tokens. Yields Result<Token, LexError>.