great-tokenizer-0.2.0
- great-tokenizer 0.2.0
- Docs.rs crate page
- MIT
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- daiyan1145
- Dependencies
- anyhow ^1.0.86 normal
  derive_more ^1.0.0 normal
  regex ^1.10.6 normal
  serde ^1.0.208 normal
  serde_derive ^1.0.208 normal
  serde_json ^1.0.125 normal
  thiserror ^1.0.63 normal
  unicode-segmentation ^1.11.0 normal
- Versions
- 0% of the crate is documented
Platform
Feature flags

docs.rs

Rust

build_tokenizer

great_tokenizer0.2.0

great_tokenizer

Function build_tokenizer

pub fn build_tokenizer<TSrc: ToString>(
    val: Vec<Matcher>,
    src: TSrc,
) -> Tokenizer