general-sam-1.0.5
- general-sam 1.0.5
- Permalink
- Docs.rs crate page
- MIT OR Apache-2.0
- 21 July 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- ChieloNewctle
- Dependencies
- rand ^0.10.2 normal optional
  criterion ^0.8.2 dev
  rand ^0.10.2 dev
  tokenizers ^0.23.1 dev
- Versions
- 7.81% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

Module trie

general_sam1.0.5

Module trie

Module Items

Functions

In general_sam::utils::tokenize

general_sam::utils::tokenize

Module trie

Functions§

greedy_tokenize_with_trie: Greedy tokenizer with a trie of the vocabulary.