cargo-context-core-0.5.0
- cargo-context-core 0.5.0
- Permalink
- Docs.rs crate page
- Apache-2.0
- 20 July 2026
- Links
- Repository
- crates.io
- Source
- Owners
- asmuelle
- Dependencies
- cargo_metadata ^0.23 normal
  globset ^0.4 normal
  prettyplease ^0.2 normal
  proc-macro2 ^1 normal
  quote ^1 normal
  regex ^1 normal
  serde ^1 normal
  serde_json ^1 normal
  serde_yaml ^0.9 normal
  sha2 ^0.11 normal
  syn ^2 normal
  thiserror ^2 normal
  tiktoken-rs ^0.11 normal
  tokenizers ^0.22 normal
  tempfile ^3 dev
- Versions
- 21.48% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

parse_tokenizer

cargo_context_core0.5.0

In cargo_context_core::config

cargo_context_core::config

Function parse_tokenizer

pub fn parse_tokenizer(
    value: &str,
    hf_vocab_path: Option<PathBuf>,
) -> Result<Tokenizer>