probe-code-0.6.0
- probe-code 0.6.0
- Docs.rs crate page
- MIT
- Links
- Repository
- crates.io
- Source
- Owners
- buger
- Dependencies
- ahash ^0.8.6 normal
  anyhow ^1.0 normal
  arboard ^3.4.1 normal
  ast-grep-core ^0.36.1 normal
  ast-grep-language ^0.36.1 normal
  async-trait ^0.1 normal
  atty ^0.2.13 normal
  clap ^4.3 normal
  colored ^3.0.0 normal
  dashmap ^5.5.3 normal
  decompound ^0.3.0 normal
  dirs ^5.0.1 normal
  futures ^0.3 normal
  glob ^0.3.1 normal
  grep ^0.3.2 normal
  ignore ^0.4 normal
  itertools ^0.14.0 normal
  lazy_static ^1.4.0 normal
  once_cell ^1.19.0 normal
  rand ^0.8.5 normal
  rayon ^1.8 normal
  regex ^1.9 normal
  rig-core ^0.9.1 normal
  rust-stemmers ^1.2 normal
  serde ^1.0 normal
  serde_json ^1.0 normal
  thiserror ^2.0.12 normal
  tiktoken-rs ^0.6.0 normal
  tokio ^1 normal
  tracing-subscriber ^0.3 normal
  tree-sitter ^0.24.5 normal
  tree-sitter-c ^0.23.4 normal
  tree-sitter-c-sharp ^0.23.1 normal
  tree-sitter-cpp ^0.23.4 normal
  tree-sitter-go ^0.23.4 normal
  tree-sitter-java ^0.23.5 normal
  tree-sitter-javascript ^0.23.1 normal
  tree-sitter-php ^0.23.11 normal
  tree-sitter-python ^0.23.6 normal
  tree-sitter-ruby ^0.23.1 normal
  tree-sitter-rust ^0.23.2 normal
  tree-sitter-swift ^0.7.0 normal
  tree-sitter-typescript ^0.23.2 normal
  criterion ^0.5 dev
  jsonschema ^0.17.1 dev
  proptest ^1.2 dev
  roxmltree ^0.18.0 dev
  tempfile ^3.10.1 dev
  xml-schema ^0.2.0 dev
- Versions
- 44.72% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

get_tokenizer

probe_code0.6.0

In probe_code::search::search_tokens

probe_code::search::search_tokens

Function get_tokenizer

pub fn get_tokenizer() -> &'static CoreBPE

Expand description

Returns a reference to the tiktoken tokenizer