rumtk-core-0.17.4
- rumtk-core 0.17.4
- Permalink
- Docs.rs crate page
- LGPL-3.0
- 23 August 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- luissantosHCIT
- Dependencies
- ahash ^0.8.12 normal
  base64 ^0.22.1 normal
  chardetng ^1.0.0 normal
  clap ^4.5.38 normal
  encoding_rs ^0.8.35 normal
  indexmap ^2.13.0 normal
  maplit ^1.0.2 normal
  nanoid ^0.4.0 normal
  num_cpus ^1.16.0 normal
  once_cell ^1.20.2 normal
  pyo3 ^0.28.3 normal
  pyo3-ffi ^0.28.3 normal
  rand ^0.10.1 normal
  regex ^1.12.2 normal
  rumtk-arena ^0.5.2 normal optional
  serde ^1.0.228 normal
  serde-big-array ^0.5.1 normal
  serde_json ^1.0.150 normal
  tokio ^1.43.0 normal
  unicode-segmentation ^1.13.2 normal
  uuid ^1.19.0 normal
- Versions
- 42.69% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

cpu_tokenize

rumtk_core0.17.4

In rumtk_core::cpu

rumtk_core::cpu

Function cpu_tokenize

pub fn cpu_tokenize<const WINDOW_SIZE: usize>(
    haystack: &[u8],
    bytes: &[u8],
) -> Vec<(u8, u32)>