semchunk-rs-0.1.1
- semchunk-rs 0.1.1
- Permalink
- Docs.rs crate page
- MIT
- Links
- Repository
- crates.io
- Source
- Owners
- dominictarro
- Dependencies
- bisection ^0.1.0 normal
  
  regex ^1.11.1 normal
  
  rust-bert ^0.23.0 normal optional
  
  rust_tokenizers ^8.1.1 normal optional
  
  tiktoken-rs ^0.6.0 normal optional
  
  criterion ^0.5.1 dev
  
  glob ^0.3.1 dev
- Versions
- 81.82% of the crate is documented
Platform
Feature flags

docs.rs

Rust

semchunk_rs0.1.1

Module chunker

Module Items

Structs

In crate semchunk_rs

Module chunker

Structs§

Chunker: A struct for chunking texts into segments based on a maximum number of tokens per chunk and a token counter function.