chonkier 0.0.2

🦛 Chonkie, now in Rust 🦀: No-nonsense, ultra-fast, ultra-light chunking library
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

/*
The main module for Chonkie. It contains all the chunkers.

The Chonkie Chunkers are:
- Token Chunker: Chunks the text into tokens.
*/

// Exports all the chunkers
pub mod recursive;
pub mod sentence;
pub mod token;

// Re-export the Chunkers
pub use recursive::RecursiveChunker;
pub use sentence::SentenceChunker;
pub use token::TokenChunker;