chonkie 0.1.1

🦛 Chonkie, now in Rust 🦀: No-nonsense, ultra-fast, ultra-light chunking library
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107

<div align='center'>

![ChonkieR Logo](./assets/logo.png)

# 🦛 ChonkieR 🦀✨

[![Crates.io version](https://img.shields.io/crates/v/chonkie.svg)](https://crates.io/crates/chonkie)
[![License](https://img.shields.io/github/license/chonkie-inc/chonkie.svg)](https://github.com/chonkie-inc/chonkie-rs/blob/main/LICENSE)
[![Package size](https://img.shields.io/badge/size-TBD-blue)](https://github.com/chonkie-inc/chonkie-rs/blob/main/README.md#installation)
[![Discord](https://dcbadge.limes.pink/api/server/https://discord.gg/rYYp6DC4cv?style=flat)](https://discord.gg/rYYp6DC4cv)
[![GitHub stars](https://img.shields.io/github/stars/chonkie-inc/chonkie.svg)](https://github.com/chonkie-inc/chonkie-rs/stargazers) 

_The no-nonsense, lightweight and fast chunking library that's ready to CHONK your text, in Rust 🦀!_

[Installation](#installation) •
[Usage](#usage) •
[Chunkers](#chunkers) •
[Acknowledgements](#acknowledgements) •
[Citation](#citation)

</div>

Chonkie just got low-leveled! 🦀 Your favorite python chunking library is now in Rust~ even faster, smaller and reliable than ever!

**🦀 Rusty & Reliable**: Built with Rust for memory safety and performance. </br>
**🚀 Feature-rich**: All the CHONKs you'd ever need </br>
**✨ Easy to use**: Add Crate, Use Crate, CHONK </br>
**⚡ Blazingly Fast**: CHONK at the speed of Rust! zooooom </br>
**🪶 Light-weight**: No bloat, just CHONK </br>
**🦛 Cute CHONK mascot**: psst it's a pygmy hippo btw </br>
**❤️ [Moto Moto](#acknowledgements)'s favorite Rust library** </br>

**ChonkieR** is a chunking library that "**just works**" ✨

# Installation

To add `ChonkieR` to your project, run:

```bash
cargo add chonkie # Or add it to your Cargo.toml
```

`ChonkieR` follows the rule of minimum dependencies. Features can be enabled via Cargo features.
Don't want to think about it? Simply enable `all` features (Not recommended for production binaries unless needed)

```toml
# Cargo.toml
[dependencies]
chonkie = { version = "0.1.0", features = ["all"] } # Replace with desired version
```

# Usage

Here's a basic example to get you started:

```rust
use chonkie::CharacterTokenizer;
use chonkie::RecursiveChunker;
use chonkie::types::RecursiveRules;

fn main() { 
    // Initialize the chunker
    let chunker = RecursiveChunker::new(CharacterTokenizer::new(), 512, RecursiveRules::default());

    // Chunk some text
    let text = "ChonkieR is the goodest boi! My favorite chunking hippo hehe.";
    let chunks: Vec<RecursiveChunk> = chunker.chunk(text); 

    // Access chunks
    for chunk in chunks {
        println!("Chunk: {}", chunk.text); 
        println!("Tokens: {}", chunk.token_count); 
    }
}

```

Check out more usage examples in the [examples](https://github.com/chonkie-inc/chonkie-rs/tree/main/examples) folder!

# Chunkers

ChonkieR currently supports the following chunkers:

- **TokenChunker**: Split text into fixed-size token chunks.
- **SentenceChunker**: Split text into chunks based on sentence boundaries. 
- **RecursiveChunker**: Recursively split the text into chunks based on the rules provided.

# Acknowledgements

ChonkieR would like to CHONK its way through a special thanks to all the users and contributors who have helped make this library what it is today! Your feedback, issue reports, and improvements have helped make ChonkieR the CHONKIEST it can be.

And of course, special thanks to [Moto Moto](https://www.youtube.com/watch?v=I0zZC4wtqDQ&t=5s) for endorsing ChonkieR with his famous quote:
> "I like them big, I like them chonkieR." ~ Moto Moto (He really said this)

# Citation

If you use ChonkieR in your research, please cite it as follows:

```bibtex
@software{chonkie2025,
  author = {Minhas, Bhavnick AND Nigam, Shreyash},
  title = {Chonkie: A no-nonsense fast, lightweight, and efficient text chunking library},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/chonkie-inc/chonkie-rs}},
}
```