kiru ⚡🗡️

Cut through text at the speed of light

The fastest text chunking library for RAG applications. Available for both Rust and Python.

What is kiru?

kiru is a high-performance text chunking library designed for modern RAG (Retrieval-Augmented Generation) systems. When you need to split millions of documents for vector databases or process streaming data in real-time, kiru delivers unmatched speed without sacrificing correctness.

Key Features

⚡ Blazing Fast (Python): 1000+ MB/s throughput for bytes, 300+ MB/s for characters
🎯 UTF-8 Safe: Never breaks multi-byte characters or emoji
💾 Memory Efficient: Stream gigabyte files with constant memory usage
🚀 Parallel Processing: Utilize all CPU cores automatically
🔌 Multiple Sources: Files, URLs, strings, and glob patterns
🛠️ Flexible Strategies: Chunk by bytes or characters
🦀 Rust Core: Rust performance and memory safety
🐍 Python Bindings: Pythonic API for ease of use

Performance

Benchmarked on 1MB text file, 1MB chunks, 1KB overlap:

Implementation	Strategy	Source	Time (ms)	Memory (MB)	Throughput (MB/s)
kiru (Rust)	bytes	string	0.23	-	4,370
kiru (Python)	bytes	string	0.71	2.9	1,408
kiru (Python)	chars	string	3.13	2.9	319
LangChain	chars	string	2,982	18.6	0.34

kiru is 4,000x faster than LangChain for byte chunking and 940x faster for character chunking!

Key insights:

Rust native performance: Up to 4,370 MB/s for byte chunking
Python bindings overhead: Still 1,400+ MB/s, beating all pure Python alternatives
Character-aware chunking: 300+ MB/s while respecting grapheme boundaries
Memory efficient: Uses 6x less memory than LangChain

Quick Start

Python 🐍

pip install kiru

from kiru import Chunker

# Create a chunker
chunker = Chunker.by_bytes(
    chunk_size=1024,  # 1KB chunks
    overlap=128       # 128 bytes overlap
)

# Chunk text
chunks = chunker.on_string("Your text here...").all()

# Chunk files in parallel
sources = ["file://doc1.txt", "https://example.com/page", "glob://*.md"]
for chunk in chunker.on_sources_par(sources):
    process(chunk)

Rust 🦀

Add to your Cargo.toml:

[dependencies]
kiru = "0.1"

use kiru::{BytesChunker, Chunker};

// Create a chunker
let chunker = BytesChunker::new(1024, 128)?;

// Chunk text
let chunks: Vec<String> = chunker
    .chunk_string("Your text here...".to_string())
    .collect();

// Stream large files
use kiru::{Source, StreamType};
let stream = StreamType::from_source(&Source::File("huge.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
    process(chunk);
}

Use Cases

Building RAG Systems

# Perfect for vector database ingestion
chunker = Chunker.by_bytes(512, 50)  # Tuned for embedding models

documents = ["glob://knowledge_base/**/*.md"]
chunks = chunker.on_sources_par(documents, channel_size=10000)

for chunk in chunks:
    embedding = model.encode(chunk)
    vector_db.insert(chunk, embedding)

Real-time Processing

# Stream processing without memory overhead
for chunk in chunker.on_file("10GB_file.txt"):
    # Each chunk generated on-demand
    send_to_queue(chunk)

Parallel Document Processing

// Process hundreds of documents concurrently
use kiru::{ChunkerBuilder, ChunkerEnum};

let chunker = ChunkerBuilder::by_bytes(ChunkerEnum::Bytes {
    chunk_size: 4096,
    overlap: 512,
});

let sources = vec!["glob://docs/**/*.txt"];
let chunks = chunker.on_sources_par_stream(sources, 1000)?;

Chunking Strategies

Bytes Chunking

Splits on byte boundaries while respecting UTF-8
Fastest performance (1000+ MB/s in Rust, 1400+ MB/s in Python)
Ideal for token-limited models and consistent memory usage

Characters Chunking

Splits on character (grapheme) boundaries
Ensures exact character counts regardless of byte representation
Perfect for character-limited APIs (300+ MB/s in Python)

API Reference

Python API

Creating Chunkers

from kiru import Chunker

# Byte-based chunking
chunker = Chunker.by_bytes(chunk_size=1024, overlap=128)

# Character-based chunking
chunker = Chunker.by_characters(chunk_size=1000, overlap=100)

Input Sources

# Single string
chunks = chunker.on_string("text...").all()

# Single file
chunks = chunker.on_file("/path/to/file.txt").all()

# HTTP/HTTPS URL
chunks = chunker.on_http("https://example.com/page").all()

# Multiple sources (serial)
sources = ["file://doc1.txt", "https://example.com/page", "glob://*.md"]
chunks = chunker.on_sources(sources).all()

# Multiple sources (parallel)
chunks = chunker.on_sources_par(sources, channel_size=1000).all()

# Or iterate lazily
for chunk in chunker.on_sources_par(sources):
    process(chunk)

Source Prefixes

file://path/to/file.txt - Local files
http://example.com or https://example.com - URLs
text://Inline text content - Raw text strings
glob://*.md - Glob patterns
No prefix - Treated as raw text

Rust API

Creating Chunkers

use kiru::{BytesChunker, CharactersChunker, Chunker};

// Byte-based chunking
let chunker = BytesChunker::new(1024, 128)?;

// Character-based chunking
let chunker = CharactersChunker::new(1000, 100)?;

Basic Usage

use kiru::Chunker;

// Chunk a string
let chunks: Vec<String> = chunker
    .chunk_string("Your text here".to_string())
    .collect();

// Stream a file
use kiru::{Source, StreamType};
let stream = StreamType::from_source(&Source::File("file.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
    // Process chunk
}

Advanced Usage

use kiru::{ChunkerBuilder, ChunkerEnum, Source, HigherOrderSource, SourceGenerator};

// Create chunker with builder pattern
let chunker = ChunkerBuilder::by_bytes(ChunkerEnum::Bytes {
    chunk_size: 4096,
    overlap: 512,
});

// Single source
let chunks = chunker.on_source(Source::File("doc.txt".to_string()))?;

// Multiple sources (serial)
let sources = vec![
    Source::File("doc1.txt".to_string()),
    Source::Http("https://example.com".to_string()),
];
let chunks = chunker.on_sources(sources)?;

// Multiple sources (parallel) - returns Vec
let chunks: Vec<String> = chunker.on_sources_par(sources)?;

// Multiple sources (parallel streaming) - returns iterator
let chunks = chunker.on_sources_par_stream(sources, 1000)?;
for chunk in chunks {
    // Process as they arrive
}

// Using glob patterns
let sources = vec![HigherOrderSource::SourceGenerator(
    SourceGenerator::Glob("**/*.md".to_string())
)];
let flattened = HigherOrderSource::into_flattened_sources(sources)?;

Architecture

┌─────────────────────────────────────────┐
│           Application Layer              │
│     (Python or Rust Application)        │
├─────────────────────────────────────────┤
│          kiru-py (PyO3 Bindings)        │
│              [Python only]               │
├─────────────────────────────────────────┤
│         kiru-core (Rust Library)        │
│                                          │
│        ┌──────────┬───────────┐         │
│        │ Chunkers │ Streaming │         │  
│        │  Engine  │   Engine  │         │
│        └──────────┴───────────┘         │
└─────────────────────────────────────────┘

Project Structure

kiru/
├── README.md              # This file (shared documentation)
├── kiru-core/             # Rust implementation
│   ├── src/               # Core chunking algorithms
│   │   ├── bytes_chunker.rs
│   │   ├── characters_chunker.rs
│   │   ├── chunker.rs     # Builder pattern & parallel processing
│   │   └── stream.rs      # File/HTTP streaming
│   ├── benches/           # Criterion benchmarks
│   └── tests/             # Property-based tests
├── kiru-py/               # Python bindings (PyO3)
│   ├── src/lib.rs         # Python wrapper
│   └── python/            # Python tests & benchmarks
└── utils/                 # Version management scripts

Streaming & Memory Efficiency

kiru's killer feature: true streaming with constant memory usage.

Unlike traditional chunkers that load entire files into memory, kiru processes data as it arrives using an intelligent buffering system. This means you can chunk gigabyte-sized files with minimal RAM usage.

How Streaming Works

File/HTTP Source → Read Blocks (8KB) → UTF-8 Buffer → Chunk Iterator → Your Code
                      ↓                      ↓
                 As needed              Constant size

Key advantages:

Constant Memory: Process 10GB files with ~10MB RAM
Immediate Results: First chunks available instantly, no waiting for full file load
Works Everywhere: Local files, HTTP/HTTPS streams, any data source
UTF-8 Safe: Buffer maintains character boundaries automatically

Python Examples

from kiru import Chunker

chunker = Chunker.by_bytes(chunk_size=4096, overlap=512)

# ⚡ Stream a 10GB file - uses only ~10MB RAM
for chunk in chunker.on_file("huge_dataset.txt"):
    # Process chunk immediately as it arrives
    vector_db.insert(chunk)
    # No waiting, no memory explosion!

# ⚡ Stream from HTTP - process as data downloads
for chunk in chunker.on_http("https://example.com/large_document.txt"):
    process(chunk)
    # Chunks ready while download continues

# ⚡ Stream multiple sources in parallel
sources = [
    "file://10gb_file1.txt",
    "https://example.com/doc.txt",
    "file://10gb_file2.txt"
]
for chunk in chunker.on_sources_par(sources, channel_size=1000):
    # All sources stream in parallel
    # Memory stays constant regardless of file sizes
    send_to_queue(chunk)

Rust Examples

use kiru::{BytesChunker, Chunker, Source, StreamType};

let chunker = BytesChunker::new(4096, 512)?;

// ⚡ Stream a massive file with constant memory
let stream = StreamType::from_source(&Source::File("10gb_file.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
    // Process immediately, no memory buildup
    vector_db.insert(chunk);
}

// ⚡ Stream from HTTP as data arrives
let stream = StreamType::from_source(&Source::Http("https://example.com/doc.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
    process(chunk);
}

Memory Comparison

Processing a 1GB file with 4KB chunks:

Library	Memory Usage	Loads Full File?	Streaming?
kiru	~10 MB	❌ No	✅ Yes
LangChain	1000+ MB	✅ Yes	❌ No
tiktoken	1000+ MB	✅ Yes	❌ No

Result: kiru uses 100x less memory while being 4,000x faster!

Development

Setup

# Clone repository
git clone https://github.com/yourusername/kiru.git
cd kiru

# Run all tests
cargo test --workspace

# Run Rust benchmarks
cd kiru-core
cargo bench

# Build Python package
cd ../kiru-py
pip install maturin
maturin develop --release

# Run Python tests
pip install pytest hypothesis
pytest python/test.py

# Run Python benchmarks
python python/bench.py

Running Benchmarks

# Rust benchmarks
cd kiru-core
cargo bench

# Python benchmarks
cd kiru-py
python python/bench.py

Performance Tips

Use byte chunking for maximum throughput (1000+ MB/s)
Use character chunking when exact character counts matter (300+ MB/s)
Enable parallel processing with on_sources_par() for multiple files
Tune chunk size based on your embedding model's context window
Adjust overlap to balance context preservation and storage
Stream large files to maintain constant memory usage

Why "kiru"?

"Kiru" (切る) is Japanese for "to cut" - reflecting the library's purpose of cutting text into chunks at lightning speed ⚡🗡️

Contributing

We welcome contributions! Please check out our Contributing Guide for guidelines.

License

MIT License - see LICENSE for details.

Credits

Built with:

PyO3 - Rust bindings for Python
Rayon - Data parallelism for Rust
maturin - Build and publish Rust Python extensions

Ready to cut through text at the speed of light?

🐍 Python: pip install kiru
🦀 Rust: Add kiru = "0.1" to Cargo.toml

Get started with PyPI | Crates.io | Documentation

kiru 0.1.11

kiru ⚡🗡️

What is kiru?

Key Features

Performance

Quick Start

Python 🐍

Rust 🦀

Use Cases

Building RAG Systems

Real-time Processing

Parallel Document Processing

Chunking Strategies

Bytes Chunking

Characters Chunking

API Reference

Python API

Creating Chunkers

Input Sources

Source Prefixes

Rust API

Creating Chunkers

Basic Usage

Advanced Usage

Architecture

Project Structure

Streaming & Memory Efficiency

How Streaming Works

Python Examples

Rust Examples

Memory Comparison

Development

Setup

Running Benchmarks

Performance Tips

Why "kiru"?

Contributing

License

Credits