recursive-text-splitter 0.2.0

A high-performance recursive character text splitter that mirrors Python LangChain's RecursiveCharacterTextSplitter
Documentation

recursive-text-splitter

A fast, dependency-free-by-default Rust port of Python LangChain's RecursiveCharacterTextSplitter, designed for RAG pipelines and crates such as rig.

It tries separators in priority order, recursively splits oversized pieces, and merges small pieces with configurable overlap. Output content matches langchain-text-splitters for the same configuration, with one deliberate improvement: this crate automatically falls back to Unicode scalar values, so chunk_size is enforceable even when no user separator matches.

Installation

[dependencies]
recursive-text-splitter = "0.2"

Optional features:

Feature Enables
rayon ParallelStrategy and the explicit *_par methods
regex Regex separators via with_regex_separators
language Language presets such as Language::Markdown (implies regex)
serde Serialize/Deserialize for Chunk and Document
rig, rig-types Deprecated no-ops; use the mapping below instead
recursive-text-splitter = { version = "0.2", features = ["rayon", "language", "serde"] }

Quick start

use recursive_text_splitter::RecursiveCharacterTextSplitter;

let splitter = RecursiveCharacterTextSplitter::new()
    .with_separators(["\nINT.", "\nEXT.", "\n\n", "\n", " ", ""])
    .with_chunk_size(200)
    .with_chunk_overlap(50);

let text = "INT. COCKPIT - DAY\n\nLUKE: I am a Jedi.\n\nEXT. FOREST - NIGHT\n\nVADER: Join me.";
let chunks = splitter.split_text_content(text);

for chunk in chunks {
    println!("{chunk}");
}

The defaults match Python LangChain's RecursiveCharacterTextSplitter: chunk_size = 4000, chunk_overlap = 200, separators ["\n\n", "\n", " ", ""], keep_separator = Start, and strip_whitespace = true.

Chunks and exact byte offsets

Use split_text when you need chunk metadata:

let splitter = RecursiveCharacterTextSplitter::new()
    .with_chunk_size(500)
    .with_chunk_overlap(50)
    .with_add_start_index(true);

let chunks = splitter.split_text(text);
for chunk in &chunks {
    let start = chunk.start_index.unwrap();
    let end = chunk.end_index.unwrap();

    // Both offsets are UTF-8 byte offsets and are safe to slice with.
    assert_eq!(&text[start..end], chunk.page_content);
}

start_index and end_index are byte offsets, unlike Python's character offsets. They are always valid UTF-8 boundaries. For KeepSeparator::Start and KeepSeparator::End, the range is exact. With KeepSeparator::None, separators are removed from the content, so the range covers the corresponding source region but does not necessarily equal page_content byte-for-byte.

Documents and metadata

use std::collections::HashMap;
use recursive_text_splitter::{Document, RecursiveCharacterTextSplitter};

let document = Document::with_metadata(
    "First paragraph.\n\nSecond paragraph.",
    HashMap::from([("source".to_owned(), "guide.md".to_owned())]),
);

let splitter = RecursiveCharacterTextSplitter::new()
    .with_chunk_size(20)
    .with_add_start_index(true);

let chunks = splitter.split_documents(&[document]);
assert_eq!(chunks[0].metadata["source"], "guide.md");
assert!(chunks[0].metadata.contains_key("start_index"));
assert!(chunks[0].metadata.contains_key("end_index"));

create_documents(&["text"]) starts with empty metadata instead. Chunk implements From<Chunk> for Document, which copies the offsets into metadata.

Using with rig

No rig feature or dependency is needed. Map chunks to rig::completion::Document in a few lines, preserving metadata:

use recursive_text_splitter::{Document as TextDocument, RecursiveCharacterTextSplitter};

let splitter = RecursiveCharacterTextSplitter::new()
    .with_chunk_size(1000)
    .with_chunk_overlap(100)
    .with_add_start_index(true);

let chunks: Vec<TextDocument> = splitter.create_documents(&[text.as_str()]);

let rig_documents: Vec<rig::completion::Document> = chunks
    .into_iter()
    .enumerate()
    .map(|(index, chunk)| rig::completion::Document {
        id: format!("chunk-{index}"),
        text: chunk.page_content,
        additional_props: chunk.metadata,
    })
    .collect();

rig::completion::Document has id: String, text: String, and additional_props: HashMap<String, String>, which lines up directly with this crate's Document.

Configuration

Builder method Default Description
with_separators(iter) ["\n\n", "\n", " ", ""] Separator priority list; accepts &str or String
with_chunk_size(n) 4000 Maximum chunk size, measured by the length function
with_chunk_overlap(n) 200 Units reserved as overlap between chunks
with_add_start_index(bool) false Populate start_index/end_index
with_strip_whitespace(bool) true Trim chunk edges
with_keep_separator(bool) true Python-compatible boolean shorthand
with_keep_separator_mode(mode) Start None, Start, or End
with_char_length_function() yes Measure Unicode scalar values
with_byte_length_function() Measure UTF-8 bytes
with_length_function(f) Custom measurement, e.g. a tokenizer
with_strategy::<S>() sequential SequentialStrategy or ParallelStrategy
with_strategy_instance(s) Install a configured or custom strategy

An empty separator is appended when missing. This guarantees a final Unicode-scalar fallback and prevents oversized chunks. As in Python, separators listed after the first empty separator are ignored.

Overlap semantics

Overlap follows Python's merge algorithm. It is a target for the suffix retained from the previous chunk, not a strict bound on source ranges. With a large overlap relative to chunk_size, consecutive chunks can start at the same offset, contain identical text, or be nested. Start and end offsets never move backwards. Lower the overlap if your pipeline requires chunks that always advance.

Validation and fallible APIs

chunk_size must be greater than zero and chunk_overlap must not exceed it. The convenience methods panic with a descriptive message on invalid configuration. Use the try_* variants for user-provided or untrusted configuration:

use recursive_text_splitter::{RecursiveCharacterTextSplitter, SplitterError};

let splitter = RecursiveCharacterTextSplitter::new()
    .with_chunk_size(0)
    .with_chunk_overlap(0);

match splitter.try_split_text_content("text") {
    Err(SplitterError::InvalidChunkSize { chunk_size }) => {
        eprintln!("bad chunk size: {chunk_size}");
    }
    Err(error) => eprintln!("{error}"),
    Ok(chunks) => println!("{} chunks", chunks.len()),
}

Regex and language separators

use recursive_text_splitter::{Language, RecursiveCharacterTextSplitter};

let whitespace = RecursiveCharacterTextSplitter::new()
    .with_regex_separators([r"\s+"])
    .unwrap();

let markdown = RecursiveCharacterTextSplitter::new()
    .with_language(Language::Markdown)
    .unwrap();

let rust = RecursiveCharacterTextSplitter::new()
    .with_language(Language::Rust)
    .unwrap();

let preset = RecursiveCharacterTextSplitter::markdown().unwrap();

Patterns use Rust's regex syntax, which has linear-time matching and cannot suffer from catastrophic backtracking. Patterns that can match the empty string are rejected because they are not meaningful split points. Language presets mirror Python's lists where Rust regex supports them; Visual Basic 6 is approximated because lookaround is unavailable.

Parallel splitting (rayon)

recursive-text-splitter = { version = "0.2", features = ["rayon"] }
use recursive_text_splitter::{ParallelStrategy, RecursiveCharacterTextSplitter};

let splitter = RecursiveCharacterTextSplitter::new()
    .with_chunk_size(500)
    .with_chunk_overlap(50)
    .with_strategy::<ParallelStrategy>();

// Or use the explicit APIs:
let chunks = splitter.split_text_par(text);
let contents = splitter.split_text_content_par(text);

Output is identical to sequential splitting, including chunk order and byte offsets. Parallelism helps on large inputs with many oversized recursive splits; for small inputs, thread-pool overhead can make it slower.

Serde

let json = serde_json::to_string(&chunks).unwrap();
let decoded: Vec<recursive_text_splitter::Chunk> = serde_json::from_str(&json).unwrap();
assert_eq!(decoded, chunks);

Python compatibility

Content parity is covered by tests and was additionally checked by differential fuzzing against langchain-text-splitters 0.3.11 across literal separators, regex separators, Unicode text, and all KeepSeparator modes.

Deliberate differences:

  1. Byte offsets. start_index/end_index are Rust byte offsets; Python reports character offsets. Python's start index is a heuristic str.find and can be wrong or -1; this crate reports the actual source range.
  2. Extra end_index. Python only exposes start_index.
  3. Enforced chunk size. When user separators do not match, Python may emit an oversized chunk. This crate appends "" and falls back to Unicode scalars.
  4. Fallible configuration API. Python raises ValueError; this crate offers try_* methods and documents the panicking convenience methods.
  5. No arbitrary metadata values. Document::metadata is HashMap<String, String> for straightforward rig and serde interop.
  6. Rust regex syntax. Lookaround is unsupported; a few language presets (notably LaTeX/RST) are written to match the same literal intent.

Benchmarks

Illustrative results on the development machine (Apple Silicon, macOS, rustc 1.99.0, Python 3.9, langchain-text-splitters 0.3.11). Generated Star Wars-style scripts were measured with 20 iterations; the large corpus is 21.8M characters. Run the included benchmarks on your own hardware:

# Rust vs Python (same generated inputs)
cargo bench --bench cross_lang_bench
PYTHONPATH="" python3 benches/cross_lang_bench.py

# Sequential vs parallel
cargo bench --bench parallel_bench --features rayon
Input Characters Rust (ms) Python (ms) Speedup
generated_100 30,670 0.063 0.318 5.0x
generated_500 154,670 0.288 1.52 5.3x
generated_1000 309,670 0.469 3.07 6.5x
generated_5000 1,561,670 1.710 15.4 9.0x
large corpus 21,863,380 20.3 220 10.8x

Sequential vs parallel on the same inputs:

Input Sequential (ms) Parallel (ms) Speedup
generated_100 0.057 0.151 0.38x
generated_500 0.243 0.284 0.86x
generated_1000 0.433 0.367 1.18x
generated_5000 1.759 0.969 1.82x
large corpus 20.0 12.7 1.58x

An interactive version of these charts is in docs/benchmark_comparison.html.

MSRV and license

Rust 1.85 (edition 2024). Licensed under MIT.