recursive-text-splitter
A fast, dependency-free-by-default Rust port of Python LangChain's
RecursiveCharacterTextSplitter, designed for RAG pipelines and crates such as
rig.
It tries separators in priority order, recursively splits oversized pieces, and
merges small pieces with configurable overlap. Output content matches
langchain-text-splitters for the same configuration, with one deliberate
improvement: this crate automatically falls back to Unicode scalar values, so
chunk_size is enforceable even when no user separator matches.
Installation
[]
= "0.2"
Optional features:
| Feature | Enables |
|---|---|
rayon |
ParallelStrategy and the explicit *_par methods |
regex |
Regex separators via with_regex_separators |
language |
Language presets such as Language::Markdown (implies regex) |
serde |
Serialize/Deserialize for Chunk and Document |
rig, rig-types |
Deprecated no-ops; use the mapping below instead |
= { = "0.2", = ["rayon", "language", "serde"] }
Quick start
use RecursiveCharacterTextSplitter;
let splitter = new
.with_separators
.with_chunk_size
.with_chunk_overlap;
let text = "INT. COCKPIT - DAY\n\nLUKE: I am a Jedi.\n\nEXT. FOREST - NIGHT\n\nVADER: Join me.";
let chunks = splitter.split_text_content;
for chunk in chunks
The defaults match Python LangChain's RecursiveCharacterTextSplitter:
chunk_size = 4000, chunk_overlap = 200, separators
["\n\n", "\n", " ", ""], keep_separator = Start, and
strip_whitespace = true.
Chunks and exact byte offsets
Use split_text when you need chunk metadata:
let splitter = new
.with_chunk_size
.with_chunk_overlap
.with_add_start_index;
let chunks = splitter.split_text;
for chunk in &chunks
start_index and end_index are byte offsets, unlike Python's character
offsets. They are always valid UTF-8 boundaries. For KeepSeparator::Start and
KeepSeparator::End, the range is exact. With KeepSeparator::None, separators
are removed from the content, so the range covers the corresponding source
region but does not necessarily equal page_content byte-for-byte.
Documents and metadata
use HashMap;
use ;
let document = with_metadata;
let splitter = new
.with_chunk_size
.with_add_start_index;
let chunks = splitter.split_documents;
assert_eq!;
assert!;
assert!;
create_documents(&["text"]) starts with empty metadata instead. Chunk
implements From<Chunk> for Document, which copies the offsets into metadata.
Using with rig
No rig feature or dependency is needed. Map chunks to
rig::completion::Document in a few lines, preserving metadata:
use ;
let splitter = new
.with_chunk_size
.with_chunk_overlap
.with_add_start_index;
let chunks: = splitter.create_documents;
let rig_documents: = chunks
.into_iter
.enumerate
.map
.collect;
rig::completion::Document has id: String, text: String, and
additional_props: HashMap<String, String>, which lines up directly with this
crate's Document.
Configuration
| Builder method | Default | Description |
|---|---|---|
with_separators(iter) |
["\n\n", "\n", " ", ""] |
Separator priority list; accepts &str or String |
with_chunk_size(n) |
4000 |
Maximum chunk size, measured by the length function |
with_chunk_overlap(n) |
200 |
Units reserved as overlap between chunks |
with_add_start_index(bool) |
false |
Populate start_index/end_index |
with_strip_whitespace(bool) |
true |
Trim chunk edges |
with_keep_separator(bool) |
true |
Python-compatible boolean shorthand |
with_keep_separator_mode(mode) |
Start |
None, Start, or End |
with_char_length_function() |
yes | Measure Unicode scalar values |
with_byte_length_function() |
Measure UTF-8 bytes | |
with_length_function(f) |
Custom measurement, e.g. a tokenizer | |
with_strategy::<S>() |
sequential | SequentialStrategy or ParallelStrategy |
with_strategy_instance(s) |
Install a configured or custom strategy |
An empty separator is appended when missing. This guarantees a final Unicode-scalar fallback and prevents oversized chunks. As in Python, separators listed after the first empty separator are ignored.
Overlap semantics
Overlap follows Python's merge algorithm. It is a target for the suffix
retained from the previous chunk, not a strict bound on source ranges. With a
large overlap relative to chunk_size, consecutive chunks can start at the
same offset, contain identical text, or be nested. Start and end offsets never
move backwards. Lower the overlap if your pipeline requires chunks that always
advance.
Validation and fallible APIs
chunk_size must be greater than zero and chunk_overlap must not exceed it.
The convenience methods panic with a descriptive message on invalid
configuration. Use the try_* variants for user-provided or untrusted
configuration:
use ;
let splitter = new
.with_chunk_size
.with_chunk_overlap;
match splitter.try_split_text_content
Regex and language separators
use ;
let whitespace = new
.with_regex_separators
.unwrap;
let markdown = new
.with_language
.unwrap;
let rust = new
.with_language
.unwrap;
let preset = markdown.unwrap;
Patterns use Rust's regex syntax, which has
linear-time matching and cannot suffer from catastrophic backtracking. Patterns
that can match the empty string are rejected because they are not meaningful
split points. Language presets mirror Python's lists where Rust regex supports
them; Visual Basic 6 is approximated because lookaround is unavailable.
Parallel splitting (rayon)
= { = "0.2", = ["rayon"] }
use ;
let splitter = new
.with_chunk_size
.with_chunk_overlap
.;
// Or use the explicit APIs:
let chunks = splitter.split_text_par;
let contents = splitter.split_text_content_par;
Output is identical to sequential splitting, including chunk order and byte offsets. Parallelism helps on large inputs with many oversized recursive splits; for small inputs, thread-pool overhead can make it slower.
Serde
let json = to_string.unwrap;
let decoded: = from_str.unwrap;
assert_eq!;
Python compatibility
Content parity is covered by tests and was additionally checked by differential
fuzzing against langchain-text-splitters 0.3.11 across literal separators,
regex separators, Unicode text, and all KeepSeparator modes.
Deliberate differences:
- Byte offsets.
start_index/end_indexare Rust byte offsets; Python reports character offsets. Python's start index is a heuristicstr.findand can be wrong or-1; this crate reports the actual source range. - Extra
end_index. Python only exposesstart_index. - Enforced chunk size. When user separators do not match, Python may emit
an oversized chunk. This crate appends
""and falls back to Unicode scalars. - Fallible configuration API. Python raises
ValueError; this crate offerstry_*methods and documents the panicking convenience methods. - No arbitrary metadata values.
Document::metadataisHashMap<String, String>for straightforwardrigand serde interop. - Rust regex syntax. Lookaround is unsupported; a few language presets (notably LaTeX/RST) are written to match the same literal intent.
Benchmarks
Illustrative results on the development machine (Apple Silicon, macOS,
rustc 1.99.0, Python 3.9, langchain-text-splitters 0.3.11). Generated
Star Wars-style scripts were measured with 20 iterations; the large corpus is
21.8M characters. Run the included benchmarks on your own hardware:
# Rust vs Python (same generated inputs)
PYTHONPATH=""
# Sequential vs parallel
| Input | Characters | Rust (ms) | Python (ms) | Speedup |
|---|---|---|---|---|
| generated_100 | 30,670 | 0.063 | 0.318 | 5.0x |
| generated_500 | 154,670 | 0.288 | 1.52 | 5.3x |
| generated_1000 | 309,670 | 0.469 | 3.07 | 6.5x |
| generated_5000 | 1,561,670 | 1.710 | 15.4 | 9.0x |
| large corpus | 21,863,380 | 20.3 | 220 | 10.8x |
Sequential vs parallel on the same inputs:
| Input | Sequential (ms) | Parallel (ms) | Speedup |
|---|---|---|---|
| generated_100 | 0.057 | 0.151 | 0.38x |
| generated_500 | 0.243 | 0.284 | 0.86x |
| generated_1000 | 0.433 | 0.367 | 1.18x |
| generated_5000 | 1.759 | 0.969 | 1.82x |
| large corpus | 20.0 | 12.7 | 1.58x |
An interactive version of these charts is in
docs/benchmark_comparison.html.
MSRV and license
Rust 1.85 (edition 2024). Licensed under MIT.