Expand description
A high-performance, dependency-free recursive character text splitter.
This crate is a Rust port of Python LangChain’s
RecursiveCharacterTextSplitter. It tries each separator in priority order,
splits on the first viable one, and recursively splits oversized pieces with
the remaining separators. Small pieces are merged with configurable overlap.
§Example
use recursive_text_splitter::RecursiveCharacterTextSplitter;
let splitter = RecursiveCharacterTextSplitter::new()
.with_separators(vec!["\n\n", "\n", " ", ""])
.with_chunk_size(20)
.with_chunk_overlap(5);
let chunks = splitter.split_text_content(
"First paragraph.\n\nSecond paragraph, which is a little longer.",
);
assert!(!chunks.is_empty());§Guarantees
- No
unsafecode, and no panics while splitting arbitrary UTF-8 input with a valid configuration. - With the built-in length functions, every chunk fits within
chunk_sizewhenever the text can be divided into indivisible units (Unicode scalar values) that are small enough. start_indexandend_indexare byte offsets into the input, so they can be used directly with Rust slicing. They are valid UTF-8 boundaries and are non-decreasing across the returned chunks.- The splitter is
Send + Syncand does not mutate shared state.
§Overlap semantics
RecursiveCharacterTextSplitter::with_chunk_overlap follows Python
LangChain’s merge algorithm. Overlap is a target for the suffix retained
from the previous chunk, not a strict lower or upper bound on the source
ranges. When the overlap is large relative to chunk_size, consecutive
chunks can start at the same offset, contain the same text (duplicates), or
be nested. This matches Python’s output. Lower the overlap if your RAG
pipeline requires strictly advancing chunks.
Invalid configuration is reported by SplitterError. The infallible
convenience methods panic with a descriptive message for invalid
configuration; use the try_* variants when configuration is not trusted.
§Feature flags
rayon— enablesParallelStrategy.regex— enables regex separators viaRecursiveCharacterTextSplitter::with_regex_separators.language— enablesLanguagepresets (impliesregex).serde— derivesSerialize/DeserializeforChunkandDocument.
The rig and rig-types features are deprecated no-ops. They are kept so
existing Cargo.toml files continue to resolve; mapping chunks to rig’s
document type is a few lines of ordinary Rust (see the README).
Structs§
- Chunk
- A chunk of text produced by the splitter.
- Document
- A document with text content and string metadata.
- Parallel
Strategy - Parallel splitting strategy powered by
rayon. - Recursive
Character Text Splitter - A recursive character text splitter.
- Sequential
Strategy - Default single-threaded splitting strategy.
Enums§
- Keep
Separator - Where to keep separators when splitting.
- Language
- Language presets for
RecursiveCharacterTextSplitter::with_language. - Length
Function - How to measure the length of a piece of text.
- Splitter
Error - Errors produced by invalid splitter configuration or invalid regex separators.
Traits§
- Splitter
Strategy - A strategy for turning text into chunks.
Functions§
- markdown_
text_ splitter - Creates a Markdown-configured splitter.