Skip to main content

Crate recursive_text_splitter

Crate recursive_text_splitter 

Source
Expand description

A high-performance, dependency-free recursive character text splitter.

This crate is a Rust port of Python LangChain’s RecursiveCharacterTextSplitter. It tries each separator in priority order, splits on the first viable one, and recursively splits oversized pieces with the remaining separators. Small pieces are merged with configurable overlap.

§Example

use recursive_text_splitter::RecursiveCharacterTextSplitter;

let splitter = RecursiveCharacterTextSplitter::new()
    .with_separators(vec!["\n\n", "\n", " ", ""])
    .with_chunk_size(20)
    .with_chunk_overlap(5);

let chunks = splitter.split_text_content(
    "First paragraph.\n\nSecond paragraph, which is a little longer.",
);

assert!(!chunks.is_empty());

§Guarantees

  • No unsafe code, and no panics while splitting arbitrary UTF-8 input with a valid configuration.
  • With the built-in length functions, every chunk fits within chunk_size whenever the text can be divided into indivisible units (Unicode scalar values) that are small enough.
  • start_index and end_index are byte offsets into the input, so they can be used directly with Rust slicing. They are valid UTF-8 boundaries and are non-decreasing across the returned chunks.
  • The splitter is Send + Sync and does not mutate shared state.

§Overlap semantics

RecursiveCharacterTextSplitter::with_chunk_overlap follows Python LangChain’s merge algorithm. Overlap is a target for the suffix retained from the previous chunk, not a strict lower or upper bound on the source ranges. When the overlap is large relative to chunk_size, consecutive chunks can start at the same offset, contain the same text (duplicates), or be nested. This matches Python’s output. Lower the overlap if your RAG pipeline requires strictly advancing chunks.

Invalid configuration is reported by SplitterError. The infallible convenience methods panic with a descriptive message for invalid configuration; use the try_* variants when configuration is not trusted.

§Feature flags

The rig and rig-types features are deprecated no-ops. They are kept so existing Cargo.toml files continue to resolve; mapping chunks to rig’s document type is a few lines of ordinary Rust (see the README).

Structs§

Chunk
A chunk of text produced by the splitter.
Document
A document with text content and string metadata.
ParallelStrategy
Parallel splitting strategy powered by rayon.
RecursiveCharacterTextSplitter
A recursive character text splitter.
SequentialStrategy
Default single-threaded splitting strategy.

Enums§

KeepSeparator
Where to keep separators when splitting.
Language
Language presets for RecursiveCharacterTextSplitter::with_language.
LengthFunction
How to measure the length of a piece of text.
SplitterError
Errors produced by invalid splitter configuration or invalid regex separators.

Traits§

SplitterStrategy
A strategy for turning text into chunks.

Functions§

markdown_text_splitter
Creates a Markdown-configured splitter.