pub struct RecursiveCharacterTextSplitter { /* private fields */ }Expand description
A high-performance recursive character text splitter.
Mirrors Python LangChain’s RecursiveCharacterTextSplitter API.
Tries each separator in priority order; splits on the first viable separator,
then recursively splits chunks that exceed chunk_size using the remaining
separators. Merges splits with overlap, matching Python’s merge_splits behavior.
§Efficiency advantages over Python version:
- Uses
&strbyte slicing — no string cloning until final chunk output - Pre-allocates output capacity based on text length
- Single-pass separator search using
str::find(no regex overhead — Python usesre.escape+re.search) - Minimal allocations in the hot path
Implementations§
Source§impl RecursiveCharacterTextSplitter
impl RecursiveCharacterTextSplitter
Sourcepub fn new() -> Self
pub fn new() -> Self
Create a new RecursiveCharacterTextSplitter with default separators.
Default separators: ["\n\n", "\n", " ", ""]
Default chunk size: 4096 characters
Default chunk overlap: 0
Sourcepub fn with_separators(self, separators: Vec<&'static str>) -> Self
pub fn with_separators(self, separators: Vec<&'static str>) -> Self
Set custom separators (tried in priority order)
Sourcepub fn with_chunk_size(self, size: usize) -> Self
pub fn with_chunk_size(self, size: usize) -> Self
Set the maximum chunk size (in characters by default)
Sourcepub fn with_chunk_overlap(self, overlap: usize) -> Self
pub fn with_chunk_overlap(self, overlap: usize) -> Self
Set the chunk overlap (in characters by default)
Sourcepub fn with_add_start_index(self, add: bool) -> Self
pub fn with_add_start_index(self, add: bool) -> Self
Whether to add start_index metadata to each chunk
Sourcepub fn with_strip_whitespace(self, strip: bool) -> Self
pub fn with_strip_whitespace(self, strip: bool) -> Self
Whether to strip whitespace from chunk edges
Sourcepub fn with_keep_separator(self, keep: bool) -> Self
pub fn with_keep_separator(self, keep: bool) -> Self
Whether to keep separators attached to each piece (Python default for RecursiveCharacterTextSplitter: True)
Sourcepub fn with_byte_length_function(self) -> Self
pub fn with_byte_length_function(self) -> Self
Use byte count instead of character count for chunk size measurement
Sourcepub fn with_strategy<S: SplitterStrategy + Default + Send + Sync + 'static>(
self,
) -> Self
pub fn with_strategy<S: SplitterStrategy + Default + Send + Sync + 'static>( self, ) -> Self
Set the splitting strategy (sequential or parallel via rayon).
When the rayon feature is enabled, you can select ParallelStrategy
for multi-core processing. When rayon is disabled, only
SequentialStrategy is available (which is also the default).
§Example
use recursive_text_splitter::{RecursiveCharacterTextSplitter, SequentialStrategy};
let splitter = RecursiveCharacterTextSplitter::new()
.with_strategy::<SequentialStrategy>();Sourcepub fn split_text(&self, text: &str) -> Vec<Chunk>
pub fn split_text(&self, text: &str) -> Vec<Chunk>
Split text into chunks.
Uses the configured strategy (sequential by default).
Sourcepub fn split_text_content(&self, text: &str) -> Vec<String>
pub fn split_text_content(&self, text: &str) -> Vec<String>
Split text into chunks, returning just the page_content strings (no metadata)
Sourcepub fn is_parallel(&self) -> bool
pub fn is_parallel(&self) -> bool
Check if rayon parallelism is available.