pub struct RecursiveCharacterTextSplitter { /* private fields */ }Expand description
A recursive character text splitter.
See the crate documentation for examples and guarantees.
Implementations§
Source§impl RecursiveCharacterTextSplitter
impl RecursiveCharacterTextSplitter
Sourcepub fn new() -> Self
pub fn new() -> Self
Creates a splitter with LangChain-compatible defaults.
Default separators: ["\n\n", "\n", " ", ""]
Default chunk size: 4000 characters
Default chunk overlap: 200 characters
Sourcepub fn with_separators<I, S>(self, separators: I) -> Self
pub fn with_separators<I, S>(self, separators: I) -> Self
Sets the separators, tried in priority order.
An empty separator ("") is appended automatically when missing. This
guarantees that the splitter can always fall back to Unicode scalar
values and honour chunk_size, unlike Python LangChain, which can emit
oversized chunks when the final separator does not match. As in Python,
separators after the first empty separator are ignored.
Both borrowed and owned separator strings are accepted:
use recursive_text_splitter::RecursiveCharacterTextSplitter;
let dynamic = vec![String::from("\n\n"), String::from("\n"), String::from("")];
let splitter = RecursiveCharacterTextSplitter::new().with_separators(dynamic);
assert_eq!(splitter.chunk_size(), 4000);Sourcepub fn with_regex_separators<I, S>(
self,
separators: I,
) -> Result<Self, SplitterError>
pub fn with_regex_separators<I, S>( self, separators: I, ) -> Result<Self, SplitterError>
Sets regex separators, tried in priority order.
The patterns use the Rust regex syntax, which
guarantees linear-time matching and therefore cannot suffer from
catastrophic backtracking. Patterns that match the empty string are
rejected because they do not describe meaningful split points.
Requires the regex feature.
§Errors
Returns SplitterError::InvalidRegex if a pattern is not valid Rust
regex syntax or can match the empty string.
Sourcepub fn with_language(self, language: Language) -> Result<Self, SplitterError>
pub fn with_language(self, language: Language) -> Result<Self, SplitterError>
Configures separators for a programming or markup language.
Requires the language feature. The preset patterns are regular
expressions compatible with the regex crate.
use recursive_text_splitter::{Language, RecursiveCharacterTextSplitter};
let splitter = RecursiveCharacterTextSplitter::new()
.with_language(Language::Markdown)
.unwrap()
.with_chunk_size(500);§Errors
Returns SplitterError::InvalidRegex if one of the preset patterns is
not valid Rust regex syntax.
Sourcepub fn with_chunk_size(self, size: usize) -> Self
pub fn with_chunk_size(self, size: usize) -> Self
Sets the maximum chunk size, measured by the configured length function.
Sourcepub fn with_chunk_overlap(self, overlap: usize) -> Self
pub fn with_chunk_overlap(self, overlap: usize) -> Self
Sets the number of units to overlap between adjacent chunks.
The value is a target, not a hard guarantee: Python-compatible merging
can produce duplicate or nested chunks when the overlap is large
relative to chunk_size. See the crate documentation.
Sourcepub fn with_add_start_index(self, add: bool) -> Self
pub fn with_add_start_index(self, add: bool) -> Self
Enables or disables start_index/end_index byte offsets on chunks.
Sourcepub fn with_strip_whitespace(self, strip: bool) -> Self
pub fn with_strip_whitespace(self, strip: bool) -> Self
Enables or disables trimming whitespace from chunk edges.
Sourcepub fn with_keep_separator(self, keep: bool) -> Self
pub fn with_keep_separator(self, keep: bool) -> Self
Sets whether separators are kept (the Python boolean API).
true is equivalent to KeepSeparator::Start, false to
KeepSeparator::None.
Sourcepub fn with_keep_separator_mode(self, mode: KeepSeparator) -> Self
pub fn with_keep_separator_mode(self, mode: KeepSeparator) -> Self
Sets where separators are kept in the output.
Sourcepub fn with_byte_length_function(self) -> Self
pub fn with_byte_length_function(self) -> Self
Measures chunks in UTF-8 bytes instead of Unicode scalar values.
This clears any custom length function.
Sourcepub fn with_char_length_function(self) -> Self
pub fn with_char_length_function(self) -> Self
Measures chunks in Unicode scalar values (the default).
This clears any custom length function.
Sourcepub fn with_length_function<F>(self, length_function: F) -> Self
pub fn with_length_function<F>(self, length_function: F) -> Self
Uses a custom function to measure text length, for example a tokenizer.
The function must be thread-safe because the splitter is Send + Sync.
Custom length functions override the built-in character/byte counters.
use recursive_text_splitter::RecursiveCharacterTextSplitter;
let splitter = RecursiveCharacterTextSplitter::new()
.with_length_function(|text| text.split_whitespace().count())
.with_chunk_size(10)
.with_chunk_overlap(0);
assert!(!splitter.split_text_content("one two three").is_empty());Sourcepub fn with_strategy<S>(self) -> Self
pub fn with_strategy<S>(self) -> Self
Sets the splitting strategy from its Default value.
use recursive_text_splitter::{RecursiveCharacterTextSplitter, SequentialStrategy};
let splitter = RecursiveCharacterTextSplitter::new()
.with_strategy::<SequentialStrategy>();Sourcepub fn with_strategy_instance<S>(self, strategy: S) -> Self
pub fn with_strategy_instance<S>(self, strategy: S) -> Self
Installs a specific strategy instance.
Use this instead of Self::with_strategy when the strategy carries
configuration. Custom strategies should delegate to
SequentialStrategy or implement splitting directly; see
SplitterStrategy for details.
Sourcepub fn separators(&self) -> impl ExactSizeIterator<Item = &str> + '_
pub fn separators(&self) -> impl ExactSizeIterator<Item = &str> + '_
Returns the configured separators, including the implicit character fallback.
Sourcepub const fn chunk_size(&self) -> usize
pub const fn chunk_size(&self) -> usize
Returns the configured chunk size.
Sourcepub const fn chunk_overlap(&self) -> usize
pub const fn chunk_overlap(&self) -> usize
Returns the configured chunk overlap.
Sourcepub const fn add_start_index(&self) -> bool
pub const fn add_start_index(&self) -> bool
Returns whether chunk byte offsets are enabled.
Sourcepub const fn strip_whitespace(&self) -> bool
pub const fn strip_whitespace(&self) -> bool
Returns whether whitespace is stripped from chunk edges.
Sourcepub const fn keep_separator(&self) -> KeepSeparator
pub const fn keep_separator(&self) -> KeepSeparator
Returns the configured separator placement mode.
Sourcepub fn validate(&self) -> Result<(), SplitterError>
pub fn validate(&self) -> Result<(), SplitterError>
Validates the configuration.
§Errors
Returns SplitterError::InvalidChunkSize when chunk_size is zero,
and SplitterError::ChunkOverlapExceedsChunkSize when the overlap is
larger than the chunk size.
Sourcepub fn split_text(&self, text: &str) -> Vec<Chunk>
pub fn split_text(&self, text: &str) -> Vec<Chunk>
Splits text into chunks.
§Panics
Panics if the configuration is invalid. Use Self::try_split_text for
a fallible variant.
Sourcepub fn try_split_text(&self, text: &str) -> Result<Vec<Chunk>, SplitterError>
pub fn try_split_text(&self, text: &str) -> Result<Vec<Chunk>, SplitterError>
Splits text into chunks, returning configuration errors instead of panicking.
§Errors
Returns a SplitterError if the configuration is invalid.
Sourcepub fn split_text_content(&self, text: &str) -> Vec<String>
pub fn split_text_content(&self, text: &str) -> Vec<String>
Splits text into chunk contents without metadata.
§Panics
Panics if the configuration is invalid. Use
Self::try_split_text_content for a fallible variant.
Sourcepub fn try_split_text_content(
&self,
text: &str,
) -> Result<Vec<String>, SplitterError>
pub fn try_split_text_content( &self, text: &str, ) -> Result<Vec<String>, SplitterError>
Splits text into contents, returning configuration errors instead of panicking.
§Errors
Returns a SplitterError if the configuration is invalid.
Sourcepub fn split_text_par(&self, text: &str) -> Vec<Chunk>
pub fn split_text_par(&self, text: &str) -> Vec<Chunk>
Parallel version of Self::split_text.
Requires the rayon feature. The result is identical to
Self::split_text.
§Panics
Panics if the configuration is invalid. Use
Self::try_split_text_par for a fallible variant.
Sourcepub fn try_split_text_par(
&self,
text: &str,
) -> Result<Vec<Chunk>, SplitterError>
pub fn try_split_text_par( &self, text: &str, ) -> Result<Vec<Chunk>, SplitterError>
Fallible parallel version of Self::split_text.
§Errors
Returns a SplitterError if the configuration is invalid.
Sourcepub fn split_text_content_par(&self, text: &str) -> Vec<String>
pub fn split_text_content_par(&self, text: &str) -> Vec<String>
Parallel version of Self::split_text_content.
Requires the rayon feature.
§Panics
Panics if the configuration is invalid.
Sourcepub fn try_split_text_content_par(
&self,
text: &str,
) -> Result<Vec<String>, SplitterError>
pub fn try_split_text_content_par( &self, text: &str, ) -> Result<Vec<String>, SplitterError>
Fallible parallel version of Self::split_text_content.
§Errors
Returns a SplitterError if the configuration is invalid.
Sourcepub const fn is_parallel(&self) -> bool
pub const fn is_parallel(&self) -> bool
Returns whether the rayon feature is enabled.
Sourcepub fn split_documents(&self, documents: &[Document]) -> Vec<Document>
pub fn split_documents(&self, documents: &[Document]) -> Vec<Document>
Splits documents, preserving their metadata.
When Self::with_add_start_index is enabled, start_index and
end_index are inserted into each chunk document’s metadata as decimal
strings.
§Panics
Panics if the configuration is invalid. Use
Self::try_split_documents for a fallible variant.
Sourcepub fn try_split_documents(
&self,
documents: &[Document],
) -> Result<Vec<Document>, SplitterError>
pub fn try_split_documents( &self, documents: &[Document], ) -> Result<Vec<Document>, SplitterError>
Fallible version of Self::split_documents.
§Errors
Returns a SplitterError if the configuration is invalid.
Sourcepub fn create_documents(&self, texts: &[&str]) -> Vec<Document>
pub fn create_documents(&self, texts: &[&str]) -> Vec<Document>
Creates documents from plain texts with empty metadata.
§Panics
Panics if the configuration is invalid. Use
Self::try_create_documents for a fallible variant.
Sourcepub fn try_create_documents(
&self,
texts: &[&str],
) -> Result<Vec<Document>, SplitterError>
pub fn try_create_documents( &self, texts: &[&str], ) -> Result<Vec<Document>, SplitterError>
Fallible version of Self::create_documents.
§Errors
Returns a SplitterError if the configuration is invalid.
Sourcepub fn markdown() -> Result<Self, SplitterError>
pub fn markdown() -> Result<Self, SplitterError>
Creates a Markdown-aware splitter.
Requires the language feature.
§Errors
Returns SplitterError::InvalidRegex if a preset pattern cannot be
compiled.
Trait Implementations§
Auto Trait Implementations§
impl !RefUnwindSafe for RecursiveCharacterTextSplitter
impl !UnwindSafe for RecursiveCharacterTextSplitter
impl Freeze for RecursiveCharacterTextSplitter
impl Send for RecursiveCharacterTextSplitter
impl Sync for RecursiveCharacterTextSplitter
impl Unpin for RecursiveCharacterTextSplitter
impl UnsafeUnpin for RecursiveCharacterTextSplitter
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more