Skip to main content

RecursiveCharacterTextSplitter

Struct RecursiveCharacterTextSplitter 

Source
pub struct RecursiveCharacterTextSplitter { /* private fields */ }
Expand description

A recursive character text splitter.

See the crate documentation for examples and guarantees.

Implementations§

Source§

impl RecursiveCharacterTextSplitter

Source

pub fn new() -> Self

Creates a splitter with LangChain-compatible defaults.

Default separators: ["\n\n", "\n", " ", ""]

Default chunk size: 4000 characters

Default chunk overlap: 200 characters

Source

pub fn with_separators<I, S>(self, separators: I) -> Self
where I: IntoIterator<Item = S>, S: Into<Cow<'static, str>>,

Sets the separators, tried in priority order.

An empty separator ("") is appended automatically when missing. This guarantees that the splitter can always fall back to Unicode scalar values and honour chunk_size, unlike Python LangChain, which can emit oversized chunks when the final separator does not match. As in Python, separators after the first empty separator are ignored.

Both borrowed and owned separator strings are accepted:

use recursive_text_splitter::RecursiveCharacterTextSplitter;

let dynamic = vec![String::from("\n\n"), String::from("\n"), String::from("")];
let splitter = RecursiveCharacterTextSplitter::new().with_separators(dynamic);
assert_eq!(splitter.chunk_size(), 4000);
Source

pub fn with_regex_separators<I, S>( self, separators: I, ) -> Result<Self, SplitterError>
where I: IntoIterator<Item = S>, S: Into<Cow<'static, str>>,

Sets regex separators, tried in priority order.

The patterns use the Rust regex syntax, which guarantees linear-time matching and therefore cannot suffer from catastrophic backtracking. Patterns that match the empty string are rejected because they do not describe meaningful split points.

Requires the regex feature.

§Errors

Returns SplitterError::InvalidRegex if a pattern is not valid Rust regex syntax or can match the empty string.

Source

pub fn with_language(self, language: Language) -> Result<Self, SplitterError>

Configures separators for a programming or markup language.

Requires the language feature. The preset patterns are regular expressions compatible with the regex crate.

use recursive_text_splitter::{Language, RecursiveCharacterTextSplitter};

let splitter = RecursiveCharacterTextSplitter::new()
    .with_language(Language::Markdown)
    .unwrap()
    .with_chunk_size(500);
§Errors

Returns SplitterError::InvalidRegex if one of the preset patterns is not valid Rust regex syntax.

Source

pub fn with_chunk_size(self, size: usize) -> Self

Sets the maximum chunk size, measured by the configured length function.

Source

pub fn with_chunk_overlap(self, overlap: usize) -> Self

Sets the number of units to overlap between adjacent chunks.

The value is a target, not a hard guarantee: Python-compatible merging can produce duplicate or nested chunks when the overlap is large relative to chunk_size. See the crate documentation.

Source

pub fn with_add_start_index(self, add: bool) -> Self

Enables or disables start_index/end_index byte offsets on chunks.

Source

pub fn with_strip_whitespace(self, strip: bool) -> Self

Enables or disables trimming whitespace from chunk edges.

Source

pub fn with_keep_separator(self, keep: bool) -> Self

Sets whether separators are kept (the Python boolean API).

true is equivalent to KeepSeparator::Start, false to KeepSeparator::None.

Source

pub fn with_keep_separator_mode(self, mode: KeepSeparator) -> Self

Sets where separators are kept in the output.

Source

pub fn with_byte_length_function(self) -> Self

Measures chunks in UTF-8 bytes instead of Unicode scalar values.

This clears any custom length function.

Source

pub fn with_char_length_function(self) -> Self

Measures chunks in Unicode scalar values (the default).

This clears any custom length function.

Source

pub fn with_length_function<F>(self, length_function: F) -> Self
where F: Fn(&str) -> usize + Send + Sync + 'static,

Uses a custom function to measure text length, for example a tokenizer.

The function must be thread-safe because the splitter is Send + Sync. Custom length functions override the built-in character/byte counters.

use recursive_text_splitter::RecursiveCharacterTextSplitter;

let splitter = RecursiveCharacterTextSplitter::new()
    .with_length_function(|text| text.split_whitespace().count())
    .with_chunk_size(10)
    .with_chunk_overlap(0);

assert!(!splitter.split_text_content("one two three").is_empty());
Source

pub fn with_strategy<S>(self) -> Self
where S: SplitterStrategy + Default + Send + Sync + 'static,

Sets the splitting strategy from its Default value.

use recursive_text_splitter::{RecursiveCharacterTextSplitter, SequentialStrategy};

let splitter = RecursiveCharacterTextSplitter::new()
    .with_strategy::<SequentialStrategy>();
Source

pub fn with_strategy_instance<S>(self, strategy: S) -> Self
where S: SplitterStrategy + Send + Sync + 'static,

Installs a specific strategy instance.

Use this instead of Self::with_strategy when the strategy carries configuration. Custom strategies should delegate to SequentialStrategy or implement splitting directly; see SplitterStrategy for details.

Source

pub fn separators(&self) -> impl ExactSizeIterator<Item = &str> + '_

Returns the configured separators, including the implicit character fallback.

Source

pub const fn chunk_size(&self) -> usize

Returns the configured chunk size.

Source

pub const fn chunk_overlap(&self) -> usize

Returns the configured chunk overlap.

Source

pub const fn add_start_index(&self) -> bool

Returns whether chunk byte offsets are enabled.

Source

pub const fn strip_whitespace(&self) -> bool

Returns whether whitespace is stripped from chunk edges.

Source

pub const fn keep_separator(&self) -> KeepSeparator

Returns the configured separator placement mode.

Source

pub fn validate(&self) -> Result<(), SplitterError>

Validates the configuration.

§Errors

Returns SplitterError::InvalidChunkSize when chunk_size is zero, and SplitterError::ChunkOverlapExceedsChunkSize when the overlap is larger than the chunk size.

Source

pub fn split_text(&self, text: &str) -> Vec<Chunk>

Splits text into chunks.

§Panics

Panics if the configuration is invalid. Use Self::try_split_text for a fallible variant.

Source

pub fn try_split_text(&self, text: &str) -> Result<Vec<Chunk>, SplitterError>

Splits text into chunks, returning configuration errors instead of panicking.

§Errors

Returns a SplitterError if the configuration is invalid.

Source

pub fn split_text_content(&self, text: &str) -> Vec<String>

Splits text into chunk contents without metadata.

§Panics

Panics if the configuration is invalid. Use Self::try_split_text_content for a fallible variant.

Source

pub fn try_split_text_content( &self, text: &str, ) -> Result<Vec<String>, SplitterError>

Splits text into contents, returning configuration errors instead of panicking.

§Errors

Returns a SplitterError if the configuration is invalid.

Source

pub fn split_text_par(&self, text: &str) -> Vec<Chunk>

Parallel version of Self::split_text.

Requires the rayon feature. The result is identical to Self::split_text.

§Panics

Panics if the configuration is invalid. Use Self::try_split_text_par for a fallible variant.

Source

pub fn try_split_text_par( &self, text: &str, ) -> Result<Vec<Chunk>, SplitterError>

Fallible parallel version of Self::split_text.

§Errors

Returns a SplitterError if the configuration is invalid.

Source

pub fn split_text_content_par(&self, text: &str) -> Vec<String>

Parallel version of Self::split_text_content.

Requires the rayon feature.

§Panics

Panics if the configuration is invalid.

Source

pub fn try_split_text_content_par( &self, text: &str, ) -> Result<Vec<String>, SplitterError>

Fallible parallel version of Self::split_text_content.

§Errors

Returns a SplitterError if the configuration is invalid.

Source

pub const fn is_parallel(&self) -> bool

Returns whether the rayon feature is enabled.

Source

pub fn split_documents(&self, documents: &[Document]) -> Vec<Document>

Splits documents, preserving their metadata.

When Self::with_add_start_index is enabled, start_index and end_index are inserted into each chunk document’s metadata as decimal strings.

§Panics

Panics if the configuration is invalid. Use Self::try_split_documents for a fallible variant.

Source

pub fn try_split_documents( &self, documents: &[Document], ) -> Result<Vec<Document>, SplitterError>

Fallible version of Self::split_documents.

§Errors

Returns a SplitterError if the configuration is invalid.

Source

pub fn create_documents(&self, texts: &[&str]) -> Vec<Document>

Creates documents from plain texts with empty metadata.

§Panics

Panics if the configuration is invalid. Use Self::try_create_documents for a fallible variant.

Source

pub fn try_create_documents( &self, texts: &[&str], ) -> Result<Vec<Document>, SplitterError>

Fallible version of Self::create_documents.

§Errors

Returns a SplitterError if the configuration is invalid.

Source

pub fn markdown() -> Result<Self, SplitterError>

Creates a Markdown-aware splitter.

Requires the language feature.

§Errors

Returns SplitterError::InvalidRegex if a preset pattern cannot be compiled.

Trait Implementations§

Source§

impl Clone for RecursiveCharacterTextSplitter

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for RecursiveCharacterTextSplitter

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for RecursiveCharacterTextSplitter

Source§

fn default() -> Self

Returns the “default value” for a type. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> Pointable for T

Source§

const ALIGN: usize

The alignment of pointer.
Source§

type Init = T

The type for initializers.
Source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
Source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
Source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
Source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.