Struct text_splitter::ChunkConfig

pub struct ChunkConfig<Sizer>
where
    Sizer: ChunkSizer,
{ /* private fields */ }

Configuration for how chunks should be created

Implementations

impl ChunkConfig<Characters>

pub fn new(capacity: impl Into<ChunkCapacity>) -> Self

Create a basic configuration for chunking with only the required value a chunk capacity.

By default, chunk sizes will be calculated based on the number of characters in each chunk. You can set a custom chunk sizer by calling Self::with_sizer.

By default, chunks will be trimmed. If you want to preserve whitespace, call Self::with_trim and set it to false.

impl<Sizer> ChunkConfig<Sizer>

where Sizer: ChunkSizer,

pub fn capacity(&self) -> &ChunkCapacity

Retrieve a reference to the chunk capacity for this configuration.

pub fn overlap(&self) -> usize

Retrieve the amount of overlap between chunks.

pub fn with_overlap(self, overlap: usize) -> Result<Self, ChunkConfigError>

Set the amount of overlap between chunks.

Errors

Will return an error if the overlap is larger than or equal to the chunk capacity.

pub fn sizer(&self) -> &Sizer

Retrieve a reference to the chunk sizer for this configuration.

pub fn with_sizer<S: ChunkSizer>(self, sizer: S) -> ChunkConfig<S>

Set a custom chunk sizer to use for determining the size of each chunk

use text_splitter::{Characters, ChunkConfig};

let config = ChunkConfig::new(512).with_sizer(Characters);

pub fn trim(&self) -> bool

Whether chunkd should have whitespace trimmed from the beginning and end or not.

pub fn with_trim(self, trim: bool) -> Self

Specify whether chunks should have whitespace trimmed from the beginning and end or not.

If false (default), joining all chunks should return the original string. If true, all chunks will have whitespace removed from beginning and end.

use text_splitter::ChunkConfig;

let config = ChunkConfig::new(512).with_trim(false);

Trait Implementations

impl<Sizer> Debug for ChunkConfig<Sizer>

where Sizer: ChunkSizer + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T> From<T> for ChunkConfig<Characters>

where T: Into<ChunkCapacity>,

fn from(capacity: T) -> Self

Converts to this type from the input type.