Struct text_splitter::ChunkCapacity

pub struct ChunkCapacity { /* private fields */ }

Describes the valid chunk size(s) that can be generated.

The desired size is the target size for the chunk. In most cases, this will also serve as the maximum size of the chunk. It is always possible that a chunk may be returned that is less than the desired value, as adding the next piece of text may have made it larger than the desired capacity.

The max size is the maximum possible chunk size that can be generated. By setting this to a larger value than desired, it means that the chunk should be as close to desired as possible, but can be larger if it means staying at a larger semantic level.

The splitter will consume text until at maxumum somewhere between desired and max, if they differ, but never above max.

If you need to ensure a fixed size, set desired and max to the same value. For example, if you are trying to maximize the context window for an embedding.

If you are loosely targeting a size, but have some extra room, for example in a RAG use case where you roughly want a certain part of a document, you can set max to your absolute maxumum, and the splitter can stay at a higher semantic level when determining the chunk.

Implementations

impl ChunkCapacity

pub fn new(size: usize) -> Self

Create a new ChunkCapacity with the same desired and max size.

pub fn desired(&self) -> usize

The desired size is the target size for the chunk. In most cases, this will also serve as the maximum size of the chunk. It is always possible that a chunk may be returned that is less than the desired value, as adding the next piece of text may have made it larger than the desired capacity.

pub fn max(&self) -> usize

The max size is the maximum possible chunk size that can be generated. By setting this to a larger value than desired, it means that the chunk should be as close to desired as possible, but can be larger if it means staying at a larger semantic level.

pub fn with_max(self, max: usize) -> Result<Self, ChunkCapacityError>

If you need to ensure a fixed size, set desired and max to the same value. For example, if you are trying to maximize the context window for an embedding.

If you are loosely targeting a size, but have some extra room, for example in a RAG use case where you roughly want a certain part of a document, you can set max to your absolute maxumum, and the splitter can stay at a higher semantic level when determining the chunk.

Errors

If the max size is less than the desired size, an error is returned.

pub fn fits(&self, chunk_size: usize) -> Ordering

Validate if a given chunk fits within the capacity

• Ordering::Less indicates more could be added
• Ordering::Equal indicates the chunk is within the capacity range
• Ordering::Greater indicates the chunk is larger than the capacity

Trait Implementations

impl Clone for ChunkCapacity

fn clone(&self) -> ChunkCapacity

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkCapacity

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Range<usize>> for ChunkCapacity

fn from(range: Range<usize>) -> Self

Converts to this type from the input type.

impl From<RangeFrom<usize>> for ChunkCapacity

fn from(range: RangeFrom<usize>) -> Self

Converts to this type from the input type.

impl From<RangeFull> for ChunkCapacity

fn from(_: RangeFull) -> Self

Converts to this type from the input type.

impl From<RangeInclusive<usize>> for ChunkCapacity

fn from(range: RangeInclusive<usize>) -> Self

Converts to this type from the input type.

impl From<RangeTo<usize>> for ChunkCapacity

fn from(range: RangeTo<usize>) -> Self

Converts to this type from the input type.

impl From<RangeToInclusive<usize>> for ChunkCapacity

fn from(range: RangeToInclusive<usize>) -> Self

Converts to this type from the input type.

impl From<usize> for ChunkCapacity

fn from(size: usize) -> Self

Converts to this type from the input type.

impl PartialEq for ChunkCapacity

fn eq(&self, other: &ChunkCapacity) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for ChunkCapacity

impl StructuralPartialEq for ChunkCapacity