Struct SemanticChunkingConfiguration 

#[non_exhaustive]
pub struct SemanticChunkingConfiguration {
    pub max_tokens: i32,
    pub buffer_size: i32,
    pub breakpoint_percentile_threshold: i32,
}

Settings for semantic document chunking for a data source. Semantic chunking splits a document into smaller documents based on groups of similar content derived from the text with natural language processing.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

max_tokens: i32

The maximum number of tokens that a chunk can contain.

buffer_size: i32

The buffer size.

breakpoint_percentile_threshold: i32

The dissimilarity threshold for splitting chunks.

Implementations

impl SemanticChunkingConfiguration

pub fn max_tokens(&self) -> i32

The maximum number of tokens that a chunk can contain.

pub fn buffer_size(&self) -> i32

The buffer size.

pub fn breakpoint_percentile_threshold(&self) -> i32

The dissimilarity threshold for splitting chunks.

impl SemanticChunkingConfiguration

pub fn builder() -> SemanticChunkingConfigurationBuilder

Creates a new builder-style object to manufacture SemanticChunkingConfiguration.

Trait Implementations

impl Clone for SemanticChunkingConfiguration

fn clone(&self) -> SemanticChunkingConfiguration

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SemanticChunkingConfiguration

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for SemanticChunkingConfiguration

fn eq(&self, other: &SemanticChunkingConfiguration) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for SemanticChunkingConfiguration