Struct TextSegmenterConfig 

pub struct TextSegmenterConfig {
    pub min_chars: usize,
    pub target_chars: usize,
    pub max_chars: usize,
    pub overlap_chars: usize,
}

Size limits for text segments, in UTF-8 bytes (characters, for ASCII text).

Defaults are chosen for retrieval quality, not to fill the 8K context. A vector that averages several ideas scores lower for a query matching only one of them, so segments follow the author’s own units: every paragraph of at least min_chars is its own segment. Shorter fragments (labels, one-line notes) are merged with their neighbours up to target_chars.

Gemma’s tokenizer averages roughly four characters per token on English prose: 60 characters is a short sentence, the 1,000-character target is about 250 tokens and the 1,800-character maximum about 450.

Fields

min_chars: usize

Paragraphs at least this long are never merged with each other.

target_chars: usize

Stop merging short fragments once a chunk reaches this size.

max_chars: usize

Longer paragraphs are split at sentences. A merged chunk can exceed this slightly, since headings and the gaps between paragraphs are not counted.

overlap_chars: usize

When a long paragraph is cut at a sentence, its last sentence (if no longer than this) is repeated at the start of the next piece. There is no overlap across paragraph breaks the author chose.

Trait Implementations

impl Clone for TextSegmenterConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TextSegmenterConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TextSegmenterConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TextSegmenterConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for TextSegmenterConfig

impl PartialEq for TextSegmenterConfig

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for TextSegmenterConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for TextSegmenterConfig