pub struct ChunkerConfig {
pub max_chars: usize,
pub overlap: usize,
pub markdown: bool,
pub parent_max_chars: Option<usize>,
}Expand description
Chunking parameters.
Fields§
§max_chars: usizeMaximum characters per chunk (excluding overlap). Roughly 4 characters per token.
overlap: usizeCharacters of the previous chunk repeated at the start of the next one.
markdown: boolSplit at Markdown headings and record them as chunk metadata.
parent_max_chars: Option<usize>Parent-child mode: split the text into parents of up to this many characters,
then each parent into children of max_chars (without overlap). Children are
embedded and searched; retrieval returns their parent’s text. Must be larger
than max_chars.
Implementations§
Source§impl ChunkerConfig
impl ChunkerConfig
Sourcepub fn from_tokens(max_tokens: usize, overlap_tokens: usize) -> Self
pub fn from_tokens(max_tokens: usize, overlap_tokens: usize) -> Self
Size limits expressed in approximate tokens (1 token ≈ 4 characters).
Trait Implementations§
Source§impl Clone for ChunkerConfig
impl Clone for ChunkerConfig
Source§impl Debug for ChunkerConfig
impl Debug for ChunkerConfig
Source§impl Default for ChunkerConfig
impl Default for ChunkerConfig
Source§impl<'de> Deserialize<'de> for ChunkerConfig
impl<'de> Deserialize<'de> for ChunkerConfig
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
Source§impl PartialEq for ChunkerConfig
impl PartialEq for ChunkerConfig
Source§impl Serialize for ChunkerConfig
impl Serialize for ChunkerConfig
impl StructuralPartialEq for ChunkerConfig
Auto Trait Implementations§
impl Forget for ChunkerConfig
impl Freeze for ChunkerConfig
impl Move for ChunkerConfig
impl RefUnwindSafe for ChunkerConfig
impl Send for ChunkerConfig
impl Sync for ChunkerConfig
impl Unpin for ChunkerConfig
impl UnsafeUnpin for ChunkerConfig
impl UnwindSafe for ChunkerConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more