pub struct LateChunkConfig {
pub chunk_size: usize,
pub chunk_overlap: usize,
}Expand description
Configuration for late chunking.
Fields§
§chunk_size: usizeChunk size in bytes (chunks are byte windows over the document).
chunk_overlap: usizeOverlap between consecutive chunks in bytes.
Implementations§
Source§impl LateChunkConfig
impl LateChunkConfig
Sourcepub fn with_chunk_size(self, chunk_size: usize) -> Self
pub fn with_chunk_size(self, chunk_size: usize) -> Self
Sets the chunk size in bytes.
Sourcepub fn with_chunk_overlap(self, chunk_overlap: usize) -> Self
pub fn with_chunk_overlap(self, chunk_overlap: usize) -> Self
Sets the chunk overlap in bytes.
Sourcepub fn validate(&self) -> Result<(), EmbeddingError>
pub fn validate(&self) -> Result<(), EmbeddingError>
Validates the config: chunk_overlap < chunk_size (otherwise chunks
would not advance).
Sourcepub fn chunk_ranges(&self, text: &str) -> Vec<(usize, usize)>
pub fn chunk_ranges(&self, text: &str) -> Vec<(usize, usize)>
Computes the chunk byte ranges [start, end) over text.
Ranges are monotonically increasing, non-empty, cover the whole
document, and — 0.22.0 C6 fix — are always snapped to UTF-8 char
boundaries (the window advances in bytes for stable sizing, then
edges snap inward; slicing text[start..end] can no longer panic on
CJK / emoji text whose character straddles a window edge).
Trait Implementations§
Source§impl Clone for LateChunkConfig
impl Clone for LateChunkConfig
Source§fn clone(&self) -> LateChunkConfig
fn clone(&self) -> LateChunkConfig
Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source. Read moreSource§impl Debug for LateChunkConfig
impl Debug for LateChunkConfig
Auto Trait Implementations§
impl Freeze for LateChunkConfig
impl RefUnwindSafe for LateChunkConfig
impl Send for LateChunkConfig
impl Sync for LateChunkConfig
impl Unpin for LateChunkConfig
impl UnsafeUnpin for LateChunkConfig
impl UnwindSafe for LateChunkConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more