pub struct ChunkConfig {
pub target_tokens: usize,
pub overlap_tokens: usize,
pub min_tokens: usize,
}Expand description
How to size chunks.
Fields§
§target_tokens: usizeAim for this many tokens. A single unit longer than this is emitted whole rather than split – cutting a paragraph mid-sentence costs more than an oversized chunk.
overlap_tokens: usizeRepeat units from the previous chunk’s tail, up to this many tokens.
min_tokens: usizeDrop chunks below this. A stray “Read more” is noise in a vector store.
Trait Implementations§
Auto Trait Implementations§
impl Forget for ChunkConfig
impl Freeze for ChunkConfig
impl Move for ChunkConfig
impl RefUnwindSafe for ChunkConfig
impl Send for ChunkConfig
impl Sync for ChunkConfig
impl Unpin for ChunkConfig
impl UnsafeUnpin for ChunkConfig
impl UnwindSafe for ChunkConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more