pub struct ChunkConfig {
pub max_tokens: usize,
pub overlap_tokens: usize,
pub normalization: Normalization,
pub snap_words: bool,
pub include_context: bool,
pub merge_blocks: bool,
pub strict: bool,
}Expand description
Chunking options.
§include_context selects the chunking mode
true(default): the Markdown block parser runs. Headings are tracked and attached to every chunk asheaders; the compact label[ctx:A > B]produced bycontext_prefixcounts againstmax_tokens(at most about a quarter of the budget is reserved for it). Fenced code blocks and list items are kept intact when they fit. Other constructs (tables, block quotes, HTML blocks, indented code) are ordinary blank-line-separated text blocks with no special handling.false(“flat mode”):crate::smart_chunkandcrate::smart_chunk_mappeddo not parse Markdown. The normalized text is cut into token windows (withoverlap_tokenshonoured) that prefer paragraph, line and word boundaries. Fences, lists and tables are not protected, headings are ordinary text,headersis empty andmerge_blockshas no effect.crate::FileChunkeruses the same whole-document path, so file results equalsmart_chunkon the file’s text at any size. Because that needs the whole file in memory, flat-mode files larger thancrate::FLAT_FILE_MAX_BYTES(32 MiB) are rejected with an error; stream big files withinclude_context = trueinstead.
§strict selects the budget guarantee
true(default): every chunk is re-tokenized exactly as delivered (context prefix plus body) and must fitmax_tokens;token_countis exact. If the context prefix alone prevents the first grapheme from fitting, the headers of that chunk are dropped. If a single grapheme cluster still exceeds the budget, chunking fails withChipError::ChunkTooLarge.false: the final exact check is skipped. This is faster, but there is no guarantee that every chunk fits andtoken_countmay be an estimate for chunks without headers. An unsplittable grapheme cluster larger than the budget becomes its own oversized chunk.
Fields§
§max_tokens: usizeToken budget per chunk (context prefix included). 0 disables the limit.
overlap_tokens: usizeTokens repeated at the start of the next chunk when text has to be split. Must be
smaller than max_tokens (see ChunkConfig::validate).
normalization: NormalizationUnicode normalization applied before chunking; offsets are mapped back to the input.
snap_words: boolPrefer cutting at word/sentence boundaries instead of at an arbitrary token when a hard split is unavoidable.
include_context: boolChunking mode: structure-aware with header context (true) or flat (false).
See the type-level documentation.
merge_blocks: boolPack adjacent Markdown blocks into one chunk while they fit. When false, every
block starts a new chunk. Only relevant when include_context is true.
strict: boolBudget guarantee: exact re-count of every delivered chunk (true) or best effort
(false). See the type-level documentation.
Implementations§
Trait Implementations§
Source§impl Clone for ChunkConfig
impl Clone for ChunkConfig
Source§impl Debug for ChunkConfig
impl Debug for ChunkConfig
Auto Trait Implementations§
impl Forget for ChunkConfig
impl Freeze for ChunkConfig
impl Move for ChunkConfig
impl RefUnwindSafe for ChunkConfig
impl Send for ChunkConfig
impl Sync for ChunkConfig
impl Unpin for ChunkConfig
impl UnsafeUnpin for ChunkConfig
impl UnwindSafe for ChunkConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more