Structs§
- Chunk
- Chunk
Config - Configuration for chunking behavior
- Chunk
Metadata - Stride
Info - Information about chunk striding for large chunks that exceed token limits
- Token
Estimator - Import token estimation from ck-embed Simple token estimation for code and text This is a rough approximation since we don’t have access to the actual model tokenizer
Enums§
Functions§
- chunk_
text - chunk_
text_ with_ config - chunk_
text_ with_ model - New function that accepts model name for model-specific chunking
- get_
model_ chunk_ config - Get model-specific chunk configuration (target_tokens, overlap_tokens) Balanced for precision vs context - larger models can handle bigger chunks but not too big