Struct TokenChunker

pub struct TokenChunker { /* private fields */ }

Chunkers

Module to contain all the methods of chunking allowing for prepping text before embedding and storing it.

TokenChunker

This struct allows you to do fixed size chunking based on the number of tokens in each chunk. We build around specific embedding models and based on which embedding model is being used we will use the correlating tokenizer.

Examples

use rag_toolchain::chunkers::*;
use rag_toolchain::common::*;
use std::num::NonZeroUsize;

fn generate_chunks() {
    let raw_text: &str = "This is a test string";
    let window_size: usize = 1;
    let chunk_size: NonZeroUsize = NonZeroUsize::new(2).unwrap();

    const EMBEDDING_MODEL: OpenAIEmbeddingModel = OpenAIEmbeddingModel::TextEmbedding3Small;

    let chunker: TokenChunker = TokenChunker::try_new(
        chunk_size,
        window_size,
        EMBEDDING_MODEL,
    )
    .unwrap();

    let chunks: Chunks = chunker.generate_chunks(raw_text).unwrap();
}

Implementations

impl TokenChunker

pub fn try_new( chunk_size: NonZeroUsize, chunk_overlap: usize, embedding_model: impl EmbeddingModel, ) -> Result<Self, TokenChunkingError>

TokenChunker::try_new

Arguments

• chunk_size: NonZeroUsize - The size in tokens of each chunk
• chunk_overlap: usize - The number of tokens that overlap between each chunk
• embedding_model: impl EmbeddingModel - The embedding model to use, this tells us what tokenizer to use

Errors

• [ChunkingError::InvalidChunkSize] - Chunk size must be smaller than the maximum number of tokens
• [ChunkingError::ChunkOverlapTooLarge] - Chunk overlap must be smaller than chunk size

Returns

• TokenChunker - The token chunker

Trait Implementations

impl Chunker for TokenChunker

fn generate_chunks(&self, raw_text: &str) -> Result<Chunks, Self::ErrorType>

TokenChunker::generate_chunks

function to generate chunks from raw text

Arguments

• raw_text: &str - The raw text to generate chunks from

Errors

• [ChunkingError::TokenizationError] - Unable to tokenize text

Returns

Chunks - The generated chunks

type ErrorType = TokenChunkingError