Struct Chunker

pub struct Chunker { /* private fields */ }

A struct for chunking texts into segments based on a maximum number of tokens per chunk and a token counter function.

Fields

• chunk_size - The maximum number of tokens that can be in a chunk.
• token_counter - A function that counts the number of tokens in a string.
• splitter - The Splitter instance used to split the text.

Example

use semchunk_rs::Chunker;
let chunker = Chunker::new(4, Box::new(|s: &str| s.len() - s.replace(" ", "").len() + 1));
let text = "The quick brown fox jumps over the lazy dog.";
let chunks = chunker.chunk(text);
assert_eq!(chunks, vec!["The quick brown fox", "jumps over the lazy", "dog."]);

With rust_tokenizers:

use rust_tokenizers::tokenizer::{RobertaTokenizer, Tokenizer};
use semchunk_rs::Chunker;
let tokenizer = RobertaTokenizer::from_file("data/roberta-base-vocab.json", "data/roberta-base-merges.txt", false, false)
   .expect("Error loading tokenizer");
let token_counter = Box::new(move |s: &str| {
   tokenizer.tokenize(s).len()
});
let chunker = Chunker::new(10, token_counter);

Implementations

impl Chunker

pub fn new(chunk_size: usize, token_counter: Box<dyn Fn(&str) -> usize>) -> Self

Creates a new Chunker instance. Uses the default Splitter instance. S

Arguments

• chunk_size - The maximum number of tokens that can be in a chunk.
• token_counter - A function that counts the number of tokens in a string.

Returns

A new Chunker instance.

pub fn splitter(self, splitter: Splitter) -> Self

Sets the splitter for the Chunker instance.

pub fn _chunk(&self, text: &str, recursion_depth: usize) -> Vec<String>

Recursively chunks the given text into segments based on the maximum number of tokens per chunk.

Arguments

• text - A string slice that holds the text to be chunked.
• recursion_depth - The current recursion depth.

Returns

A vector of string slices representing the chunks of the split text.

pub fn merge_splits(&self, splits: &[&str], separator: &str) -> (usize, String)

Merges first N splits into a chunk that has <= chunk_size tokens.

Arguments

• splits - A vector of string slices representing the splits to merge.
• separator - The separator used to split the text.

Returns

A tuple containing:

• The index merging stopped at (not inclusive).
• The merged text.

Examples

use semchunk_rs::Chunker;
let chunker = Chunker::new(4, Box::new(|s: &str| s.len() - s.replace(" ", "").len() + 1));
let splits = vec!["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"];
let separator = " ";
let (split_idx, merged) = chunker.merge_splits(&splits, separator);
assert_eq!(split_idx, 4);
assert_eq!(merged, "The quick brown fox");

pub fn chunk(&self, text: &str) -> Vec<String>

Chunks the given text into segments based on the maximum number of tokens per chunk.

Arguments

• text - A string slice that holds the text to be chunked.

Examples

use semchunk_rs::Chunker;
 
let chunker = Chunker::new(4, Box::new(|s: &str| s.len() - s.replace(" ", "").len() + 1));
let text = "The quick brown fox jumps over the lazy dog.";
let chunks = chunker._chunk(text, 0);
assert_eq!(chunks, vec!["The quick brown fox", "jumps over the lazy", "dog."]);