Struct Splitter

pub struct Splitter<T: Sizer> { /* private fields */ }

A struct for splitting code into chunks.

Implementations

impl<T> Splitter<T>

where T: Sizer,

pub fn new(language: Language, sizer: T) -> Result<Self>

Create a new Splitter that counts the size of code chunks with the given sizer.

Example: split by characters

use code_splitter::{CharCounter, Splitter};

let lang = tree_sitter_md::language();
let splitter = Splitter::new(lang, CharCounter).unwrap();
let chunks = splitter.split(b"hello, world!").unwrap();

Example: split by words

use code_splitter::{Splitter, WordCounter};

let lang = tree_sitter_md::language();
let splitter = Splitter::new(lang, WordCounter).unwrap();
let chunks = splitter.split(b"hello, world!").unwrap();

Example: split by tokens with huggingface tokenizer

use code_splitter::Splitter;
use tokenizers::Tokenizer;

let lang = tree_sitter_md::language();
let tokenizer = Tokenizer::from_pretrained("bert-base-cased", None).unwrap();
let splitter = Splitter::new(lang, tokenizer).unwrap();
let chunks = splitter.split(b"hello, world!").unwrap();

Example: split by tokens with tiktoken core BPE

use code_splitter::Splitter;
use tiktoken_rs::cl100k_base;

let lang = tree_sitter_md::language();
let bpe = cl100k_base().unwrap();
let splitter = Splitter::new(lang, bpe).unwrap();
let chunks = splitter.split(b"hello, world!").unwrap();

pub fn with_max_size(self, max_size: usize) -> Self

Set the maximum size of a chunk. The default is 512.

Example: set the maximum size to 256

use code_splitter::{CharCounter, Splitter};

let lang = tree_sitter_md::language();
let splitter = Splitter::new(lang, CharCounter)
  .unwrap()
  .with_max_size(256);
let chunks = splitter.split(b"hello, world!").unwrap();

pub fn split(&self, code: &[u8]) -> Result<Vec<Chunk>>

Split the code into chunks with no larger than max_size.