Struct ChunkCode 

pub struct ChunkCode { /* private fields */ }

Available on crate feature tree-sitter only.

The ChunkCode struct is responsible for chunking code into smaller pieces based on the specified language and chunk size.

It uses tree-sitter under the hood, and tries to split the code into smaller, meaningful chunks.

Example

// Chunk rust code with a maximum chunk size of 1000 bytes.
ChunkCode::try_for_language_and_chunk_size(SupportedLanguages::Rust, 1000);

// Chunk python code with a minimum chunk size of 500 bytes and maximum chunk size of 2048.
// Smaller chunks than 500 bytes will be discarded.
ChunkCode::try_for_language_and_chunk_size(SupportedLanguages::Python, 500..2048);

Implementations

impl ChunkCode

pub fn builder() -> ChunkCodeBuilder

pub fn try_for_language(lang: impl TryInto<SupportedLanguages>) -> Result<Self>

Tries to create a ChunkCode instance for a given programming language.

Parameters

• lang: The programming language to be used for chunking. It should implement TryInto<SupportedLanguages>.

Returns

• Result<Self>: Returns an instance of ChunkCode if successful, otherwise returns an error.

Errors

• Returns an error if the language is not supported or if the CodeSplitter fails to build.

pub fn try_for_language_and_chunk_size( lang: impl TryInto<SupportedLanguages>, chunk_size: impl Into<ChunkSize>, ) -> Result<Self>

Tries to create a ChunkCode instance for a given programming language and chunk size.

Parameters

• lang: The programming language to be used for chunking. It should implement TryInto<SupportedLanguages>.
• chunk_size: The size of the chunks. It should implement Into<ChunkSize>.

Returns

• Result<Self>: Returns an instance of ChunkCode if successful, otherwise returns an error.

Errors

• Returns an error if the language is not supported, if the chunk size is invalid, or if the CodeSplitter fails to build.

pub fn with_concurrency(self, concurrency: usize) -> Self

Trait Implementations

impl ChunkerTransformer for ChunkCode

fn transform_node<'life0, 'async_trait>( &'life0 self, node: TextNode, ) -> Pin<Box<dyn Future<Output = IndexingStream<String>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Transforms a TextNode by splitting its code chunk into smaller pieces.

Parameters

• node: The TextNode containing the code chunk to be split.

Returns

• IndexingStream: A stream of TextNode instances, each containing a smaller chunk of code.

Errors

• If the code splitting fails, an error is sent downstream.

type Input = String

type Output = String

fn concurrency(&self) -> Option<usize>

Overrides the default concurrency of the pipeline

fn name(&self) -> &'static str

impl Clone for ChunkCode

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkCode

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.