semtree-parse 0.3.0

Tree-sitter parsing and chunk extraction for semtree
Documentation

semtree-parse

Tree-sitter parsing and chunk extraction for semtree.

Turns source files into structured Chunks (functions, structs, methods, classes) ready to embed. Non-code files fall back to overlapping fixed-size text windows.

Usage

[dependencies]
semtree-parse = "0.2"
semtree-core  = "0.2"

use semtree_core::Language;
use semtree_parse::{parse_and_extract, parse_and_extract_file};

// From a string with a known language:
let chunks = parse_and_extract(source, Language::Rust)?;

// Or from a path, with the language inferred from the extension:
let chunks = parse_and_extract_file("src/lib.rs".as_ref())?;

API

Item Purpose
parse_and_extract(source, lang) Parse a string and extract chunks
parse_and_extract_file(path) Parse a file, inferring the language from its extension
extract_file(path) Extract chunks, using the text-window fallback for non-code files
SemtreeParser / ParsedTree Lower-level tree-sitter parsing
Extractor Language-aware chunk extraction from a parsed tree
chunk_text / is_text_file Fixed-window chunking for plain-text and config files

Supported languages

Rust, Python, TypeScript, JavaScript, and Go extract structured chunks (functions, types, methods). Other files (.md, .json, .toml, .yaml, ...) are chunked into overlapping line windows. See the workspace README for the full support matrix.

License

MIT

Part of rustkit-ai - open source Rust tools for the AI development era.