semtree-parse 0.3.0

Tree-sitter parsing and chunk extraction for semtree
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

# semtree-parse

Tree-sitter parsing and chunk extraction for [semtree](https://github.com/rustkit-ai/semtree).

Turns source files into structured `Chunk`s (functions, structs, methods, classes) ready to embed. Non-code files fall back to overlapping fixed-size text windows.

## Usage

```toml
[dependencies]
semtree-parse = "0.2"
semtree-core  = "0.2"
```

```rust
use semtree_core::Language;
use semtree_parse::{parse_and_extract, parse_and_extract_file};

// From a string with a known language:
let chunks = parse_and_extract(source, Language::Rust)?;

// Or from a path, with the language inferred from the extension:
let chunks = parse_and_extract_file("src/lib.rs".as_ref())?;
```

## API

| Item | Purpose |
|---|---|
| `parse_and_extract(source, lang)` | Parse a string and extract chunks |
| `parse_and_extract_file(path)` | Parse a file, inferring the language from its extension |
| `extract_file(path)` | Extract chunks, using the text-window fallback for non-code files |
| `SemtreeParser` / `ParsedTree` | Lower-level tree-sitter parsing |
| `Extractor` | Language-aware chunk extraction from a parsed tree |
| `chunk_text` / `is_text_file` | Fixed-window chunking for plain-text and config files |

## Supported languages

Rust, Python, TypeScript, JavaScript, and Go extract structured chunks (functions, types, methods). Other files (`.md`, `.json`, `.toml`, `.yaml`, ...) are chunked into overlapping line windows. See the [workspace README](https://github.com/rustkit-ai/semtree) for the full support matrix.

## License

MIT

Part of [rustkit-ai](https://github.com/rustkit-ai) - open source Rust tools for the AI development era.