devgen-splitter 0.4.7

Devgen Splitter is a Rust library that breaks down source code into contextual chunks.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

Devgen Splitter is a Rust library that breaks down source code into contextual chunks. 
It utilizes tree-sitter to identify code entities (such as classes, functions, and methods) and generate chunks with contextual metadata.

[![Crates.io Version](https://img.shields.io/crates/v/devgen-splitter)](https://crates.io/crates/devgen-splitter)
[![codecov](https://codecov.io/github/imotai/devgen-splitter/graph/badge.svg?token=3YSFEZ4JLI)](https://codecov.io/github/imotai/devgen-splitter)
![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/imotai/devgen-splitter/rust.yml)


![splitter](./images/splitter.png)

## Features

- Language-aware code splitting
- Generate chunks with contextual metadata
- Support for multiple programming languages

why devgen-splitter?

If you are building a code search agent, you may want to the LLM to generate the link for related class, struct, enum, etc. Devgen Splitter can help you generate the chunks with contextual metadata.

## Usage


Add `devgen-splitter` to your project:

```bash
cargo add devgen-splitter
```

Basic usage example:

```rust
use devgen_splitter::{SplitOptions, split};
let code = "fn main() { println!(\"Hello, world!\"); }";
let options = SplitOptions { chunk_line_limit: 10};
let chunks = split("example.rs", code, &options).unwrap();
for chunk in chunks {
    println!("Chunk: {:?}", chunk);
}
```
For more examples, go to [examples](./examples)

## Supported Languages

| Language   | Query Rules | Splitter | Test |
|------------|-------------|----------|------|
| Rust       | ✅          | ✅       | ✅   |
| TypeScript | ✅          | ✅       | ✅   |
| Java       | ✅          | ✅       | ✅   |
| Python     | ✅          | ✅       | ✅   |
| Solidity   | ✅          | ✅       | ✅   |
| Go         | 🚧          | 🚧       | 🚧   |
| C++        | 🚧          | 🚧       | 🚧   |
| C          | 🚧          | 🚧       | 🚧   |

More languages coming soon!

## Language Mapping

The following table shows how different code structures are represented across various programming languages and their corresponding tree-sitter query rule names:

| Type       | Tree-sitter Query | Rust     | Java     | TypeScript | Python   | Go       | C++      |
|------------|-------------------|----------|----------|------------|----------|----------|----------|
| Function   | function.definition | function     | N/A   | function/array function   | function     | function  | function |
| Method   | method.definition | method    | method   | method   | method      | method     | method |
| Struct     | struct.declaration | struct  | class    | interface  | class    | struct   | struct   |
| Class      | class.declaration | impl     | class    | class      | class    | N/A      | class    |
| Interface  | interface.declaration | trait | interface| N/A  | N/A      | N/A| N/A      | N/A      |
| Enum       | enum.declaration  | enum     | enum     | enum       | N/A      | N/A      | enum     |


## Development Status

Devgen Splitter is in active development. We welcome community contributions and feedback.