snipsplit-core 0.1.1

Pure-Rust core for snipsplit: token-aware text chunker for RAG ingestion.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# snipsplit-core

Pure-Rust core for [snipsplit](https://github.com/MukundaKatta/snipsplit):
a token-aware text chunker for RAG ingestion.

```rust
use snipsplit_core::{ChunkConfig, Chunker};

let chunker = Chunker::new(ChunkConfig {
    max_tokens: 256,
    overlap_tokens: 32,
    ..Default::default()
})?;
let chunks = chunker.split("Some long document...")?;
for c in &chunks {
    println!("{}..{} ({} tokens)", c.start, c.end, c.token_count);
}
# Ok::<(), snipsplit_core::ChunkerError>(())
```

## License

Dual-licensed under MIT or Apache-2.0.