slabs 0.1.2

Text chunking for RAG: fixed, sentence, recursive, and semantic strategies
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

# slabs

[![crates.io](https://img.shields.io/crates/v/slabs.svg)](https://crates.io/crates/slabs)
[![Documentation](https://docs.rs/slabs/badge.svg)](https://docs.rs/slabs)
[![CI](https://github.com/arclabs561/slabs/actions/workflows/ci.yml/badge.svg)](https://github.com/arclabs561/slabs/actions/workflows/ci.yml)

Text chunking for RAG pipelines.

Dual-licensed under MIT or Apache-2.0.

## Quickstart

```toml
[dependencies]
slabs = "0.1.0"
```

```rust
use slabs::{Chunker, RecursiveChunker};

let chunker = RecursiveChunker::prose(500);
let text = "Your long document here...";
let slabs = chunker.chunk(text);

for slab in slabs {
    println!("[{}..{}]: {}", slab.start, slab.end, slab.text);
}
```

## Strategies

| Strategy | Use Case | Complexity |
|----------|----------|------------|
| Fixed | Homogeneous content, baselines | $O(n)$ |
| Sentence | Prose, articles | $O(n)$ |
| Recursive | General-purpose | $O(n \log n)$ |
| Semantic | Topic coherence (`semantic` feature) | $O(nd)$ |
| Late | Contextual embeddings across chunk boundaries | Depends on base chunker |

## Features

| Feature | What it enables |
|---------|----------------|
| `semantic` | Semantic chunker (requires `fastembed`, `innr`, `textprep`) |
| `code` | Code-aware chunker via tree-sitter (Rust, Python, TypeScript, Go) |
| `cli` | `slabs` CLI binary |