julienne 0.1.0

Range-preserving Rust text chunkers for retrieval and embedding pipelines
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

# Structured Chunks And Offsets

Julienne's structured APIs return `TextChunk<'a>`. A structured chunk borrows
from the original input and carries byte and character offsets.

```rust
use julienne::RecursiveCharacterTextSplitter;

let input = "Intro.\n\nDetails with cafe.";
let splitter = RecursiveCharacterTextSplitter::new(80, 10);

for chunk in splitter.split_chunks(input) {
    assert_eq!(&input[chunk.start_byte..chunk.end_byte], chunk.text);
}
```

## Offset Contract

For every `TextChunk` returned by a structured API:

```text
&input[chunk.start_byte..chunk.end_byte] == chunk.text
```

`start_byte` and `end_byte` are byte indexes into the original string.
`start_char` and `end_char` are character offsets counted from the start of the
same string. `measured_length` is the length under the splitter's active sizing
function.

## API Shapes

- `split_text`: returns owned `String` chunks for convenience.
- `split_chunks`: returns collected `TextChunk<'_>` values.
- `chunks`: streams structured chunks for splitters that can operate
  incrementally.
- `try_split_text` / `try_split_chunks`: return `Result` for fallible chunkers
  or fallible integrations.

Use structured chunks whenever downstream systems need traceability back to the
source text, citation spans, UI highlighting, or deterministic deduplication.