Struct Slab 

pub struct Slab {
    pub text: String,
    pub start: usize,
    pub end: usize,
    pub char_start: Option<usize>,
    pub char_end: Option<usize>,
    pub index: usize,
}

A chunk of text with its position in the original document.

The name “slab” evokes a physical slice of material—concrete, wood, stone. Each slab is a self-contained piece that can be embedded, indexed, and retrieved independently.

Offsets

Primary offsets (start/end) are byte offsets into the original text, matching Rust’s string slicing semantics:

use code_chunker::Slab;

let text = "Hello, world!";
let slab = Slab::new("world", 7, 12, 0);

// The offsets let you recover the original position
assert_eq!(&text[slab.start..slab.end], "world");

Character offsets (char_start/char_end) are automatically populated when using Chunker::chunk. They count Unicode scalar values (chars), useful for NLP systems that index by character position. Only None when using Chunker::chunk_bytes directly.

Overlap Handling

When chunks overlap, adjacent slabs share some text. The index field identifies each slab’s position in the sequence:

Original: "The quick brown fox"
Slab 0:   "The quick b"     [0..11]
Slab 1:   "ck brown fox"    [8..19]  <- overlaps with slab 0
               ^
           overlap region [8..11]

Fields

text: String

The chunk text.

start: usize

Byte offset where this chunk starts in the original document.

end: usize

Byte offset where this chunk ends (exclusive) in the original document.

char_start: Option<usize>

Character offset where this chunk starts (Unicode scalar values). None until with_char_offsets or compute_char_offsets is called.

char_end: Option<usize>

Character offset where this chunk ends (exclusive, Unicode scalar values).

index: usize

Zero-based index of this chunk in the sequence.

Implementations

impl Slab

pub fn new( text: impl Into<String>, start: usize, end: usize, index: usize, ) -> Self

Create a new slab (byte offsets only; char offsets unset).

pub fn with_char_offsets(self, char_start: usize, char_end: usize) -> Self

Set character offsets on this slab.

pub fn len(&self) -> usize

The length of this chunk in bytes.

pub fn char_len(&self) -> usize

The length of this chunk in characters (Unicode scalar values).

pub fn is_empty(&self) -> bool

Whether this chunk is empty.

pub fn span(&self) -> Range<usize>

The byte span of this chunk in the original document.

pub fn char_span(&self) -> Option<Range<usize>>

The character span, if computed.

Trait Implementations

impl Clone for Slab

fn clone(&self) -> Slab

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Slab

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Slab

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Slab

fn eq(&self, other: &Slab) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Slab

impl StructuralPartialEq for Slab