Struct Chunk 

pub struct Chunk {
    pub id: Option<i64>,
    pub buffer_id: i64,
    pub content: String,
    pub byte_range: Range<usize>,
    pub index: usize,
    pub metadata: ChunkMetadata,
}

Represents a chunk of text from a buffer.

Chunks are created by chunking strategies and contain a portion of buffer content along with metadata about their position and origin.

Examples

use rlm_rs::core::Chunk;

let chunk = Chunk::new(
    1,
    "Hello, world!".to_string(),
    0..13,
    0,
);
assert_eq!(chunk.size(), 13);

Fields

id: Option<i64>

Unique identifier (assigned by storage layer).

buffer_id: i64

ID of the buffer this chunk belongs to.

content: String

Chunk content.

byte_range: Range<usize>

Byte range in the original buffer.

index: usize

Sequential index within the buffer (0-based).

metadata: ChunkMetadata

Chunk metadata.

Implementations

impl Chunk

pub fn new( buffer_id: i64, content: String, byte_range: Range<usize>, index: usize, ) -> Self

Creates a new chunk.

Arguments

• buffer_id - ID of the parent buffer.
• content - Chunk content.
• byte_range - Byte range in the original buffer.
• index - Sequential index within the buffer.

Examples

use rlm_rs::core::Chunk;

let chunk = Chunk::new(1, "content".to_string(), 0..7, 0);
assert_eq!(chunk.buffer_id, 1);
assert_eq!(chunk.index, 0);

pub fn with_strategy( buffer_id: i64, content: String, byte_range: Range<usize>, index: usize, strategy: &str, ) -> Self

Creates a chunk with a specific strategy name.

Arguments

• buffer_id - ID of the parent buffer.
• content - Chunk content.
• byte_range - Byte range in the original buffer.
• index - Sequential index within the buffer.
• strategy - Name of the chunking strategy.

pub const fn size(&self) -> usize

Returns the size of the chunk in bytes.

pub const fn range_size(&self) -> usize

Returns the byte range size.

pub const fn is_empty(&self) -> bool

Checks if the chunk is empty.

pub const fn start(&self) -> usize

Returns the start byte offset in the original buffer.

pub const fn end(&self) -> usize

Returns the end byte offset in the original buffer.

pub const fn set_token_count(&mut self, count: usize)

Sets the token count estimate.

pub const fn estimate_tokens(&self) -> usize

Estimates token count using a simple heuristic.

Uses the approximation of ~4 characters per token for ASCII text. For a more accurate estimate, use Self::estimate_tokens_accurate.

Accuracy

This simple method is typically accurate within 20-30% for English text and code. It tends to undercount for text with many short words and overcount for text with long technical terms.

pub fn estimate_tokens_accurate(&self) -> usize

Estimates token count with improved accuracy.

Uses a more sophisticated heuristic that accounts for:

• Word boundaries (whitespace-separated tokens)
• Punctuation and operators (often separate tokens)
• Non-ASCII characters (typically 1-2 chars per token)

Accuracy

This method is typically accurate within 10-15% for mixed content. For production use requiring exact counts, consider integrating a proper tokenizer like tiktoken-rs.

Performance

This method iterates over the content string, so it’s O(n) where n is the content length. For very large chunks, the simple Self::estimate_tokens method may be preferred.

pub const fn set_line_range(&mut self, start_line: usize, end_line: usize)

Sets the line range in the original buffer.

pub const fn set_has_overlap(&mut self, has_overlap: bool)

Marks this chunk as having overlap with the previous chunk.

pub fn compute_hash(&mut self)

Computes and sets the content hash.

pub fn preview(&self, max_len: usize) -> &str

Returns a preview of the chunk content (first N characters).

Arguments

• max_len - Maximum number of characters to include.

pub const fn overlaps_with(&self, other_range: &Range<usize>) -> bool

Checks if this chunk’s byte range overlaps with another range.

pub fn contains_offset(&self, offset: usize) -> bool

Checks if this chunk’s byte range contains a specific byte offset.

Trait Implementations

impl Clone for Chunk

fn clone(&self) -> Chunk

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Chunk

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Chunk

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for Chunk

fn eq(&self, other: &Chunk) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Chunk

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for Chunk

impl StructuralPartialEq for Chunk