Struct EmbedChunker 

pub struct EmbedChunker { /* private fields */ }

Core chunker for generating embedding chunks

Implementations

impl EmbedChunker

pub fn new(settings: EmbedSettings, limits: ResourceLimits) -> Self

Create a new chunker with the given settings and limits

pub fn with_defaults(settings: EmbedSettings) -> Self

Create a new chunker with default limits

pub fn with_repo_id(self, repo_id: RepoIdentifier) -> Self

Set the repository identifier for multi-tenant RAG

This identifier is attached to all generated chunks, enabling:

• Multi-repository search with proper attribution
• Access control filtering by repository
• Cross-repository dependency tracking

Example

let chunker = EmbedChunker::with_defaults(settings)
    .with_repo_id(RepoIdentifier::new("github.com/myorg", "auth-service"));

pub fn set_repo_id(&mut self, repo_id: RepoIdentifier)

Set the repository identifier (mutable borrow version)

pub fn repo_id(&self) -> &RepoIdentifier

Get the current repository identifier

pub fn chunk_repository_filtered( &self, repo_path: &Path, only_files: &HashSet<PathBuf>, progress: &dyn ProgressReporter, ) -> Result<Vec<EmbedChunk>, EmbedError>

Generate chunks only for specific files in a repository

This is used for git-diff-driven incremental updates where only changed files need to be re-chunked. The only_files set contains relative paths (from repo root) of files to process.

Guarantees

Same as chunk_repository: deterministic, thread-safe, resource-limited.

pub fn chunk_repository( &mut self, repo_path: &Path, progress: &dyn ProgressReporter, ) -> Result<Vec<EmbedChunk>, EmbedError>

Generate all chunks for a repository

Guarantees

1. Deterministic output (same input = same output)
2. Thread-safe parallel processing
3. Resource limits enforced
4. Errors collected, not swallowed

pub fn chunk_repository_streaming<W: Write>( &self, repo_path: &Path, writer: &mut W, progress: &dyn ProgressReporter, ) -> Result<StreamingStats, EmbedError>

Generate chunks in streaming mode, writing each batch to a writer as JSONL.

Instead of collecting all chunks into memory, this method processes files in batches, writes each batch’s chunks to the writer, then drops them. This reduces peak memory from O(all chunks) to O(batch_size worth of chunks).

Determinism

• Files are sorted lexicographically before batching, so batches are processed in a deterministic order.
• Within each batch, chunks are sorted by (file, start_line, end_line, symbol, id).
• Across batch boundaries, ordering is NOT globally sorted (batch N’s last chunk may sort after batch N+1’s first chunk). For full global sorting, use the non-streaming chunk_repository() method.

called_by trade-off

The reverse call graph (called_by field) is populated within each batch only. Cross-batch called_by references are not captured because that would require keeping all chunks in memory. For full called_by coverage, use the non-streaming chunk_repository() method.

Writer protocol

Each chunk is serialized as a single JSON line (JSONL) via serde_json. The caller is responsible for writing any header/footer lines around the chunks.