embed_anything 0.6.7

Embed anything at lightning speed
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

//! Text chunking algorithm implementations.
//!
//! Provides low-level chunking algorithms that support the text splitting
//! strategies available in the embedding pipeline. These algorithms balance
//! context preservation with computational efficiency.
//!
//! # Available Algorithms
//!
//! - **Statistical** - Length-based chunking with statistical overlap
//! - **Cumulative** - Boundary-aware accumulation chunking
//!
//! # Usage
//!
//! These algorithms are used internally by the embedding pipeline.
//! End users configure chunking through [`TextEmbedConfig`].
//!
//! [`TextEmbedConfig`]: crate::config::TextEmbedConfig

pub mod cumulative;
pub mod statistical;