Struct SemanticCoherenceScorer 

pub struct SemanticCoherenceScorer { /* private fields */ }

Semantic coherence scorer using sentence embeddings

Implementations

impl SemanticCoherenceScorer

pub fn new( config: CoherenceConfig, embedding_provider: Arc<dyn EmbeddingProvider>, ) -> Self

Create a new semantic coherence scorer

pub async fn score_chunk_coherence(&self, text: &str) -> Result<f32>

Score the semantic coherence of a text chunk

Returns a score between 0.0 (incoherent) and 1.0 (highly coherent). High coherence = high cosine similarity between sentence embeddings.

pub async fn find_optimal_split( &self, text: &str, candidate_boundaries: &[usize], ) -> Result<OptimalSplit>

Find optimal split points in text to maximize chunk coherence

Uses a greedy algorithm:

1. Start with no splits
2. Try all candidate split points
3. Pick split that maximizes average chunk coherence
4. Repeat until coherence stops improving

pub fn cosine_similarity(&self, a: &[f32], b: &[f32]) -> f32

Calculate cosine similarity between two embedding vectors

pub fn calculate_adaptive_threshold(&self, text: &str) -> f32

Calculate adaptive threshold based on content characteristics