pub struct SemanticSplitter<E> { /* private fields */ }Expand description
语义分块器
在相邻句相似度低于 breakpoint_threshold 处断块;
累积长度超过 max_chunk_size 时强制断。
Implementations§
Source§impl<E: Embeddings> SemanticSplitter<E>
impl<E: Embeddings> SemanticSplitter<E>
Sourcepub fn new(
embeddings: E,
breakpoint_threshold: f32,
max_chunk_size: usize,
) -> Self
pub fn new( embeddings: E, breakpoint_threshold: f32, max_chunk_size: usize, ) -> Self
创建语义分块器
§参数
embeddings- 嵌入模型breakpoint_threshold- 相邻句相似度断点阈值(0.0–1.0,越低越不易断)max_chunk_size- 单块最大字符数
Sourcepub fn with_defaults(embeddings: E) -> Self
pub fn with_defaults(embeddings: E) -> Self
使用默认参数创建(threshold=0.5, max=1000)
Sourcepub async fn split_text(
&self,
text: &str,
) -> Result<Vec<String>, EmbeddingError>
pub async fn split_text( &self, text: &str, ) -> Result<Vec<String>, EmbeddingError>
异步分块
Sourcepub async fn split_document(
&self,
document: &Document,
) -> Result<Vec<Document>, EmbeddingError>
pub async fn split_document( &self, document: &Document, ) -> Result<Vec<Document>, EmbeddingError>
异步分块文档,保留 metadata(写入 chunk 序号)
Auto Trait Implementations§
impl<E> Freeze for SemanticSplitter<E>where
E: Freeze,
impl<E> RefUnwindSafe for SemanticSplitter<E>where
E: RefUnwindSafe,
impl<E> Send for SemanticSplitter<E>where
E: Send,
impl<E> Sync for SemanticSplitter<E>where
E: Sync,
impl<E> Unpin for SemanticSplitter<E>where
E: Unpin,
impl<E> UnsafeUnpin for SemanticSplitter<E>where
E: UnsafeUnpin,
impl<E> UnwindSafe for SemanticSplitter<E>where
E: UnwindSafe,
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more