Expand description
§Searchlight - 高性能中英文全文搜索引擎
searchlight 是一个功能丰富的 Rust 全文搜索引擎库,支持:
- 中英文分词: 中文 n-gram 分词 + 英文/数字分词
- 倒排索引: 高效的检索数据结构
- 布尔查询: AND/OR/NOT 组合查询
- 模糊匹配: 基于编辑距离的容错搜索
- 拼音搜索: 用拼音搜索中文内容
- 精准短语搜索: 双引号短语匹配
- BM25 排序: 专业的相关性排序算法
- 搜索高亮: 结果自动生成高亮片段
- 批量检索: 一次处理多个查询
- 自动补全: 基于前缀的搜索建议
§快速开始
use searchlight::SearchEngine;
let mut engine = SearchEngine::new();
// 索引文档
engine.index("Rust 是一门现代系统编程语言,安全且高效");
engine.index("Go 语言以其简洁和并发性能著称");
engine.index("Python 是数据科学和 AI 领域的首选语言");
// 搜索
let results = engine.search("编程语言").unwrap();
for r in &results {
println!("[{}] {}", r.score, r.snippet.as_deref().unwrap_or(&r.document));
}
// 拼音搜索
let results = engine.search_pinyin("biancheng").unwrap();
// 模糊搜索
let results = engine.search_fuzzy("programing", 2).unwrap();Structs§
- Bm25
Params - BM25 ranking parameters
- Executor
Context - Context passed to the query executor.
- Executor
Limits - Limits applied during query evaluation.
- Fuzzy
Match - A fuzzy match result
- Highlighter
- Highlighter
Config - Inverted
Index - The inverted index: term -> list of (document_id, positions)
- Match
Reason - Why a document matched the query (retrieval-level explain for agents).
- Parsed
Query - Parsed query with associated metadata
- Pinyin
Converter - Convert Chinese text to pinyin tokens
- Pinyin
Index - Build a pinyin search index: maps pinyin representations to original terms
- Query
Evaluation - Result of evaluating a query AST against the index.
- Query
Executor - Evaluates
QueryOpAST with correct boolean semantics. - Query
Parser - Query parser for the search engine. Supports:
- Ranker
- Rank documents using BM25 scoring with LRU-cached IDF and composite bonuses.
- Related
Suggestion - A deterministic related term suggestion derived from already indexed documents.
- Score
Breakdown - Per-component score breakdown (useful for debugging and agent explain).
- Scoring
Input - Input for the combined scoring pipeline.
- Search
Engine - The main search engine.
- Search
Options - Search options for fine-grained control
- Search
Result - A search result containing the matched document and metadata
- Snippet
- A highlighted snippet from a document
- Token
Enums§
- QueryOp
- Represents different types of query operations
- Searchlight
Error - Unified error type for searchlight operations.
- Token
Kind
Functions§
- contains_
chinese - damerau_
levenshtein - Compute Damerau-Levenshtein distance (includes transpositions).
- explain_
document - Collect retrieval-level match reasons for a document against a query AST.
- fuzzy_
match - Find fuzzy matches for a query term from a candidate list, within max distance.
- fuzzy_
match_ limited - Find fuzzy matches with a hard result cap. This prevents typo-search from collecting a huge result vector on large indexes.
- fuzzy_
match_ with_ prefix - Find fuzzy matches with prefix boost (matches starting with same prefix score higher)
- is_
fuzzy_ match - Check if two strings are a fuzzy match within tolerance
- jaccard_
similarity - Compute Jaccard similarity between two token sets (for bag-of-words comparison)
- lcs_
similarity - Longest Common Subsequence ratio between two strings
- levenshtein_
distance - Compute Levenshtein (edit) distance between two strings. Distance = minimum number of single-character edits (insertions, deletions, substitutions) to change a into b.
- pinyin_
matches - Check if a pinyin query matches Chinese text
- tokenize
- Low-memory tokenizer for mixed Chinese / English text.
- tokenize_
chars - tokenize_
ngrams