Crate paradedb_tantivy
source ·Expand description
tantivy
Tantivy is a search engine library.
Think Lucene
, but in Rust.
// First we need to define a schema ...
// `TEXT` means the field should be tokenized and indexed,
// along with its term frequency and term positions.
//
// `STORED` means that the field will also be saved
// in a compressed, row-oriented key-value store.
// This store is useful to reconstruct the
// documents that were selected during the search phase.
let mut schema_builder = Schema::builder();
let title = schema_builder.add_text_field("title", TEXT | STORED);
let body = schema_builder.add_text_field("body", TEXT);
let schema = schema_builder.build();
// Indexing documents
let index = Index::create_in_dir(index_path, schema.clone())?;
// Here we use a buffer of 100MB that will be split
// between indexing threads.
let mut index_writer = index.writer(100_000_000)?;
// Let's index one documents!
index_writer.add_document(doc!(
title => "The Old Man and the Sea",
body => "He was an old man who fished alone in a skiff in \
the Gulf Stream and he had gone eighty-four days \
now without taking a fish."
))?;
// We need to call .commit() explicitly to force the
// index_writer to finish processing the documents in the queue,
// flush the current index to the disk, and advertise
// the existence of new documents.
index_writer.commit()?;
// # Searching
let reader = index.reader()?;
let searcher = reader.searcher();
let query_parser = QueryParser::for_index(&index, vec![title, body]);
// QueryParser may fail if the query is not in the right
// format. For user facing applications, this can be a problem.
// A ticket has been opened regarding this problem.
let query = query_parser.parse_query("sea whale")?;
// Perform search.
// `topdocs` contains the 10 most relevant doc ids, sorted by decreasing scores...
let top_docs: Vec<(Score, DocAddress)> =
searcher.search(&query, &TopDocs::with_limit(10))?;
for (_score, doc_address) in top_docs {
// Retrieve the actual content of documents given its `doc_address`.
let retrieved_doc = searcher.doc(doc_address)?;
println!("{}", schema.to_json(&retrieved_doc));
}
A good place for you to get started is to check out the example code ( literate programming / source code)
Re-exports
pub use crate::error::TantivyError;
pub use crate::core::SingleSegmentIndexWriter;
pub use crate::directory::Directory;
pub use crate::indexer::merge_filtered_segments;
pub use crate::indexer::merge_indices;
pub use crate::postings::Postings;
pub use crate::schema::DateOptions;
pub use crate::schema::Document;
pub use crate::schema::Term;
pub use columnar;
pub use query_grammar;
pub use time;
Modules
- Aggregations
- Collectors
- WORM (Write Once Read Many) directory abstraction.
- Definition of Tantivy’s errors and results.
- Column oriented field storage for tantivy.
- The fieldnorm represents the length associated with a given Field of a given document.
- Defines tantivy’s merging strategy
- Tantivy can (if instructed to do so in the schema) store the term positions in a given field. This position is expressed as token ordinal. For instance, In “The beauty and the beast”, the term “the” appears in position 0 and position 3. This information is useful to run phrase queries.
- Postings module (also called inverted index)
- Module containing the different query implementations.
- Schema definition for tantivy’s indices.
- Representations for the space usage of various parts of a Tantivy index.
- Compressed/slow/row-oriented storage for documents.
- Tokenizer are in charge of chopping text into a stream of tokens ready for indexing.
Macros
doc!
is a shortcut that helps buildingDocument
objects.- Enable fail_point if feature is enabled.
Structs
- A date/time value with nanoseconds precision.
DocAddress
contains all the necessary information to identify a document given aSearcher
object.FutureResult
is a handle that makes it possible to wait for the completion of an ongoing task.- Search Index
- IndexBuilder can be used to create an index.
- Meta information about the
Index
. IndexReader
is your entry point to read and search the index.IndexReader
builder- Search Index Settings.
- Settings to presort the documents in an index
IndexWriter
is the user entry-point to add document to an index.- The
Inventory
register and keeps track of all of the objects alive. - The inverted index reader is in charge of accessing the inverted index associated with a specific field.
- A prepared commit
- Holds a list of
SegmentReader
s ready for search. - Identifies the searcher generation accessed by a
Searcher
. - A segment is a piece of the index.
- Uuid identifying a segment.
SegmentMeta
contains simple meta information about a segment.- Entry point to access all of the datastructures of the
Segment
Snippet
Contains a fragment of a document, and some highlighted parts inside it.SnippetGenerator
- Your tracked object.
- Structure version for the index.
Enums
- Precision with which datetimes are truncated when stored in fast fields. This setting is only relevant for fast fields. In the docstore, datetimes are always saved with nanosecond precision.
- Search executor whether search request are single thread or multithread.
- The order to sort by
- Defines when a new version of the index should be reloaded.
- Enum describing each component of a tantivy segment. Each component is stored in its own file, using the pattern
segment_uuid
.component_extension
, except the delete component that takes ansegment_uuid
.delete_opstamp
.component_extension
- UserOperation is an enum type that encapsulates other operation types.
Constants
- Sentinel value returned when a
DocSet
has been entirely consumed.
Traits
- Represents an iterable set of sorted doc ids.
- Has length trait
Warmer
can be used to maintain segment-level state e.g. caches.
Functions
- Maps a
f64
tou64
- Maps a
i64
tou64
- Reverse the mapping given by
f64_to_u64()
. - Reverse the mapping given by
i64_to_u64()
. - Expose the current version of tantivy as found in Cargo.toml during compilation. eg. “0.11.0” as well as the compression scheme used in the docstore.
- Exposes the complete version of tantivy as found in Cargo.toml during compilation as a string. eg. “tantivy v0.11.0, index_format v1, store_compression: lz4”.
Type Aliases
- DatePrecisionDeprecated
- A
u32
identifying a document within a segment. Documents have theirDocId
assigned incrementally, as they are added in the segment. - A u64 assigned to every operation incrementally
- Tantivy result.
- A Score that represents the relevance of the document to the query
- A
SegmentOrdinal
identifies a segment, within aSearcher
orMerger
.