Expand description
Tokenizing text for lexical (BM25) search, shared by every backend: the
SurrealDB full-text query and the in-memory index of
VectorDB. Language-independent: no stopword lists.
Words too common to help are found from the data instead, by document
frequency (VectorDB::query_terms).
Functions§
- tokens
- The words of
text, in order and with repeats: lowercased, split on anything that is not a letter or digit (any script).