plugmem-core 0.2.0

plugmem engine: data model, temporal facts, indexes (BM25, graph, time, vectors incl. HNSW), hybrid recall, snapshot/journal.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13

//! Input normalization for the tokenizer.

use alloc::string::String;

use super::unicode::UnicodeBackend;

/// Replaces `out` with the NFKC form of `text`.
pub(super) fn normalize_into(backend: &UnicodeBackend, text: &str, out: &mut String) {
    // ICU4X performs the same NFKC identity fast path internally for already
    // normalized input while covering all Unicode versions represented by its
    // compiled data.
    backend.normalize_into(text, out);
}