Enum Tokenizer 

pub enum Tokenizer {
    Whitespace,
    Standard,
    Letter,
    NGram {
        min_gram: usize,
        max_gram: usize,
    },
    Pattern {
        pattern: String,
    },
    Keyword,
}

Variants

Whitespace

Standard

Letter

NGram

Fields

min_gram: usize

max_gram: usize

Pattern

Fields

pattern: String

Keyword

Implementations

impl Tokenizer

pub fn validate(&self) -> AnalysisResult<()>

Validate configuration without tokenizing input. This is used when an analyzer is registered, while Self::tokenize repeats the checks so deserialized legacy values can never bypass them.

pub fn tokenize(&self, text: &str) -> AnalysisResult<Vec<String>>

pub fn tokenize_with_offsets(&self, text: &str) -> AnalysisResult<AnalyzedText>

Tokenize source text with explicit offsets, positions, and final source coordinates.

pub fn tokenize_with_offsets_budgeted( &self, text: &str, budget: &MemoryBudget, poll: impl FnMut() -> AnalysisResult<()>, ) -> AnalysisResult<Budgeted<AnalyzedText>>

Tokenize with reservations for owned buffers and retained source provenance.

Immutable configuration resources remain separately managed. Built-in word tokenizers and every prepared pattern search poll while scanning; regex fallback workspace reserves its buffers from the caller’s allowance. Emission and projection also poll. Cloning the underlying analyzed value creates separate, unreserved token buffers.

pub fn tokenize_mapped_budgeted( &self, text: &FilteredText<'_>, budget: &MemoryBudget, poll: impl FnMut() -> AnalysisResult<()>, ) -> AnalysisResult<Budgeted<AnalyzedText>>

Tokenize a retained character-filter result under a caller allowance.

Previously prepared source allocations keep their existing owner. Newly built coordinate indexes, token buffers, terms, and retained source copies use budget; failed calls leave the input’s coordinate caches unchanged.

use uqa_analysis::{CharFilter, Tokenizer};
use uqa_core::memory::MemoryBudget;
let budget = MemoryBudget::new(16 * 1024);
let filtered = CharFilter::HTMLStrip.filter_with_offsets_budgeted(
    "<b>韓🙂</b>", &budget, &mut || Ok(()),
)?;
let tokens = Tokenizer::Whitespace.tokenize_mapped_budgeted(
    &filtered, &budget, || Ok(()),
)?;
assert_eq!(tokens.tokens()[0].term(), "韓🙂");
assert_eq!(tokens.tokens()[0].offsets().unwrap().utf8, 3..10);
drop(filtered);
assert!(budget.used() > 0);
drop(tokens);
assert_eq!(budget.used(), 0);

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Tokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Tokenizer

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for Tokenizer

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.