Struct Analyzer 

pub struct Analyzer { /* private fields */ }

A compiled analyzer.

Construction validates the configuration once so Analyzer::analyze cannot fail on well-formed UTF-8.

Implementations

impl Analyzer

pub fn new(config: TokenizerConfig) -> Result<Self, TokenizerError>

Compiles a configuration.

Errors

Returns TokenizerError::Vocab when the vocabulary is malformed.

pub const fn epoch(&self) -> TokenizerEpoch

Returns the epoch identifying this analyzer’s behaviour.

pub const fn config(&self) -> &TokenizerConfig

Returns the configuration.

pub fn analyze(&self, text: &str) -> Vec<Token>

Analyzes text into tokens.

Index-time and query-time analysis are this same call.

pub fn analyze_bytes(&self, bytes: &[u8]) -> Result<Vec<Token>, TokenizerError>

Analyzes bytes, rejecting invalid UTF-8 with a typed error.

Errors

Returns TokenizerError::InvalidUtf8 rather than lossily converting, and TokenizerError::TextTooLong rather than truncating.

Trait Implementations

impl Clone for Analyzer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Analyzer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.