Struct TokenizerEpoch 

pub struct TokenizerEpoch(/* private fields */);

A tokenizer epoch: the identity of one analysis behaviour.

Stored as a little-endian u64 in EpochMeta.tokenizer.

Implementations

impl TokenizerEpoch

pub fn of(config: &TokenizerConfig) -> Self

Computes the epoch of a configuration.

The canonical digest input

Little-endian throughout, length-prefixed for every variable field, in exactly this order: magic, segmenter id, folding form id, folding table version, Unicode version string, stopword list id, stopword term count, stemmer id, the three boolean knobs, profile id, then the vocabulary. The vocabulary contributes its entry count followed by every (surface form, canonical term) pair in sorted order, so two vocabularies built by different insertion orders digest identically.

This layout is a persisted-meaning contract frozen by golden_streams_cannot_change_without_an_epoch_bump.

pub const fn value(self) -> u64

Returns the raw digest.

pub fn to_hex(self) -> String

Returns the lowercase hex form used in diagnostics.

Trait Implementations

impl Clone for TokenizerEpoch

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenizerEpoch

impl Debug for TokenizerEpoch

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerEpoch

fn fmt(&self, formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerEpoch

impl Hash for TokenizerEpoch

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for TokenizerEpoch

fn cmp(&self, other: &Self) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

fn clamp_to<R>(self, range: R) -> Self

where Self: Sized, R: ClampBounds<Self>,

🔬This is a nightly-only experimental API. (clamp_to)

Restrict a value to a certain range.

impl PartialEq for TokenizerEpoch

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl PartialOrd for TokenizerEpoch

fn partial_cmp(&self, other: &Self) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl StructuralPartialEq for TokenizerEpoch