Module tokenizer 

The versioned analysis pipeline and its epoch identity.

The product feature

Text goes in; tokens come out carrying a term, a position, and the byte range they came from. The pipeline that produces them — segmentation, folding, stopwords, stemming, number words, and a user vocabulary — is identified by one digest, the tokenizer epoch. Same epoch, same bytes in, same tokens out, forever.

That identity is the point. Analyzer drift silently invalidating an index is failure class U11 (research/03:594), and no off-the-shelf tokenizer versions its own behaviour as index metadata (research/05:202). The digest goes into EpochMeta.tokenizer, which task 07 already reserved, so no format change is needed to carry it.

Query-time and index-time are one code path

Analyzer::analyze is used for both. tantivy’s missing search-analyzer is a known gap this design avoids by construction. A query-side extras hook is deliberately absent in v1 rather than stubbed.

Re-exports

pub use profiles::Profile;

pub use stopwords::StopwordList;

pub use vocab::MAX_SURFACE_TERMS;

pub use vocab::VocabError;

pub use vocab::Vocabulary;

Modules

profiles

Named analysis profiles.

stopwords

Versioned stopword lists shipped as data, never invented here.

vocab

User vocabulary and synonym stacking — the product feature of task 12.

Structs

Analyzer

A compiled analyzer.

Token

One analyzed token.

TokenFlags

Bit flags carried by every token.

TokenOffset

The byte range a token came from, into the analyzed text.

TokenizerConfig

The complete analysis configuration.

TokenizerEpoch

A tokenizer epoch: the identity of one analysis behaviour.

Enums

FoldingForm

Character folding form; an epoch-digest input.

Segmenter

Word segmentation strategy; an epoch-digest input.

Stemmer

Stemming algorithm; an epoch-digest input.

TokenizerError

Analysis rejection at the engine boundary.

Constants

MAX_TEXT_BYTES

The largest text one call will analyze.