Module piston_meta::tokenizer[][src]

Tracking tokenizer state.

Structs

TokenizerState

Stores the number of tokens received.

Functions

read_data

Reads meta data.