[][src]Module piston_meta::tokenizer

Tracking tokenizer state.

Structs

TokenizerState

Stores the number of tokens received.

Functions

read_data

Reads meta data.