Skip to main content

Rank

riptoken

Type Alias Rank 

Source 
pub type Rank = u32;
Expand description

Integer rank of a token in the BPE vocabulary.

Lower ranks are merged first. Rank::MAX is reserved as a sentinel meaning “this byte span is not in the vocabulary”.