Skip to main content

encode

Function encode 

Source
pub fn encode(
    spec: &VocabularySpace,
    tok: &Tokenizer,
    ex: &TaggerExample,
    max_len: usize,
) -> Result<Encoded, String>
Expand description

Tokenize + project char-span labels onto tokens, padding to max_len. Reuses the offset projection already proven in [crate::tagger_data], so alignment semantics are identical between data generation and training.