Struct Encoding 

pub struct Encoding {
    pub input_ids: Vec<u32>,
    pub attention_mask: Vec<u32>,
    pub offsets: Vec<(usize, usize)>,
}

Encoding result from tokenization

Contains the token IDs, attention mask, and character offsets needed for model inference.

Fields

input_ids: Vec<u32>

Token IDs (vocabulary indices)

attention_mask: Vec<u32>

Attention mask (1 for real tokens, 0 for padding)

offsets: Vec<(usize, usize)>

Character offsets in original text for each token (start_char, end_char) for each token Special tokens (CLS, SEP, PAD) have offset (0, 0)

Implementations

impl Encoding

pub fn new(input_ids: Vec<u32>, attention_mask: Vec<u32>) -> Self

Create a new encoding

pub fn with_offsets( input_ids: Vec<u32>, attention_mask: Vec<u32>, offsets: Vec<(usize, usize)>, ) -> Self

Create a new encoding with offsets

pub fn len(&self) -> usize

Get the length of the encoding

pub fn is_empty(&self) -> bool

Check if encoding is empty

pub fn to_arrays(&self) -> (Vec<i64>, Vec<i64>)

Convert to arrays suitable for ONNX inference

Returns (input_ids, attention_mask) as i64 arrays

Trait Implementations

impl Clone for Encoding

fn clone(&self) -> Encoding

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Encoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.