Struct OffsetMapping 

pub struct OffsetMapping { /* private fields */ }

Offset mapping from tokenizer.

Maps each token to its character span in the original text. Used to convert between token indices and character positions.

Research Note (HuggingFace Tokenizers)

The offset_mapping from HuggingFace tokenizers is a list of (char_start, char_end) for each token. Special tokens like [CLS] and [SEP] have offset (0, 0).

Implementations

impl OffsetMapping

pub fn new(offsets: Vec<(usize, usize)>) -> Self

Create from tokenizer output.

Arguments

• offsets - List of (char_start, char_end) for each token

pub fn get(&self, token_idx: usize) -> Option<(usize, usize)>

Get character span for a token.

pub fn char_span_to_tokens( &self, char_start: usize, char_end: usize, ) -> Option<(usize, usize)>

Find tokens that overlap with a character span.

Returns (first_token, last_token_exclusive).

Note on Label Propagation

For entity “playing” tokenized as ["play", "##ing"]:

• Assign B-PER to “play” (first token)
• Assign I-PER to “##ing” (continuation)

pub fn tokens_to_char_span( &self, token_start: usize, token_end: usize, ) -> Option<(usize, usize)>

Convert token span to character span.

pub fn len(&self) -> usize

Number of tokens.

pub fn is_empty(&self) -> bool

Check if empty.

Trait Implementations

impl Clone for OffsetMapping

fn clone(&self) -> OffsetMapping

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for OffsetMapping

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.