Struct TokenAlignment 

pub struct TokenAlignment {
    pub token_index: usize,
    pub word_index: Option<usize>,
    pub char_start: usize,
    pub char_end: usize,
    pub is_special: bool,
    pub starts_word: bool,
    pub ends_word: bool,
}

Represents the alignment between tokens and words

Fields

token_index: usize

Token index in the tokenized sequence

word_index: Option<usize>

Word index that this token belongs to

char_start: usize

Character start position in the original text

char_end: usize

Character end position in the original text

is_special: bool

Whether this token is a special token

starts_word: bool

Whether this token starts a word

ends_word: bool

Whether this token ends a word

Trait Implementations

impl Clone for TokenAlignment

fn clone(&self) -> TokenAlignment

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenAlignment

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenAlignment

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenAlignment

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.