Struct BertEncoding 

pub struct BertEncoding {
    pub input_ids: Vec<u32>,
    pub attention_mask: Vec<u32>,
    pub token_type_ids: Vec<u32>,
}

A single encoded BERT input sequence.

Contains the input IDs, an attention mask (1 = real token, 0 = padding), and token type IDs (0 = first segment, 1 = second segment).

Fields

input_ids: Vec<u32>

Token IDs including special tokens.

attention_mask: Vec<u32>

Attention mask: 1 for real tokens, 0 for padding.

token_type_ids: Vec<u32>

Segment indicator: 0 for text_a, 1 for text_b.

Implementations

impl BertEncoding

pub fn new( input_ids: Vec<u32>, attention_mask: Vec<u32>, token_type_ids: Vec<u32>, ) -> Self

Create a new encoding with consistent lengths.

pub fn len(&self) -> usize

Returns the sequence length (number of tokens including padding).

pub fn is_empty(&self) -> bool

Returns true if there are no tokens.

Trait Implementations

impl Clone for BertEncoding

fn clone(&self) -> BertEncoding

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BertEncoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for BertEncoding

fn eq(&self, other: &BertEncoding) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for BertEncoding