Struct PairSequence 

pub struct PairSequence {
    pub tokens: Vec<u32>,
    pub segments: Vec<u32>,
}

One two-segment encoder input: the token ids and, for each of them, which half of the pair it belongs to.

The two vectors are built together and travel together on purpose. They are the repo’s dominant bug shape waiting to happen — two structures that must agree, here about a length and about where the boundary is — and the single place that can get them right is TextEncoder::wrap_special_pair, which inserts the [SEP] that the boundary is defined by. Handing a caller the ids alone (what this seam used to do) meant the segment ids did not exist at all and every position was scored as “Sentence A”.

Fields

tokens: Vec<u32>

[CLS] a [SEP] b [SEP] for BERT.

segments: Vec<u32>

0 for the [CLS] a [SEP] half, 1 for the b [SEP] half — HuggingFace’s token_type_ids. Always the same length as Self::tokens.

Trait Implementations

impl Clone for PairSequence

fn clone(&self) -> PairSequence

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PairSequence

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for PairSequence

impl PartialEq for PairSequence

fn eq(&self, other: &PairSequence) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for PairSequence