Struct sentencepiece::PieceWithId

pub struct PieceWithId {
    pub piece: String,
    pub id: u32,
    pub span: (u32, u32),
}

Sentence piece with its identifier and string span.

Fields

piece: String

The sentence piece as a string.

id: u32

The vocabulary identifier of the sentence piece.

span: (u32, u32)

The span of the sentence piece in the tokenized string.

The span is encoded as the byte offsets [begin, end).

Trait Implementations

impl Clone for PieceWithId

fn clone(&self) -> PieceWithId

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PieceWithId

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq<PieceWithId> for PieceWithId

fn eq(&self, other: &PieceWithId) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &PieceWithId) -> bool

This method tests for !=.

impl Eq for PieceWithId

impl StructuralEq for PieceWithId

impl StructuralPartialEq for PieceWithId