Struct rust_tokenizers::Token

pub struct Token {
    pub text: String,
    pub offset: Offset,
    pub reference_offsets: Vec<OffsetSize>,
    pub mask: Mask,
}

Owned token that references the original text but stores its own string representation.

Fields

text: String

String representation

offset: Offset

Start and end positions of the token with respect to the original text

reference_offsets: Vec<OffsetSize>

Sequence of positions with respect to the original text contained in the token. For example, if the token offset is start: 4, end: 10, corresponding reference_offsets are [4, 5, 6, 7, 8, 9]

mask: Mask

Mask indicating the type of the token

Implementations

impl Token

pub fn new(text: String) -> Token

Creates a new owned token from a String.

Parameters

• text (String): text reference

Example

use rust_tokenizers::Token;
let text = "world".to_string();
let token = Token::new(text);

pub fn as_ref(&self) -> TokenRef<'_>

Converts an owned token to a reference form

Example

use rust_tokenizers::Token;
let text = "world".to_string();
let token = Token::new(text);

let token_ref = token.as_ref();

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a copy of the value.

pub fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ConsolidatableTokens<Token> for Vec<Token>

fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, Token>

Creates an iterator from a sequence of ConsolidatableTokens.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&'_ str> for Token

fn from(text: &str) -> Self

Performs the conversion.

impl<'a> From<&'a Token> for TokenRef<'a>

fn from(other: &'a Token) -> Self

Performs the conversion.

impl<'a> From<TokenRef<'a>> for Token

fn from(other: TokenRef<'a>) -> Self

Performs the conversion.

impl PartialEq<Token> for Token

fn eq(&self, other: &Token) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Token) -> bool

This method tests for !=.

impl StructuralPartialEq for Token

impl TokenTrait for Token

fn offset(&self) -> Option<Offset>

Returns the offset of the token with respect to the original string

fn mask(&self) -> Mask

Returns the token mask

fn as_str(&self) -> &str

Returns a string representation for the token