Struct rust_tokenizers::TokenRef

pub struct TokenRef<'a> {
    pub text: &'a str,
    pub offset: Offset,
    pub reference_offsets: &'a [OffsetSize],
    pub mask: Mask,
}

Reference token that references the original text, with a string slice representation

Fields

text: &'a str

String representation

offset: Offset

Start and end positions of the token with respect to the original text

reference_offsets: &'a [OffsetSize]

Sequence of positions with respect to the original text contained in the token. For example, if the token offset is start: 4, end: 10, corresponding reference_offsets are [4, 5, 6, 7, 8, 9]

mask: Mask

Mask indicating the type of the token

Implementations

impl<'a> TokenRef<'a>

pub fn new(text: &'a str, offsets: &'a [OffsetSize]) -> TokenRef<'a>

Creates a new token reference from a text and list of offsets.

Parameters

• text (&str): text reference
• offsets (&[OffsetSize]): reference positions with respect to the original text

Example

use rust_tokenizers::TokenRef;
let _original_text = "Hello, world";
let text = "world";
let offsets = &[7, 8, 9, 10, 11];

let token_ref = TokenRef::new(text, offsets);

pub fn to_owned(self) -> Token

Converts a token reference to an owned form.

Example

use rust_tokenizers::TokenRef;
let _original_text = "Hello, world";
let text = "world";
let offsets = &[7, 8, 9, 10, 11];
let token_ref = TokenRef::new(text, offsets);

let owned_token = token_ref.to_owned();

Trait Implementations

impl<'a> Clone for TokenRef<'a>

fn clone(&self) -> TokenRef<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> ConsolidatableTokens<TokenRef<'a>> for Vec<TokenRef<'a>>

fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, TokenRef<'a>>

Creates an iterator from a sequence of ConsolidatableTokens.

impl<'a> Debug for TokenRef<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> From<&'a Token> for TokenRef<'a>

fn from(other: &'a Token) -> Self

Converts to this type from the input type.

impl<'a> From<TokenRef<'a>> for Token

fn from(other: TokenRef<'a>) -> Self

Converts to this type from the input type.

impl<'a> PartialEq for TokenRef<'a>

fn eq(&self, other: &TokenRef<'a>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> TokenTrait for TokenRef<'a>

fn offset(&self) -> Option<Offset>

Returns the offset of the token with respect to the original string

fn mask(&self) -> Mask

Returns the token mask

fn as_str(&self) -> &str

Returns a string representation for the token

impl<'a> Copy for TokenRef<'a>

impl<'a> Eq for TokenRef<'a>

impl<'a> StructuralEq for TokenRef<'a>

impl<'a> StructuralPartialEq for TokenRef<'a>