Struct tokenizers::Token

pub struct Token<'a> {
    pub offset: Option<usize>,
    // some fields omitted
}

Encapsulates the string term of a token.

The term that the token encapsulates is wrapped in a Cow (copy-on-write), so that the string slice is only copied if it is modified.

Fields

offset: Option<usize>

The index of the token within the original un-tokenized string slice.

Methods

impl<'a> Token<'a>

pub fn new(term: &'a str, offset: Option<usize>) -> Self

Create a new Token object.

pub fn from(term: &'a str) -> Self

Initializes a Token from a string slice.

pub fn as_str(&self) -> &str

Extracts the string slice contained in the Token.

Examples

let s = "Hello";
let token = tokenizers::Token::from(s);
assert_eq!(token.as_str(), s);

pub fn as_mut_str(&mut self) -> &mut str

Returns a mutable reference to the contained string slice.

Examples

let mut token = tokenizers::Token::from("Hello");
token.as_mut_str().make_ascii_lowercase();
assert_eq!(token.as_str(), "hello");

pub fn into_string(self) -> String

Clones the contained string slice if it is not already owned (i.e. if it has not already been modified) and returns it as a String.

Examples

let mut token = tokenizers::Token::from("Hello");
assert_eq!(token.into_string(), String::from("Hello"));

Trait Implementations

impl<'a> Eq for Token<'a>

impl<'a> PartialEq<Token<'a>> for Token<'a>

fn eq(&self, other: &Self) -> bool

This method tests for self and other values to be equal, and is used by ==.

#[must_use] fn ne(&self, other: &Rhs) -> bool

This method tests for !=.

impl<'a> PartialEq<String> for Token<'a>

fn eq(&self, other: &String) -> bool

This method tests for self and other values to be equal, and is used by ==.

#[must_use] fn ne(&self, other: &Rhs) -> bool

This method tests for !=.

impl<'a, '_> PartialEq<&'_ str> for Token<'a>

fn eq(&self, other: &&str) -> bool

This method tests for self and other values to be equal, and is used by ==.

#[must_use] fn ne(&self, other: &Rhs) -> bool

This method tests for !=.

impl<'a> Display for Token<'a>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.