Struct Token

pub struct Token { /* private fields */ }

Token

The Token type stores a CoNLL-U token. Various information can be associated with a token, such as its lemma, (universal) part-of-speech, morphological features, and other (miscellaneous) features.

Tokens are typically stored as vertices in a DepGraph, so that their dependency heads and dependents can be queried.

Implementations

impl Token

pub fn new(form: impl Into<String>) -> Token

Create a new token where all the non-form fields are absent.

pub fn form(&self) -> &str

Get the word form or punctuation symbol.

pub fn lemma(&self) -> Option<&str>

Get the lemma or stem of the word form.

pub fn upos(&self) -> Option<&str>

Get the universal part-of-speech tag.

pub fn xpos(&self) -> Option<&str>

Get the language-specific part-of-speech tag.

pub fn features(&self) -> &Features

Get the syntactic and/or morphological features of the token.

pub fn features_mut(&mut self) -> &mut Features

Get the syntactic and/or morphological features of the token.

Returns a mutable reference, so that the features can be updated.

pub fn deps(&self) -> Option<&str>

Get enhanced dependencies.

Warning: this method will be removed once proper support for enhanced dependencies is added.

pub fn misc(&self) -> &Misc

Get miscellaneous token features.

pub fn misc_mut(&mut self) -> &mut Misc

Get miscellaneous token features.

Returns a mutable reference, so that the information can be updated.

pub fn set_form(&mut self, form: impl Into<String>) -> String

Set the word form or punctuation symbol.

Returns the form that is replaced.

pub fn set_lemma<S>(&mut self, lemma: Option<S>) -> Option<String>

where S: Into<String>,

Set the lemma or stem of the word form.

Returns the lemma that is replaced.

pub fn set_upos<S>(&mut self, upos: Option<S>) -> Option<String>

where S: Into<String>,

Set the universal part-of-speech tag.

Returns the universal part-of-speech tag that is replaced.

pub fn set_xpos<S>(&mut self, xpos: Option<S>) -> Option<String>

where S: Into<String>,

Set the language-specific part-of-speech tag.

Returns the language-specific part-of-speech tag that is replaced.

pub fn set_features(&mut self, features: Features) -> Features

Set the syntactic and/or morphological features of the token.

Returns the features that are replaced.

pub fn set_deps(&mut self, deps: Option<impl Into<String>>) -> Option<String>

Set UD enhanced dependencies.

Warning: this method will be removed once proper support for enhanced dependencies is added.

pub fn set_misc(&mut self, misc: Misc) -> Misc

Set miscellaneous token features.

Returns the features that are replaced.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Token> for TokenBuilder

fn from(token: Token) -> Self

Converts to this type from the input type.

impl From<TokenBuilder> for Token

fn from(builder: TokenBuilder) -> Self

Converts to this type from the input type.

impl FromIterator<Token> for Sentence

fn from_iter<T>(iter: T) -> Self

where T: IntoIterator<Item = Token>,

Creates a value from an iterator.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Token

impl StructuralPartialEq for Token