Struct TokenView 

pub struct TokenView {
    pub surface: String,
    pub byte_start: usize,
    pub byte_end: usize,
    pub position: usize,
    pub word_id: u32,
    pub is_unknown: bool,
    pub details: Vec<String>,
}

FFI-independent view of a lindera::token::Token.

Holds the pure data each binding needs to expose, so the extraction logic (loading details, reading the word id) lives in one place. Each binding then maps these fields onto its own FFI token type.

Fields

surface: String

byte_start: usize

byte_end: usize

position: usize

word_id: u32

is_unknown: bool

details: Vec<String>

Implementations

impl TokenView

pub fn from_token(token: Token<'_>) -> Self

Extracts the binding-facing data from a lindera token.

Trait Implementations

impl Clone for TokenView

fn clone(&self) -> TokenView

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenView

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.