Struct TokenSpan 

pub struct TokenSpan {
    pub start: usize,
    pub end: usize,
    pub text_span: TextSpan,
}

Span in subword token space.

Research Context (BERT for NER, NAACL 2019)

Transformer models operate on subword tokens, not characters. Entity boundaries often split mid-token:

Text:       "New York City"
Tokens:     ["New", "York", "City"]      <- clean split
Token IDs:  [2739, 1816, 2103]
TokenSpan:  (0, 3) for "New York City"

Text:       "playing"
Tokens:     ["play", "##ing"]            <- mid-word split
Token IDs:  [2377, 2075]
TokenSpan:  (0, 2) for "playing"

Key insight: When propagating BIO labels to continuation tokens (##), use I- prefix to avoid treating them as separate entities.

Fields

start: usize

Token index (start, inclusive)

end: usize

Token index (end, exclusive)

text_span: TextSpan

Original text span (for reconstruction)

Implementations

impl TokenSpan

pub const fn new(start: usize, end: usize, text_span: TextSpan) -> Self

Create a token span with its corresponding text span.

pub const fn len(&self) -> usize

Number of tokens in this span.

pub const fn is_empty(&self) -> bool

Check if empty.

pub const fn token_range(&self) -> Range<usize>

Token range.

Trait Implementations

impl Clone for TokenSpan

fn clone(&self) -> TokenSpan

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenSpan

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenSpan

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Hash for TokenSpan

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenSpan

fn eq(&self, other: &TokenSpan) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for TokenSpan

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for TokenSpan

impl Eq for TokenSpan

impl StructuralPartialEq for TokenSpan