Struct Tokenization

pub struct Tokenization<'a> { /* private fields */ }

A tokenized slice of bytes.

Implementations

impl<'a> Tokenization<'a>

pub fn new(data: &'a [u8], tokens: &'a [HashedSpan]) -> Self

pub fn data(&self) -> &'a [u8]

pub fn split_at(&self, lo: isize, hi: isize) -> (Self, Self)

Split self in two tokenizations:

• the first one from the start to lo;
• the second one from hi to the end.

pub fn nb_tokens(&self) -> usize

Get self’s number of tokens.

pub fn nth_token(&self, n: isize) -> HashedSlice<'_>

Get self’s nth token.

pub fn nth_span(&self, n: isize) -> HashedSpan

Get the span corresponding to self’s nth token.

Trait Implementations

impl<'a> Debug for Tokenization<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), FmtErr>

Formats the value using the given formatter.