Struct diffr_lib::Tokenization

pub struct Tokenization<'a> { /* fields omitted */ }

A tokenized slice of bytes.

Methods

impl<'a> Tokenization<'a>

pub fn new(data: &'a [u8], tokens: &'a [HashedSpan]) -> Self

pub fn data(&self) -> &'a [u8]

pub fn split_at(&self, lo: isize, hi: isize) -> (Self, Self)

Split self in two tokenizations:

• the first one from the start to lo;
• the second one from hi to the end.

pub fn nb_tokens(&self) -> usize

Get self's number of tokens.

pub fn nth_token(&self, n: isize) -> HashedSlice

Get self's nth token.

pub fn nth_span(&self, n: isize) -> HashedSpan

Get the span correponding to self's nth token.

Trait Implementations

impl<'a> Debug for Tokenization<'a>

fn fmt(&self, f: &mut Formatter) -> Result<(), FmtErr>

Formats the value using the given formatter.