roketok 0.3.1

This crate provides a way to simply set up a tokenizer and use it. Not recommended for simple tokenizers as this crate adds a bunch of stuff to support many if not all kinds of tokenizers.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

/// # Record
/// Records the tokenizers data like current row, column, etc.
/// It is used in tokens, rules, etc.
#[derive(Debug, Clone, Copy)]
pub struct Record {
    pub(crate) pos: (usize, usize),
}

impl Record {
    /// Get the recorded row.
    #[must_use]
    #[inline(always)]
    pub fn row(&self) -> usize {
        self.pos.0
    }
    
    /// Get the recorded column.
    #[must_use]
    #[inline(always)]
    pub fn col(&self) -> usize {
        self.pos.1
    }
}