Struct rustlr::lexer_interface::StrTokenizer

pub struct StrTokenizer<'t> {
    pub keep_whitespace: bool,
    pub keep_newline: bool,
    pub keep_comment: bool,
    pub line_positions: Vec<usize>,
    pub specialeof: &'static str,
    pub tab_spaces: usize,
    pub allow_newline_in_string: bool,
    pub priority_symbols: BTreeMap<&'static str, u32>,
    /* private fields */
}

General-purpose, zero-copy lexical analyzer that produces RawTokens from an str. This tokenizer uses regex, although not for everything. For example, to allow for string literals that contain escaped quotations, a direct loop is implemented. The tokenizer gives the option of returning newlines, whitespaces (with count) and comments as special tokens. It recognizes mult-line string literals, multi-line as well as single-line comments, and returns the starting line and column positions of each token.

Example:

  let mut scanner = StrTokenizer::from_str("while (1) fork();//run at your own risk");
  scanner.set_line_comment("//");
  scanner.keep_comment=true;
  scanner.add_single(';'); // separates ; from following symbols
  while let Some(token) = scanner.next() {
     println!("Token,line,column: {:?}",&token);
  }

this code produces output

  Token,line,column: (Alphanum("while"), 1, 1)
  Token,line,column: (Symbol("("), 1, 7)
  Token,line,column: (Num(1), 1, 8)
  Token,line,column: (Symbol(")"), 1, 9)
  Token,line,column: (Alphanum("fork"), 1, 11)
  Token,line,column: (Symbol("("), 1, 15)
  Token,line,column: (Symbol(")"), 1, 16)
  Token,line,column: (Symbol(";"), 1, 17)
  Token,line,column: (Verbatim("//run at your own risk"), 1, 18)

Fields

keep_whitespace: bool

flag to toggle whether whitespaces should be returned as Whitespace tokens, default is false.

keep_newline: bool

flag to toggle whether newline characters (‘\n’) are returned as Newline tokens. Default is false. Note that if this flag is set to true then newline characters are treated differently from other whitespaces. For example, when parsing languages like Python, both keep_whitespace and keep_newline should be set to true. Change option in grammar with lexattribute keep_newline=true

keep_comment: bool

flag to determine if comments are kept and returned as Verbatim tokens, default is false.

line_positions: Vec<usize>

vector of starting byte position of each line, position 0 not used.

specialeof: &'static str

tab_spaces: usize

number of whitespaces to count for each tab (default 6). This can be changed with a declaration such as lexattribute tab_spaces=8. Do not set this value to zero.

allow_newline_in_string: bool

allows string literals to contain non-escaped newline characters: warning: changing the default (false) may reduce the accuracy of error reporting.

priority_symbols: BTreeMap<&'static str, u32>

Multiset of verbatim symbols that have priority over other categories; sorted by string order. The multiset is implemented as a map from strings to counts.

Implementations

impl<'t> StrTokenizer<'t>

pub fn new() -> StrTokenizer<'t>

creats a new tokenizer with defaults, does not set input.

pub fn map<G, FM: FnOnce(&mut StrTokenizer<'t>) -> G>(&mut self, f: FM) -> G

applies closure to self, can be used together with lexconditional to invoke custom actions

pub fn current_text(&self) -> &'t str

returns text of the current token, untrimed

pub fn add_double(&mut self, s: &'t str)

adds a symbol of exactly length two. If the length is not two the function has no effect. Note that these symbols override all other types except for leading whitespaces and comments markers, e.g. “//” will have precedence over “/” and “==” will have precedence over “=”.

pub fn add_single(&mut self, c: char)

add a single-character symbol. The type of the symbol overrides other types except for whitespaces, comments and double-character symbols.

pub fn add_triple(&mut self, s: &'t str)

add a 3-character symbol

pub fn add_priority_symbol(&mut self, s: &'static str)

multiset-add a verbatim string as a priority symbol: will be returned as Symbol(s)

pub fn del_priority_symbol(&mut self, s: &'static str)

multiset-remove verbative string as a priority symbol

pub fn skip_to(&mut self, target: &'static str)

Skips to last occurrence of target string, or to end of input. Returns RawToken::Skipto token.

pub fn skip_reset(&mut self)

cancels recoginition of skip_to (called internally)

pub fn skip_match( &mut self, lbr: &'static str, rbr: &'static str, offset: i32, delimit: &'static str )

StrTokenizer can do a little more than recognize just regular expressions. It can detect matching brackets, and return return the bracket-enclosed text as a RawToken::Skipto token. An offset of 1 is recommended, as this call is usually made after an instance of the opening left-bracket is seen as lookahead. The operation increases a counter, starting with the offset everytime a left-bracket is seen and decreases it with every right-bracket, until counter==0, at which point it returns the skipped text in a RawToken::Skipmatched token. It will top searching when the delimit string is reached. If delimit is the empty string, then it will search until the end of input.

pub fn add_custom(&mut self, tkind: &'static str, reg_expr: &str)

add custom defined regex, will correspond to RawToken::Custom variant. Custom regular expressions should not start with whitespaces and will override all others. Multiple Custom types will be matched by the order in which they where declared in the grammar file.

pub fn set_input(&mut self, inp: &'t str)

sets the input str to be parsed, resets position information. Note: trailing whitespaces are always trimmed from the input.

pub fn set_line_comment(&mut self, cm: &'t str)

sets the symbol that begins a single-line comment. The default is “//”. If this is set to the empty string then no line-comments are recognized.

pub fn set_multiline_comments(&mut self, cm: &'t str)

sets the symbols used to delineate multi-line comments using a whitespace separated string such as “/* */”. These symbols are also the default. Set this to the empty string to disable multi-line comments.

pub fn line(&self) -> usize

the current line that the tokenizer is on

pub fn column(&self) -> usize

the current column of the tokenizer

pub fn current_position(&self) -> usize

returns the current absolute byte position of the Tokenizer

pub fn previous_position(&self) -> usize

returns the previous absolute byte position of the Tokenizer

pub fn get_source(&self) -> &str

returns the source of the tokenizer such as URL or filename

pub fn set_source<'u: 't>(&mut self, s: &'u str)

pub fn current_line(&self) -> &str

gets the current line of the source input

pub fn get_line(&self, i: usize) -> Option<&str>

Retrieves the ith line of the raw input, if line index i is valid. This function is intended to be called once the tokenizer has completed its task of scanning and tokenizing the entire input. Otherwise, it may return None if the tokenizer has not yet scanned up to the line indicated. That is, it is intended for error message generation when evaluating the AST post-parsing.

pub fn get_slice(&self, start: usize, end: usize) -> &str

Retrieves the source string slice at the indicated indices; returns the empty string if indices are invalid. The default implementation returns the empty string.

pub fn reset(&mut self)

reset tokenizer to parse from beginning of input

pub fn backtrack(&mut self, offset: usize)

pub fn next_token(&mut self) -> Option<(RawToken<'t>, usize, usize)>

returns next token, along with starting line and column numbers. This function will return None at end of stream or LexError along with a message printed to stderr if a tokenizer error occured.

impl<'t> StrTokenizer<'t>

pub fn from_source(ls: &'t LexSource<'t>) -> StrTokenizer<'t>

creates a StrTokenizer from a LexSource structure that contains a string representing the contents of the source, and calls StrTokenizer::set_input to reference that string. To create a tokenizer that reads from, for example, a file is:

let source = LexSource::new(source_path).unwrap();
let mut tokenizer = StrTokenizer::from_source(&source);

pub fn from_str(s: &'t str) -> StrTokenizer<'t>

creates a string tokenizer and sets input to give str.

Trait Implementations

impl<'t> Iterator for StrTokenizer<'t>

type Item = (RawToken<'t>, usize, usize)

The type of the elements being iterated over.

fn next(&mut self) -> Option<(RawToken<'t>, usize, usize)>

Advances the iterator and returns the next value.

fn next_chunk<const N: usize>( &mut self ) -> Result<[Self::Item; N], IntoIter<Self::Item, N>>

where Self: Sized,

🔬This is a nightly-only experimental API. (iter_next_chunk)

Advances the iterator and returns an array containing the next N values.

fn size_hint(&self) -> (usize, Option<usize>)

Returns the bounds on the remaining length of the iterator.

fn count(self) -> usize

where Self: Sized,

Consumes the iterator, counting the number of iterations and returning it.

fn last(self) -> Option<Self::Item>

where Self: Sized,

Consumes the iterator, returning the last element.

fn advance_by(&mut self, n: usize) -> Result<(), NonZeroUsize>

🔬This is a nightly-only experimental API. (iter_advance_by)

Advances the iterator by n elements.

fn nth(&mut self, n: usize) -> Option<Self::Item>

Returns the nth element of the iterator.

fn step_by(self, step: usize) -> StepBy<Self>

where Self: Sized,

Creates an iterator starting at the same point, but stepping by the given amount at each iteration.

fn chain<U>(self, other: U) -> Chain<Self, <U as IntoIterator>::IntoIter>

where Self: Sized, U: IntoIterator<Item = Self::Item>,

Takes two iterators and creates a new iterator over both in sequence.

fn zip<U>(self, other: U) -> Zip<Self, <U as IntoIterator>::IntoIter>

where Self: Sized, U: IntoIterator,

‘Zips up’ two iterators into a single iterator of pairs.

fn intersperse_with<G>(self, separator: G) -> IntersperseWith<Self, G>

where Self: Sized, G: FnMut() -> Self::Item,

🔬This is a nightly-only experimental API. (iter_intersperse)

Creates a new iterator which places an item generated by separator between adjacent items of the original iterator.

fn map<B, F>(self, f: F) -> Map<Self, F>

where Self: Sized, F: FnMut(Self::Item) -> B,

Takes a closure and creates an iterator which calls that closure on each element.

fn for_each<F>(self, f: F)

where Self: Sized, F: FnMut(Self::Item),

Calls a closure on each element of an iterator.

fn filter<P>(self, predicate: P) -> Filter<Self, P>

where Self: Sized, P: FnMut(&Self::Item) -> bool,

Creates an iterator which uses a closure to determine if an element should be yielded.

fn filter_map<B, F>(self, f: F) -> FilterMap<Self, F>

where Self: Sized, F: FnMut(Self::Item) -> Option<B>,

Creates an iterator that both filters and maps.

fn enumerate(self) -> Enumerate<Self>

where Self: Sized,

Creates an iterator which gives the current iteration count as well as the next value.

fn peekable(self) -> Peekable<Self>

where Self: Sized,

Creates an iterator which can use the peek and peek_mut methods to look at the next element of the iterator without consuming it. See their documentation for more information.

fn skip_while<P>(self, predicate: P) -> SkipWhile<Self, P>

where Self: Sized, P: FnMut(&Self::Item) -> bool,

Creates an iterator that skips elements based on a predicate.

fn take_while<P>(self, predicate: P) -> TakeWhile<Self, P>

where Self: Sized, P: FnMut(&Self::Item) -> bool,

Creates an iterator that yields elements based on a predicate.

fn map_while<B, P>(self, predicate: P) -> MapWhile<Self, P>

where Self: Sized, P: FnMut(Self::Item) -> Option<B>,

Creates an iterator that both yields elements based on a predicate and maps.

fn skip(self, n: usize) -> Skip<Self>

where Self: Sized,

Creates an iterator that skips the first n elements.

fn take(self, n: usize) -> Take<Self>

where Self: Sized,

Creates an iterator that yields the first n elements, or fewer if the underlying iterator ends sooner.

fn scan<St, B, F>(self, initial_state: St, f: F) -> Scan<Self, St, F>

where Self: Sized, F: FnMut(&mut St, Self::Item) -> Option<B>,

An iterator adapter which, like fold, holds internal state, but unlike fold, produces a new iterator.

fn flat_map<U, F>(self, f: F) -> FlatMap<Self, U, F>

where Self: Sized, U: IntoIterator, F: FnMut(Self::Item) -> U,

Creates an iterator that works like map, but flattens nested structure.

fn map_windows<F, R, const N: usize>(self, f: F) -> MapWindows<Self, F, N>

where Self: Sized, F: FnMut(&[Self::Item; N]) -> R,

🔬This is a nightly-only experimental API. (iter_map_windows)

Calls the given function f for each contiguous window of size N over self and returns an iterator over the outputs of f. Like slice::windows(), the windows during mapping overlap as well.

fn fuse(self) -> Fuse<Self>

where Self: Sized,

Creates an iterator which ends after the first None.

fn inspect<F>(self, f: F) -> Inspect<Self, F>

where Self: Sized, F: FnMut(&Self::Item),

Does something with each element of an iterator, passing the value on.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Borrows an iterator, rather than consuming it.

fn collect<B>(self) -> B

where B: FromIterator<Self::Item>, Self: Sized,

Transforms an iterator into a collection.

fn collect_into<E>(self, collection: &mut E) -> &mut E

where E: Extend<Self::Item>, Self: Sized,

🔬This is a nightly-only experimental API. (iter_collect_into)

Collects all the items from an iterator into a collection.

fn partition<B, F>(self, f: F) -> (B, B)

where Self: Sized, B: Default + Extend<Self::Item>, F: FnMut(&Self::Item) -> bool,

Consumes an iterator, creating two collections from it.

fn is_partitioned<P>(self, predicate: P) -> bool

where Self: Sized, P: FnMut(Self::Item) -> bool,

🔬This is a nightly-only experimental API. (iter_is_partitioned)

Checks if the elements of this iterator are partitioned according to the given predicate, such that all those that return true precede all those that return false.

fn try_fold<B, F, R>(&mut self, init: B, f: F) -> R

where Self: Sized, F: FnMut(B, Self::Item) -> R, R: Try<Output = B>,

An iterator method that applies a function as long as it returns successfully, producing a single, final value.

fn try_for_each<F, R>(&mut self, f: F) -> R

where Self: Sized, F: FnMut(Self::Item) -> R, R: Try<Output = ()>,

An iterator method that applies a fallible function to each item in the iterator, stopping at the first error and returning that error.

fn fold<B, F>(self, init: B, f: F) -> B

where Self: Sized, F: FnMut(B, Self::Item) -> B,

Folds every element into an accumulator by applying an operation, returning the final result.

fn reduce<F>(self, f: F) -> Option<Self::Item>

where Self: Sized, F: FnMut(Self::Item, Self::Item) -> Self::Item,

Reduces the elements to a single one, by repeatedly applying a reducing operation.

fn try_reduce<F, R>( &mut self, f: F ) -> <<R as Try>::Residual as Residual<Option<<R as Try>::Output>>>::TryType

where Self: Sized, F: FnMut(Self::Item, Self::Item) -> R, R: Try<Output = Self::Item>, <R as Try>::Residual: Residual<Option<Self::Item>>,

🔬This is a nightly-only experimental API. (iterator_try_reduce)

Reduces the elements to a single one by repeatedly applying a reducing operation. If the closure returns a failure, the failure is propagated back to the caller immediately.

fn all<F>(&mut self, f: F) -> bool

where Self: Sized, F: FnMut(Self::Item) -> bool,

Tests if every element of the iterator matches a predicate.

fn any<F>(&mut self, f: F) -> bool

where Self: Sized, F: FnMut(Self::Item) -> bool,

Tests if any element of the iterator matches a predicate.

fn find<P>(&mut self, predicate: P) -> Option<Self::Item>

where Self: Sized, P: FnMut(&Self::Item) -> bool,

Searches for an element of an iterator that satisfies a predicate.

fn find_map<B, F>(&mut self, f: F) -> Option<B>

where Self: Sized, F: FnMut(Self::Item) -> Option<B>,

Applies function to the elements of iterator and returns the first non-none result.

fn try_find<F, R>( &mut self, f: F ) -> <<R as Try>::Residual as Residual<Option<Self::Item>>>::TryType

where Self: Sized, F: FnMut(&Self::Item) -> R, R: Try<Output = bool>, <R as Try>::Residual: Residual<Option<Self::Item>>,

🔬This is a nightly-only experimental API. (try_find)

Applies function to the elements of iterator and returns the first true result or the first error.

fn position<P>(&mut self, predicate: P) -> Option<usize>

where Self: Sized, P: FnMut(Self::Item) -> bool,

Searches for an element in an iterator, returning its index.

fn max_by_key<B, F>(self, f: F) -> Option<Self::Item>

where B: Ord, Self: Sized, F: FnMut(&Self::Item) -> B,

Returns the element that gives the maximum value from the specified function.

fn max_by<F>(self, compare: F) -> Option<Self::Item>

where Self: Sized, F: FnMut(&Self::Item, &Self::Item) -> Ordering,

Returns the element that gives the maximum value with respect to the specified comparison function.

fn min_by_key<B, F>(self, f: F) -> Option<Self::Item>

where B: Ord, Self: Sized, F: FnMut(&Self::Item) -> B,

Returns the element that gives the minimum value from the specified function.

fn min_by<F>(self, compare: F) -> Option<Self::Item>

where Self: Sized, F: FnMut(&Self::Item, &Self::Item) -> Ordering,

Returns the element that gives the minimum value with respect to the specified comparison function.

fn unzip<A, B, FromA, FromB>(self) -> (FromA, FromB)

where FromA: Default + Extend<A>, FromB: Default + Extend<B>, Self: Sized + Iterator<Item = (A, B)>,

Converts an iterator of pairs into a pair of containers.

fn copied<'a, T>(self) -> Copied<Self>

where T: 'a + Copy, Self: Sized + Iterator<Item = &'a T>,

Creates an iterator which copies all of its elements.

fn cloned<'a, T>(self) -> Cloned<Self>

where T: 'a + Clone, Self: Sized + Iterator<Item = &'a T>,

Creates an iterator which clones all of its elements.

fn array_chunks<const N: usize>(self) -> ArrayChunks<Self, N>

where Self: Sized,

🔬This is a nightly-only experimental API. (iter_array_chunks)

Returns an iterator over N elements of the iterator at a time.

fn sum<S>(self) -> S

where Self: Sized, S: Sum<Self::Item>,

Sums the elements of an iterator.

fn product<P>(self) -> P

where Self: Sized, P: Product<Self::Item>,

Iterates over the entire iterator, multiplying all the elements

fn cmp_by<I, F>(self, other: I, cmp: F) -> Ordering

where Self: Sized, I: IntoIterator, F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Ordering,

🔬This is a nightly-only experimental API. (iter_order_by)

Lexicographically compares the elements of this Iterator with those of another with respect to the specified comparison function.

fn partial_cmp<I>(self, other: I) -> Option<Ordering>

where I: IntoIterator, Self::Item: PartialOrd<<I as IntoIterator>::Item>, Self: Sized,

Lexicographically compares the PartialOrd elements of this Iterator with those of another. The comparison works like short-circuit evaluation, returning a result without comparing the remaining elements. As soon as an order can be determined, the evaluation stops and a result is returned.

fn partial_cmp_by<I, F>(self, other: I, partial_cmp: F) -> Option<Ordering>

where Self: Sized, I: IntoIterator, F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Option<Ordering>,

🔬This is a nightly-only experimental API. (iter_order_by)

Lexicographically compares the elements of this Iterator with those of another with respect to the specified comparison function.

fn eq<I>(self, other: I) -> bool

where I: IntoIterator, Self::Item: PartialEq<<I as IntoIterator>::Item>, Self: Sized,

Determines if the elements of this Iterator are equal to those of another.

fn eq_by<I, F>(self, other: I, eq: F) -> bool

where Self: Sized, I: IntoIterator, F: FnMut(Self::Item, <I as IntoIterator>::Item) -> bool,

🔬This is a nightly-only experimental API. (iter_order_by)

Determines if the elements of this Iterator are equal to those of another with respect to the specified equality function.

fn ne<I>(self, other: I) -> bool

where I: IntoIterator, Self::Item: PartialEq<<I as IntoIterator>::Item>, Self: Sized,

Determines if the elements of this Iterator are not equal to those of another.

fn lt<I>(self, other: I) -> bool

where I: IntoIterator, Self::Item: PartialOrd<<I as IntoIterator>::Item>, Self: Sized,

Determines if the elements of this Iterator are lexicographically less than those of another.

fn le<I>(self, other: I) -> bool

where I: IntoIterator, Self::Item: PartialOrd<<I as IntoIterator>::Item>, Self: Sized,

Determines if the elements of this Iterator are lexicographically less or equal to those of another.

fn gt<I>(self, other: I) -> bool

where I: IntoIterator, Self::Item: PartialOrd<<I as IntoIterator>::Item>, Self: Sized,

Determines if the elements of this Iterator are lexicographically greater than those of another.

fn ge<I>(self, other: I) -> bool

where I: IntoIterator, Self::Item: PartialOrd<<I as IntoIterator>::Item>, Self: Sized,

Determines if the elements of this Iterator are lexicographically greater than or equal to those of another.

fn is_sorted_by<F>(self, compare: F) -> bool

where Self: Sized, F: FnMut(&Self::Item, &Self::Item) -> Option<Ordering>,

🔬This is a nightly-only experimental API. (is_sorted)

Checks if the elements of this iterator are sorted using the given comparator function.

fn is_sorted_by_key<F, K>(self, f: F) -> bool

where Self: Sized, F: FnMut(Self::Item) -> K, K: PartialOrd,

🔬This is a nightly-only experimental API. (is_sorted)

Checks if the elements of this iterator are sorted using the given key extraction function.