Struct strizer::StreamTokenizer

pub struct StreamTokenizer<'a, R> { /* fields omitted */ }

StreamTokenizer<R> defines the structure for the tokenization of a given input that implements the BufRead trait. By using BufRead, the content of the text data is efficiently tokenized by reading it line-by-line.

Examples

use std::fs::File;
use std::io::BufReader;
use strizer::{StreamTokenizer, Token, TokenKind};

fn main() -> std::io::Result<()> {
  let file = File::open("log.txt")?;
  let mut reader = BufReader::new(file);

  let error_count = StreamTokenizer::new(&mut reader, &[])
    .filter(|(_, _, slice)| slice == "ERROR")
    .count();

  println!("number of error logs: {}", error_count);
  Ok(())
}

Implementations

impl<'a, R: BufRead> StreamTokenizer<'a, R>

pub fn new(input: &'a mut R, identifiers: &'a [char]) -> Self

Creates a new StreamTokenizer<R> with a given BufRead input. The different offsets and data used while iterating are initialized. An array of identifiers is passed by, and the associated characters will be identified as Character Token during the tokenization.

Examples

use std::io::Cursor;
use strizer::{StreamTokenizer, Token, TokenKind};

let cursor = &mut Cursor::new("abracadabra");
let tokenizer = StreamTokenizer::new(cursor, &['a']);

let a_count = tokenizer
  .filter(|(token, _, slice)| token.is_character() && slice == "a")
  .count();

assert_eq!(a_count, 5);

Trait Implementations

impl<'a, R: Debug> Debug for StreamTokenizer<'a, R>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a, R: BufRead> Iterator for StreamTokenizer<'a, R>

StreamTokenizer<R> implementation for Iterator.

type Item = (Token, Span, String)

The type of the elements being iterated over.

fn next(&mut self) -> Option<Self::Item>

Advances the iterator and returns the next value.

pub fn size_hint(&self) -> (usize, Option<usize>)

Returns the bounds on the remaining length of the iterator.

pub fn count(self) -> usize

Consumes the iterator, counting the number of iterations and returning it.

pub fn last(self) -> Option<Self::Item>

Consumes the iterator, returning the last element.

pub fn advance_by(&mut self, n: usize) -> Result<(), usize>

🔬 This is a nightly-only experimental API. (iter_advance_by)

recently added

Advances the iterator by n elements.

pub fn nth(&mut self, n: usize) -> Option<Self::Item>

Returns the nth element of the iterator.

pub fn step_by(self, step: usize) -> StepBy<Self>

Creates an iterator starting at the same point, but stepping by the given amount at each iteration.

pub fn chain<U>(self, other: U) -> Chain<Self, <U as IntoIterator>::IntoIter> where
    U: IntoIterator<Item = Self::Item>, 

Takes two iterators and creates a new iterator over both in sequence.

pub fn zip<U>(self, other: U) -> Zip<Self, <U as IntoIterator>::IntoIter> where
    U: IntoIterator, 

‘Zips up’ two iterators into a single iterator of pairs.

pub fn intersperse(self, separator: Self::Item) -> Intersperse<Self> where
    Self::Item: Clone, 

🔬 This is a nightly-only experimental API. (iter_intersperse)

recently added

Creates a new iterator which places a copy of separator between adjacent items of the original iterator.

pub fn intersperse_with<G>(self, separator: G) -> IntersperseWith<Self, G> where
    G: FnMut() -> Self::Item, 

🔬 This is a nightly-only experimental API. (iter_intersperse)

recently added

Creates a new iterator which places an item generated by separator between adjacent items of the original iterator.

pub fn map<B, F>(self, f: F) -> Map<Self, F> where
    F: FnMut(Self::Item) -> B, 

Takes a closure and creates an iterator which calls that closure on each element.

pub fn for_each<F>(self, f: F) where
    F: FnMut(Self::Item), 

Calls a closure on each element of an iterator.

pub fn filter<P>(self, predicate: P) -> Filter<Self, P> where
    P: FnMut(&Self::Item) -> bool, 

Creates an iterator which uses a closure to determine if an element should be yielded.

pub fn filter_map<B, F>(self, f: F) -> FilterMap<Self, F> where
    F: FnMut(Self::Item) -> Option<B>, 

Creates an iterator that both filters and maps.

pub fn enumerate(self) -> Enumerate<Self>

Creates an iterator which gives the current iteration count as well as the next value.

pub fn peekable(self) -> Peekable<Self>

Creates an iterator which can use the peek and peek_mut methods to look at the next element of the iterator without consuming it. See their documentation for more information.

pub fn skip_while<P>(self, predicate: P) -> SkipWhile<Self, P> where
    P: FnMut(&Self::Item) -> bool, 

Creates an iterator that skips elements based on a predicate.

pub fn take_while<P>(self, predicate: P) -> TakeWhile<Self, P> where
    P: FnMut(&Self::Item) -> bool, 

Creates an iterator that yields elements based on a predicate.

pub fn map_while<B, P>(self, predicate: P) -> MapWhile<Self, P> where
    P: FnMut(Self::Item) -> Option<B>, 

🔬 This is a nightly-only experimental API. (iter_map_while)

recently added

Creates an iterator that both yields elements based on a predicate and maps.

pub fn skip(self, n: usize) -> Skip<Self>

Creates an iterator that skips the first n elements.

pub fn take(self, n: usize) -> Take<Self>

Creates an iterator that yields the first n elements, or fewer if the underlying iterator ends sooner.

pub fn scan<St, B, F>(self, initial_state: St, f: F) -> Scan<Self, St, F> where
    F: FnMut(&mut St, Self::Item) -> Option<B>, 

An iterator adaptor similar to fold that holds internal state and produces a new iterator.

pub fn flat_map<U, F>(self, f: F) -> FlatMap<Self, U, F> where
    F: FnMut(Self::Item) -> U,
    U: IntoIterator, 

Creates an iterator that works like map, but flattens nested structure.

pub fn flatten(self) -> Flatten<Self> where
    Self::Item: IntoIterator, 

Creates an iterator that flattens nested structure.

pub fn fuse(self) -> Fuse<Self>

Creates an iterator which ends after the first None.

pub fn inspect<F>(self, f: F) -> Inspect<Self, F> where
    F: FnMut(&Self::Item), 

Does something with each element of an iterator, passing the value on.

pub fn by_ref(&mut self) -> &mut Self

Borrows an iterator, rather than consuming it.

#[must_use = "if you really need to exhaust the iterator, consider `.for_each(drop)` instead"]pub fn collect<B>(self) -> B where
    B: FromIterator<Self::Item>, 

Transforms an iterator into a collection.

pub fn partition<B, F>(self, f: F) -> (B, B) where
    B: Default + Extend<Self::Item>,
    F: FnMut(&Self::Item) -> bool, 

Consumes an iterator, creating two collections from it.

pub fn partition_in_place<'a, T, P>(self, predicate: P) -> usize where
    Self: DoubleEndedIterator<Item = &'a mut T>,
    T: 'a,
    P: FnMut(&T) -> bool, 

🔬 This is a nightly-only experimental API. (iter_partition_in_place)

new API

Reorders the elements of this iterator in-place according to the given predicate, such that all those that return true precede all those that return false. Returns the number of true elements found.

pub fn is_partitioned<P>(self, predicate: P) -> bool where
    P: FnMut(Self::Item) -> bool, 

🔬 This is a nightly-only experimental API. (iter_is_partitioned)

new API

Checks if the elements of this iterator are partitioned according to the given predicate, such that all those that return true precede all those that return false.

pub fn try_fold<B, F, R>(&mut self, init: B, f: F) -> R where
    R: Try<Ok = B>,
    F: FnMut(B, Self::Item) -> R, 

An iterator method that applies a function as long as it returns successfully, producing a single, final value.

pub fn try_for_each<F, R>(&mut self, f: F) -> R where
    R: Try<Ok = ()>,
    F: FnMut(Self::Item) -> R, 

An iterator method that applies a fallible function to each item in the iterator, stopping at the first error and returning that error.

pub fn fold<B, F>(self, init: B, f: F) -> B where
    F: FnMut(B, Self::Item) -> B, 

Folds every element into an accumulator by applying an operation, returning the final result.

pub fn reduce<F>(self, f: F) -> Option<Self::Item> where
    F: FnMut(Self::Item, Self::Item) -> Self::Item, 

Reduces the elements to a single one, by repeatedly applying a reducing operation.

pub fn all<F>(&mut self, f: F) -> bool where
    F: FnMut(Self::Item) -> bool, 

Tests if every element of the iterator matches a predicate.

pub fn any<F>(&mut self, f: F) -> bool where
    F: FnMut(Self::Item) -> bool, 

Tests if any element of the iterator matches a predicate.

pub fn find<P>(&mut self, predicate: P) -> Option<Self::Item> where
    P: FnMut(&Self::Item) -> bool, 

Searches for an element of an iterator that satisfies a predicate.

pub fn find_map<B, F>(&mut self, f: F) -> Option<B> where
    F: FnMut(Self::Item) -> Option<B>, 

Applies function to the elements of iterator and returns the first non-none result.

pub fn try_find<F, R>(
    &mut self,
    f: F
) -> Result<Option<Self::Item>, <R as Try>::Error> where
    R: Try<Ok = bool>,
    F: FnMut(&Self::Item) -> R, 

🔬 This is a nightly-only experimental API. (try_find)

new API

Applies function to the elements of iterator and returns the first true result or the first error.

pub fn position<P>(&mut self, predicate: P) -> Option<usize> where
    P: FnMut(Self::Item) -> bool, 

Searches for an element in an iterator, returning its index.

pub fn rposition<P>(&mut self, predicate: P) -> Option<usize> where
    Self: ExactSizeIterator + DoubleEndedIterator,
    P: FnMut(Self::Item) -> bool, 

Searches for an element in an iterator from the right, returning its index.

pub fn max(self) -> Option<Self::Item> where
    Self::Item: Ord, 

Returns the maximum element of an iterator.

pub fn min(self) -> Option<Self::Item> where
    Self::Item: Ord, 

Returns the minimum element of an iterator.

pub fn max_by_key<B, F>(self, f: F) -> Option<Self::Item> where
    B: Ord,
    F: FnMut(&Self::Item) -> B, 

Returns the element that gives the maximum value from the specified function.

pub fn max_by<F>(self, compare: F) -> Option<Self::Item> where
    F: FnMut(&Self::Item, &Self::Item) -> Ordering, 

Returns the element that gives the maximum value with respect to the specified comparison function.

pub fn min_by_key<B, F>(self, f: F) -> Option<Self::Item> where
    B: Ord,
    F: FnMut(&Self::Item) -> B, 

Returns the element that gives the minimum value from the specified function.

pub fn min_by<F>(self, compare: F) -> Option<Self::Item> where
    F: FnMut(&Self::Item, &Self::Item) -> Ordering, 

Returns the element that gives the minimum value with respect to the specified comparison function.

pub fn rev(self) -> Rev<Self> where
    Self: DoubleEndedIterator, 

Reverses an iterator’s direction.

pub fn unzip<A, B, FromA, FromB>(self) -> (FromA, FromB) where
    Self: Iterator<Item = (A, B)>,
    FromA: Default + Extend<A>,
    FromB: Default + Extend<B>, 

Converts an iterator of pairs into a pair of containers.

pub fn copied<'a, T>(self) -> Copied<Self> where
    Self: Iterator<Item = &'a T>,
    T: 'a + Copy, 

Creates an iterator which copies all of its elements.

pub fn cloned<'a, T>(self) -> Cloned<Self> where
    Self: Iterator<Item = &'a T>,
    T: 'a + Clone, 

Creates an iterator which clones all of its elements.

pub fn cycle(self) -> Cycle<Self> where
    Self: Clone, 

Repeats an iterator endlessly.

pub fn sum<S>(self) -> S where
    S: Sum<Self::Item>, 

Sums the elements of an iterator.

pub fn product<P>(self) -> P where
    P: Product<Self::Item>, 

Iterates over the entire iterator, multiplying all the elements

pub fn cmp<I>(self, other: I) -> Ordering where
    I: IntoIterator<Item = Self::Item>,
    Self::Item: Ord, 

Lexicographically compares the elements of this Iterator with those of another.

pub fn cmp_by<I, F>(self, other: I, cmp: F) -> Ordering where
    I: IntoIterator,
    F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Ordering, 

🔬 This is a nightly-only experimental API. (iter_order_by)

Lexicographically compares the elements of this Iterator with those of another with respect to the specified comparison function.

pub fn partial_cmp<I>(self, other: I) -> Option<Ordering> where
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>, 

Lexicographically compares the elements of this Iterator with those of another.

pub fn partial_cmp_by<I, F>(self, other: I, partial_cmp: F) -> Option<Ordering> where
    I: IntoIterator,
    F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Option<Ordering>, 

🔬 This is a nightly-only experimental API. (iter_order_by)

Lexicographically compares the elements of this Iterator with those of another with respect to the specified comparison function.

pub fn eq<I>(self, other: I) -> bool where
    I: IntoIterator,
    Self::Item: PartialEq<<I as IntoIterator>::Item>, 

Determines if the elements of this Iterator are equal to those of another.

pub fn eq_by<I, F>(self, other: I, eq: F) -> bool where
    I: IntoIterator,
    F: FnMut(Self::Item, <I as IntoIterator>::Item) -> bool, 

🔬 This is a nightly-only experimental API. (iter_order_by)

Determines if the elements of this Iterator are equal to those of another with respect to the specified equality function.

pub fn ne<I>(self, other: I) -> bool where
    I: IntoIterator,
    Self::Item: PartialEq<<I as IntoIterator>::Item>, 

Determines if the elements of this Iterator are unequal to those of another.

pub fn lt<I>(self, other: I) -> bool where
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>, 

Determines if the elements of this Iterator are lexicographically less than those of another.

pub fn le<I>(self, other: I) -> bool where
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>, 

Determines if the elements of this Iterator are lexicographically less or equal to those of another.

pub fn gt<I>(self, other: I) -> bool where
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>, 

Determines if the elements of this Iterator are lexicographically greater than those of another.

pub fn ge<I>(self, other: I) -> bool where
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>, 

Determines if the elements of this Iterator are lexicographically greater than or equal to those of another.

pub fn is_sorted(self) -> bool where
    Self::Item: PartialOrd<Self::Item>, 

🔬 This is a nightly-only experimental API. (is_sorted)

new API

Checks if the elements of this iterator are sorted.

pub fn is_sorted_by<F>(self, compare: F) -> bool where
    F: FnMut(&Self::Item, &Self::Item) -> Option<Ordering>, 

🔬 This is a nightly-only experimental API. (is_sorted)

new API

Checks if the elements of this iterator are sorted using the given comparator function.

pub fn is_sorted_by_key<F, K>(self, f: F) -> bool where
    K: PartialOrd<K>,
    F: FnMut(Self::Item) -> K, 

🔬 This is a nightly-only experimental API. (is_sorted)

new API

Checks if the elements of this iterator are sorted using the given key extraction function.