Struct Munch

Source

pub struct Munch { /* private fields */ }

Expand description

Many patterns, matched anchored at one offset, longest wins.

Immutable, and Send + Sync on the same terms as crate::Regex: the C handle owns the buffers its scans rewrite, so the type keeps a pool of handles and leases one per scan.

Implementations§

Source §

impl Munch

Source

pub fn new<I, S>(patterns: I) -> Result<Self, Error>
where S: AsRef<str>, I: IntoIterator<Item = S>,

Compile every pattern in patterns as one anchored slate, with the default semantics crate::Regex::new uses.

§Errors

Error::NothingLexable when no pattern could be determinized, which is the only refusal that leaves nothing to work with. A partial refusal is success — read it with Munch::declined.

Source

pub fn patterns(&self) -> &[String]

The patterns this was compiled from, in order, exactly as they were given. Index i here is the index a Token reports.

Source

pub fn len(&self) -> usize

How many patterns the slate was built from, refusals included.

Source

pub fn is_empty(&self) -> bool

Whether the slate holds no patterns at all. Such a slate matches nothing.

Source

pub fn declined(&self) -> &[Refusal]

Every pattern the engine could not take, ascending. Empty is the normal case.

// A backreference cannot be determinized, and must not cost the others.
let m = irgx::Munch::new(["[a-z]+", r"(a)\1", "[0-9]+"])?;
assert_eq!(m.declined().len(), 1);
assert_eq!(m.declined()[0].pattern, 1);
assert_eq!(m.token("123", 0).unwrap().patterns(), &[2]);

Source

pub fn admitted(&self) -> usize

How many patterns can win at once — the upper bound on Token::patterns’s length, and the capacity Munch::scan_into never needs to grow past.

The admitted count, not Munch::len: a declined pattern can never win, so it can never be reported.

Source

pub fn token(&self, text: &str, at: usize) -> Option<Token>

The longest token beginning at exactly at, over every pattern.

None when nothing starts there. at == text.len() is legal and asks the only question left at the end of the input: does anything accept the empty string.

§Panics

On an engine fault, or if at is past the end of text. See Munch::try_token.

Source

pub fn try_token(&self, text: &str, at: usize) -> Result<Option<Token>, Error>

Munch::token, reporting a fault instead of panicking.

§Errors

Error::Search or Error::OutOfMemory if the engine could not answer, and Error::Inconsistent if at is past the end of text.

Source

pub fn token_among(&self, text: &str, at: usize, allow: &[u32]) -> Option<Token>

The longest token beginning at exactly at, restricted to the patterns allow names.

Restriction happens during the walk, not after: a forbidden pattern reaching further would otherwise hide every permitted one behind it.

let m = irgx::Munch::new(["if", "[a-z]+"])?;
// Unrestricted, the identifier wins and swallows the keyword.
assert_eq!(m.token("iffy", 0).unwrap().patterns(), &[1]);
// Restricted to the keyword, the answer is `if` — not nothing, which is
// what filtering the unrestricted answer would have produced.
assert_eq!(m.token_among("iffy", 0, &[0]).unwrap().len(), 2);

An empty allow permits nothing, which is a real question with a knowable answer (None) rather than an error: a lexer state can legitimately reach a point where no terminal is legal. Naming a pattern the engine declined is a no-op, so a caller with a fallback need not also remember which its blind terminals were.

§Panics

As Munch::token. See Munch::try_token_among.

Source

pub fn try_token_among( &self, text: &str, at: usize, allow: &[u32], ) -> Result<Option<Token>, Error>

Munch::token_among, reporting a fault instead of panicking.

§Errors

As Munch::try_token.

Source

pub fn shortest_among( &self, text: &str, at: usize, allow: &[u32], ) -> Option<Token>

The shortest non-empty token beginning at exactly at, restricted to the patterns allow names. See Pick::Shortest for when that is the question you have.

§Panics

As Munch::token. See Munch::try_shortest_among.

Source

pub fn try_shortest_among( &self, text: &str, at: usize, allow: &[u32], ) -> Result<Option<Token>, Error>

Munch::shortest_among, reporting a fault instead of panicking.

§Errors

As Munch::try_token.

Source

pub fn scan_into( &self, text: &str, at: usize, allow: Option<&[u32]>, pick: Pick, winners: &mut Vec<u32>, ) -> Result<Option<usize>, Error>

One scan, writing the winning patterns into winners instead of allocating — the allocation-free form the other four are built on.

A lexer calls this once per token, and a Box<[u32]> per token is a real cost for an answer that is almost always one number long. This form reuses the caller’s buffer: winners is cleared, filled with the winning pattern indices ascending, and the token’s byte length is returned. Ok(None) means nothing starts at at, and leaves winners empty.

let m = irgx::Munch::new(["[a-z]+", "[0-9]+", r"\s+"])?;
let text = "ab 12";
let mut winners = Vec::with_capacity(m.admitted());
let mut at = 0;
let mut spans = Vec::new();
while at < text.len() {
    let len = m
        .scan_into(text, at, None, irgx::Pick::Longest, &mut winners)?
        .filter(|len| *len > 0)
        .expect("every byte of this text starts some token");
    spans.push((at..at + len, winners[0]));
    at += len;
}
assert_eq!(spans, vec![(0..2, 0), (2..3, 2), (3..5, 1)]);