perl-lexer 0.11.0

High-performance Perl lexer with context-aware tokenization
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# perl-lexer
[![Crates.io](https://img.shields.io/crates/v/perl-lexer.svg)](https://crates.io/crates/perl-lexer)
[![Documentation](https://docs.rs/perl-lexer/badge.svg)](https://docs.rs/perl-lexer)

Context-aware Perl lexer with mode-based tokenization for the
[perl-lsp](https://github.com/EffortlessMetrics/perl-lsp) workspace.

## Overview

Handles Perl's inherently context-sensitive grammar by tracking lexer mode
(`ExpectTerm` vs `ExpectOperator`) to disambiguate `/` (division vs regex),
`%` (modulo vs hash sigil), heredocs, quote-like operators, and more. Provides
checkpointing for incremental parsing and budget limits to guard against
pathological input.

## Usage

```rust
use perl_lexer::{PerlLexer, TokenType};

let mut lexer = PerlLexer::new("my $x = 42;");
while let Some(token) = lexer.next_token() {
    if matches!(token.token_type, TokenType::EOF) { break; }
    println!("{:?}: {}", token.token_type, token.text);
}
```

## License

Licensed under either of [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)
or [MIT license](http://opensource.org/licenses/MIT) at your option.