Struct Cursor 

pub struct Cursor<'a> { /* private fields */ }

A zero-copy cursor over a &str, the primitive every lexer is built on.

A Cursor walks source text one char at a time, tracking a byte position and the start of the token currently being scanned. A lexer — hand-written or generated — drives it with the peek/advance primitives (first, second, bump, bump_if, eat_while), then turns the scanned run into a Token<K> with emit. token-lang says what a token is; the cursor says where one ends.

It is zero-copy and allocation-free: it borrows the source for its whole life, reports positions as BytePos and runs as Span, and hands back lexemes as borrowed &str slices. Advancing is O(1); nothing here allocates.

Positions

Positions are reported in a global space offset by a base (0 by default). Construct with for_source to lex a SourceFile from a source-lang map, and the spans land in that map’s global position space — the same space a diag-lang label points with. Use with_base to set the base directly, or new for a standalone &str at base 0.

Examples

A whole, if tiny, lexer: identifiers, +, and whitespace, terminated by EOF.

use intern_lang::Interner;
use token_lang::{Symbol, Token, TokenKind};
use lexer_lang::Cursor;

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Kind {
    Ident(Symbol),
    Plus,
    Whitespace,
    Eof,
}
impl TokenKind for Kind {
    fn is_trivia(&self) -> bool { matches!(self, Kind::Whitespace) }
    fn is_eof(&self) -> bool { matches!(self, Kind::Eof) }
    fn symbol(&self) -> Option<Symbol> {
        match self { Kind::Ident(s) => Some(*s), _ => None }
    }
}

fn lex(src: &str, interner: &mut Interner) -> Vec<Token<Kind>> {
    let mut cursor = Cursor::new(src);
    let mut tokens = Vec::new();
    while let Some(c) = cursor.first() {
        let kind = if c.is_whitespace() {
            cursor.eat_while(char::is_whitespace);
            Kind::Whitespace
        } else if c == '+' {
            cursor.bump();
            Kind::Plus
        } else {
            cursor.eat_while(|c| !c.is_whitespace() && c != '+');
            Kind::Ident(cursor.intern_lexeme(interner))
        };
        tokens.push(cursor.emit(kind));
    }
    tokens.push(cursor.emit(Kind::Eof));
    tokens
}

let mut interner = Interner::new();
let tokens = lex("foo + bar", &mut interner);
let significant = tokens.iter().filter(|t| !t.is_trivia() && !t.is_eof()).count();
assert_eq!(significant, 3); // foo, +, bar
assert!(tokens.last().unwrap().is_eof());

Implementations

impl<'a> Cursor<'a>

pub fn new(text: &'a str) -> Self

Creates a cursor over text, reporting positions from base 0.

Examples

use lexer_lang::Cursor;

let mut cursor = Cursor::new("ab");
assert_eq!(cursor.bump(), Some('a'));
assert_eq!(cursor.bump(), Some('b'));
assert!(cursor.is_eof());

pub fn with_base(text: &'a str, base: u32) -> Self

Creates a cursor over text whose reported positions are offset by base.

Use this to lex a slice of a larger source while keeping spans in the larger source’s coordinate space. for_source is the usual way to get the base right for a source-lang file.

base + text.len() is expected to fit in u32 — the addressable envelope the span-lang / source-lang layers cap at. Position arithmetic saturates rather than wrapping if it does not, so a span is never off by 2^32.

Examples

use lexer_lang::{BytePos, Cursor};

let mut cursor = Cursor::with_base("xy", 100);
assert_eq!(cursor.pos(), BytePos::new(100));
cursor.bump();
assert_eq!(cursor.pos(), BytePos::new(101));

pub fn for_source(file: &'a SourceFile) -> Self

Creates a cursor over a SourceFile’s text, with its base set so spans land in the owning SourceMap’s global position space.

This is the bridge from source-lang to the lexer: the spans the cursor emits resolve directly against the same map a diagnostic renders through, so a token’s span points at the right file without any further arithmetic.

Examples

use source_lang::SourceMap;
use lexer_lang::Cursor;

let mut map = SourceMap::new();
map.add("a.txt", "first").expect("fits");      // 0..5
let id = map.add("b.txt", "x").expect("fits"); // 5..6

let file = map.source(id).unwrap();
let mut cursor = Cursor::for_source(file);
// The single token's span is in the map's global space, not 0-based.
assert_eq!(cursor.pos().to_u32(), 5);
cursor.bump();
assert_eq!(cursor.token_span().start().to_u32(), 5);

pub fn remaining(&self) -> &'a str

The unconsumed source text.

Examples

use lexer_lang::Cursor;

let mut cursor = Cursor::new("abc");
cursor.bump();
assert_eq!(cursor.remaining(), "bc");

pub fn is_eof(&self) -> bool

Whether the cursor has reached the end of the source.

pub fn pos(&self) -> BytePos

The current position, in the global space set by the base.

The base plus the source length must fit in u32 (the addressable envelope the source layers cap at); the addition saturates rather than wrapping if it somehow does not, so a position is never silently wrong by 2^32.

Examples

use lexer_lang::{BytePos, Cursor};

let mut cursor = Cursor::new("hi");
assert_eq!(cursor.pos(), BytePos::new(0));
cursor.bump();
assert_eq!(cursor.pos(), BytePos::new(1));

pub fn first(&self) -> Option<char>

Peeks the next character without consuming it, or None at end of input.

Examples

use lexer_lang::Cursor;

let cursor = Cursor::new("xy");
assert_eq!(cursor.first(), Some('x'));

pub fn second(&self) -> Option<char>

Peeks the character after first without consuming anything — the second character of lookahead, for tokens like ==, //, or ...

Examples

use lexer_lang::Cursor;

let cursor = Cursor::new("==");
assert_eq!(cursor.first(), Some('='));
assert_eq!(cursor.second(), Some('='));

pub fn bump(&mut self) -> Option<char>

Consumes and returns the next character, or None at end of input.

Examples

use lexer_lang::Cursor;

let mut cursor = Cursor::new("ab");
assert_eq!(cursor.bump(), Some('a'));
assert_eq!(cursor.bump(), Some('b'));
assert_eq!(cursor.bump(), None);

pub fn bump_if(&mut self, expected: char) -> bool

Consumes the next character only if it equals expected, returning whether it did. Handy for two-character operators after the first is known.

Examples

use lexer_lang::Cursor;

let mut cursor = Cursor::new("=>");
assert!(cursor.bump_if('='));
assert!(cursor.bump_if('>'));
assert!(!cursor.bump_if('!')); // no match: nothing consumed
assert!(cursor.is_eof());

pub fn eat_while(&mut self, pred: impl FnMut(char) -> bool)

Consumes characters while pred holds.

Stops at the first character for which pred is false, or at end of input, leaving that character unconsumed. This is the workhorse for scanning runs — identifier bodies, digit sequences, whitespace. Read what was consumed back with lexeme; it returns nothing so a side-effect-only call stays clean under unused_results.

Examples

use lexer_lang::Cursor;

let mut cursor = Cursor::new("abc123");
cursor.eat_while(|c| c.is_ascii_alphabetic());
assert_eq!(cursor.lexeme(), "abc");
assert_eq!(cursor.remaining(), "123");

pub fn lexeme(&self) -> &'a str

The lexeme of the in-progress token: the source text consumed since the last emit (or since construction).

Examples

use lexer_lang::Cursor;

let mut cursor = Cursor::new("let x");
cursor.eat_while(|c| c.is_ascii_alphabetic());
assert_eq!(cursor.lexeme(), "let");

pub fn token_span(&self) -> Span

The Span of the in-progress token, in the global space set by the base: from where the last emit left off to the current position.

Examples

use lexer_lang::{Cursor, Span};

let mut cursor = Cursor::with_base("let", 10);
cursor.eat_while(|c| c.is_ascii_alphabetic());
assert_eq!(cursor.token_span(), Span::new(10, 13));

pub fn intern_lexeme(&self, interner: &mut Interner) -> Symbol

Interns the current lexeme into interner, returning its Symbol — the cheap handle an identifier or keyword token carries.

Examples

use intern_lang::Interner;
use lexer_lang::Cursor;

let mut interner = Interner::new();
let mut cursor = Cursor::new("name rest");
cursor.eat_while(|c| c.is_ascii_alphabetic());
let sym = cursor.intern_lexeme(&mut interner);
assert_eq!(interner.resolve(sym), Some("name"));

pub fn emit<K>(&mut self, kind: K) -> Token<K>

Ends the in-progress token, returning a Token<K> of kind spanning the consumed run, and starts the next token at the current position.

After emit, lexeme is empty and token_span is a zero-width span at the current position, until more is consumed. Emitting without having consumed anything — at end of input, say — yields a token with an empty span, the natural shape for an end-of-input marker.

Examples

use lexer_lang::{Cursor, Span, Token};

let mut cursor = Cursor::new("ab");
cursor.bump();
let first = cursor.emit("a");
assert_eq!(first, Token::new("a", Span::new(0, 1)));

// The next token starts where the last one ended.
cursor.bump();
let second = cursor.emit("b");
assert_eq!(second, Token::new("b", Span::new(1, 2)));

pub fn reset_token(&mut self)

Discards the in-progress run without producing a token, starting the next token at the current position.

This is the counterpart to emit for trivia a lexer drops rather than keeps: consume the whitespace or comment, then reset_token so it is not folded into the span of the token that follows. (A lexer that instead emits trivia — for a formatter or a lossless tree — uses emit with a trivia kind and never needs this.)

Examples

use lexer_lang::{Cursor, Span};

let mut cursor = Cursor::new("  x");
cursor.eat_while(char::is_whitespace);
cursor.reset_token(); // drop the leading spaces

cursor.eat_while(|c| c.is_ascii_alphabetic());
// The identifier's span covers only `x`, not the spaces before it.
assert_eq!(cursor.token_span(), Span::new(2, 3));
assert_eq!(cursor.lexeme(), "x");

Trait Implementations

impl<'a> Clone for Cursor<'a>

fn clone(&self) -> Cursor<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Cursor<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.