Struct logos::Lexer

pub struct Lexer<Token: Logos, Source> {
    pub source: Source,
    pub token: Token,
    pub extras: Token::Extras,
    // some fields omitted
}

Lexer is the main struct of the crate that allows you to read through a Source and produce tokens for enums implementing the Logos trait.

Fields

source: Source

Source from which the Lexer is reading tokens.

token: Token

Current token. Call the advance method to get a new token.

extras: Token::Extras

Extras associated with the Token.

Methods

impl<'source, Token, Source> Lexer<Token, Source> where
    Token: Logos + WithSource<Source>,
    Source: Source<'source>, 

pub fn new(source: Source) -> Self

Create a new Lexer.

Due to type inference, it might be more ergonomic to construct it by calling Token::lexer(source), where Token implements Logos.

pub fn advance(&mut self)

Advance the Lexer and attempt to produce the next Token.

pub fn range(&self) -> Range<usize>

Get the range for the current token in Source.

pub fn slice(&self) -> Source::Slice

Get a string slice of the current token.

pub fn morph<Token2>(self) -> Lexer<Token2, Source> where
    Token2: Logos + WithSource<Source>,
    Token::Extras: Into<Token2::Extras>, 

Turn this lexer into a lexer for a new token type.

The new lexer continues to point at the same span as the current lexer, and the current token becomes the error token of the new token type. If you want to start reading from the new lexer immediately, consider using Lexer::advance_as instead.

pub fn advance_as<Token2>(self) -> Lexer<Token2, Source> where
    Token2: Logos + WithSource<Source>,
    Token::Extras: Into<Token2::Extras>, 

Advance the Lexer and attempt to produce the next Token of a new token type.

This function takes self by value as a lint. If you're working with a &mut Lexer, clone the old lexer to call this method, then don't forget to update the old lexer!

Trait Implementations

impl<Token: Clone + Logos, Source: Clone> Clone for Lexer<Token, Source> where
    Token::Extras: Clone, 

fn clone(&self) -> Lexer<Token, Source>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.