Trait Lexer 

pub trait Lexer<L>
where
    L: Language + Send + Sync + 'static,
{
    // Required method
    fn lex<'a, S>(
        &self,
        text: &S,
        edits: &[TextEdit],
        cache: &'a mut impl LexerCache<L>,
    ) -> OakDiagnostics<Arc<[Token<<L as Language>::TokenType>]>>
       where S: Source + ?Sized;
}

Trait for tokenizing source code into sequences of tokens.

This trait defines the interface for converting source text into a sequence of tokens that can be consumed by the parser. Implementations should handle the specific lexical rules of their target language.

Examples

struct MyLexer;

#[derive(Debug, Clone, PartialEq, Eq, Copy)]
enum MyToken {
    Number,
    Identifier,
    End,
}

impl TokenType for MyToken {
    const END_OF_STREAM: Self = MyToken::End;
    type Role = UniversalTokenRole;
    fn role(&self) -> Self::Role { UniversalTokenRole::None }
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum MyElement {}

impl ElementType for MyElement {
    type Role = UniversalElementRole;
    fn role(&self) -> Self::Role { UniversalElementRole::None }
}

struct MyLanguage;

impl Language for MyLanguage {
    const NAME: &'static str = "my-language";
    type TokenType = MyToken;
    type ElementType = MyElement;
    type TypedRoot = ();
}

impl Lexer<MyLanguage> for MyLexer {
    fn lex<'a, S: Source + ?Sized>(&self, text: &S, edits: &[TextEdit], cache: &'a mut impl LexerCache<MyLanguage>) -> LexOutput<MyLanguage> {
        // Tokenization logic here
        todo!()
    }
}

Required Methods

fn lex<'a, S>( &self, text: &S, edits: &[TextEdit], cache: &'a mut impl LexerCache<L>, ) -> OakDiagnostics<Arc<[Token<<L as Language>::TokenType>]>>

where S: Source + ?Sized,

Tokenizes the given source text into a sequence of tokens.

This method performs a full lexical analysis of the source text, creating a new sequence of tokens from scratch. It uses a default cache configuration.

Arguments

• source - The source text to tokenize

Returns

A LexOutput containing the tokens and any diagnostic messages

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors