Struct RustLexer 

pub struct RustLexer<'config> { /* private fields */ }

A lexer for the Rust programming language.

The RustLexer is responsible for tokenizing Rust source code into a sequence of tokens that can be used by the parser. It handles all Rust syntax including modern features like raw strings, byte strings, lifetimes, and all standard Rust keywords.

Examples

Basic usage:

use oak_core::{Lexer, LexerState, SourceText};
use oak_rust::{RustLanguage, RustLexer};

let language = RustLanguage::default();
let lexer = RustLexer::new(&language);
let source = SourceText::new("fn main() { println!(\"Hello, world!\"); }");
let mut cache = oak_core::parser::session::ParseSession::<RustLanguage>::default();
let output = lexer.lex(&source, &[], &mut cache);

// The output contains tokens for the entire source code
assert!(output.result.is_ok());

Tokenizing different Rust constructs:

use oak_core::{Lexer, LexerState, SourceText};
use oak_rust::{RustLanguage, RustLexer};

let language = RustLanguage::default();
let lexer = RustLexer::new(&language);

// Tokenize a function with various Rust features
let source = SourceText::new(
    r#"
fn calculate<'a>(x: &'a i32, y: i32) -> i32 {
    let result = x + y;
    println!("Result: {}", result);
    result
}
"#,
);
let mut cache = oak_core::parser::session::ParseSession::<RustLanguage>::default();
let output = lexer.lex(&source, &[], &mut cache);

// Verify that tokens were generated
assert!(output.result.is_ok());

Implementations

impl<'config> RustLexer<'config>

pub fn new(config: &'config RustLanguage) -> Self

Creates a new RustLexer with the given language configuration.

Parameters

• config - A RustLanguage configuration that controls language-specific parsing behavior.

Examples

let language = RustLanguage::default();
let lexer = RustLexer::new(&language);

Trait Implementations

impl<'config> Clone for RustLexer<'config>

fn clone(&self) -> RustLexer<'config>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'config> Lexer<RustLanguage> for RustLexer<'config>

fn lex<'a, S: Source + ?Sized>( &self, source: &'a S, _edits: &[TextEdit], cache: &'a mut impl LexerCache<RustLanguage>, ) -> LexOutput<RustLanguage>

Tokenizes the given source text into a sequence of tokens.