Struct RustLexer

Source

pub struct RustLexer<'config> { /* private fields */ }

Expand description

A lexer for the Rust programming language.

The RustLexer is responsible for tokenizing Rust source code into a sequence of tokens that can be used by the parser. It handles all Rust syntax including modern features like raw strings, byte strings, lifetimes, and all standard Rust keywords.

§Examples

Basic usage:

use oak_core::{Lexer, SourceText};
use oak_rust::{RustLanguage, RustLexer};

let language = RustLanguage::default();
let lexer = RustLexer::new(&language);
let source = SourceText::new("fn main() { println!(\"Hello, world!\"); }");
let output = lexer.lex(source, 0);

// The output contains tokens for the entire source code
assert!(!output.tokens.is_empty());

Tokenizing different Rust constructs:

use oak_core::{Lexer, SourceText};
use oak_rust::{RustLanguage, RustLexer, SourceText};

let language = RustLanguage::default();
let lexer = RustLexer::new(&language);

// Tokenize a function with various Rust features
let source = SourceText::new(
    r#"
fn calculate<'a>(x: &'a i32, y: i32) -> i32 {
    let result = x + y;
    println!("Result: {}", result);
    result
}
"#,
);
let output = lexer.lex(source, 0);

// Verify that tokens were generated
assert!(output.tokens.len() > 10);

Implementations§

Source §

impl<'config> RustLexer<'config>

Source

pub fn new(config: &'config RustLanguage) -> Self

Creates a new RustLexer with the given language configuration.

§Parameters

config - A reference to the RustLanguage configuration that controls language-specific parsing behavior.

§Examples


let language = RustLanguage::default();
let lexer = RustLexer::new(&language);

Source

pub fn whitespace_rules(&self) -> &WhitespaceConfig

Returns the whitespace configuration for the lexer.

This method defines how the lexer should handle whitespace characters. The configuration enables Unicode whitespace support, allowing the lexer to recognize all Unicode whitespace characters, not just ASCII spaces.

Source

pub fn comment_rules(&self) -> &CommentLine

Returns the comment configuration for the lexer.

This method defines how the lexer should handle line comments in Rust code. Rust uses double slashes (//) for line comments, which continue to the end of the line.

Source

pub fn string_rules(&self) -> &StringConfig

Returns the string literal configuration for the lexer.

This method defines how the lexer should handle string literals in Rust code. Rust strings are enclosed in double quotes (“) and support escape sequences using backslash () as the escape character.

Source

pub fn char_rules(&self) -> &StringConfig

Returns the character literal configuration for the lexer.

This method defines how the lexer should handle character literals in Rust code. Rust character literals are enclosed in single quotes (’) and do not use escape characters in the same way as strings.