Skip to main content

python312_lexer

Function python312_lexer 

Source
pub fn python312_lexer(
    haystack: &str,
    out_tok_types: &str,
    out_tok_starts: &str,
    out_tok_lens: &str,
    out_counts: &str,
    haystack_len: u32,
) -> Program
Expand description

GPU Python 3.12 sparse lexer.

Each invocation owns one byte offset. Token starts write their classification to the same index; all other offsets stay zero.