Function tokenize 

pub fn tokenize(source: &str) -> Result<Vec<(PtxToken, Span)>, LexError>

Tokenize a PTX source string into a sequence of tokens with their spans.

This is the main entry point for lexical analysis. It converts raw PTX source code into a vector of tokens that can be parsed.

Arguments

• source - The PTX source code as a string slice

Returns

Returns a vector of tuples containing each token and its span in the source, or a LexError if tokenization fails.

Example

use ptx_parser::tokenize;

let source = ".version 8.5\n.target sm_90";
let tokens = tokenize(source).expect("Failed to tokenize");