1 2 3
# Generic Tokenizer This crate provides a generic tokenizer that includes line and column numbers. This is useful for things like higher-level tokenizers, parsers, and language servers that need to track line and column numbers as they go.