strslice 0.1.1

ust library that provides zero copy string iterators for working with string slices. The library offers iterators similar to standard Rust string methods
Documentation 
1
2
3
4
5
6
7
8
9
10
11

// src/tokenizer.rs
// Tokenizer iterator
// Splits a string slice into tokens without allocating new strings.
// Each token is represented by the `Token<'a>` enum, which may be a keyword,
// identifier, number, operator, or other syntactic unit.
// The iterator returns references to the original string slice (&'a str) for zero-copy performance.
// Usage example:
// let mut tokenizer = Tokenizer::new("let x = 42 + y");
// while let Some(token) = tokenizer.next() {
//     println!("{:?}", token);
// }