# Python-like Tokenizer in Rust
[](
https://github.com/salam99823/tokenizer)
[](https://crates.io/crates/tokenizer_py)
[](
https://rust-lang.github.io/rfcs/2495-min-rust-version.html)
[](
https://docs.rs/tokenizer_py)
[](
https://github.com/salam99823/tokenizer/actions/workflows/rust.yml)
This project implements a Python-like tokenizer in Rust.
It can tokenize a string into a sequence of tokens, which are
represented by the [`Token`] enum. The supported tokens are:
- [`Token::Name`]: a name token, such as a function or variable name.
- [`Token::Number`]: a number token, such as a literal integer or floating-point number.
- [`Token::String`]: a string token, such as a single or double-quoted string.
- [`Token::OP`]: an operator token, such as an arithmetic or comparison operator.
- [`Token::Indent`]: an indent token, indicating that a block of code is being indented.
- [`Token::Dedent`]: a dedent token, indicating that a block of code is being dedented.
- [`Token::Comment`]: a comment token, such as a single-line or multi-line comment.
- [`Token::NewLine`]: a newline token, indicating a new line in the source code.
- [`Token::NL`]: a token indicating a new line, for compatibility with the original tokenizer.
- [`Token::EndMarker`]: an end-of-file marker.
The tokenizer recognizes the following tokens:
- [x] `Whitespace`: spaces, tabs, and newlines.
- [x] `Numbers`: integers and floating-point numbers.
- [x] `float`: floats numbers.
- [x] `int`: integer numbers.
- [x] `complex`: complex numbers.
- [x] `Names`: identifiers and keywords.
- [x] `Strings`: single- and double-quoted strings.
- [x] `basic-String`: single- and double-quoted strings.
- [x] `format-String`: format string from python.
- [x] `byte-String`: byte string from python.
- [x] `raw-String`: raw string.
- [x] `multy-line-String`: single- and double-quoted multy-line-string.
- [ ] `combined-string`: string with combined prefix.
- [x] `Operators`: arithmetic, comparison, and other operators.
- [x] `Comments`: single-line comments.
The tokenizer also provides a `tokenize`
method that takes a string as input and returns a `Result` containing a vector
of tokens.
## Usage
Add this to your `Cargo.toml`:
```toml
[dependencies]
tokenizer_py = "0.2.0"
```
## Exemples
### Example of using the tokenizer to tokenize the string "hello world"
```rust
use tokenizer_py::{tokenize, Token};
let tokens = tokenize("hello world").unwrap();
assert_eq!(tokens, vec![
Token::Name("hello".to_string()), // Token of the name "hello"
Token::Name("world".to_string()), // Token of the name "world"
Token::NewLine, // New line token
Token::EndMarker, // End of text token
]);
```
### Example of using the BinaryExp structure to evaluate the binary expression "10 + 10"
```rust
use tokenizer_py::{tokenize, Token};
// Structure representing a binary expression
struct BinaryExp {
left: Token,
center: Token,
right: Token,
}
impl BinaryExp {
// Method for creating a new instance of BinaryExp
fn new(left: Token, center: Token, right: Token) -> Self {
BinaryExp { left, center, right }
}
// Method for executing the binary expression
fn execute(&self) -> Result<isize, <isize as std::str::FromStr>::Err> {
use Token::{Number, OP};
match (&self.left, &self.center, &self.right) {
(Number(ref left), OP(ref op), Number(ref right)) => {
let (left, right) = (
left.parse::<isize>()?, right.parse::<isize>()?
);
match op.as_str() {
"+" => Ok(left + right),
"-" => Ok(left - right),
"*" => Ok(left * right),
"/" => Ok(left / right),
"%" => Ok(left % right),
_ => panic!("Invalid operator"), // Invalid operator
}
}
_ => panic!("Invalid tokens"), // Invalid tokens
}
}
}
let mut tokens = tokenize("10 + 10").unwrap();
let _ = tokens.pop(); // Remove Token::EndMarker
let _ = tokens.pop(); // Remove Token::NewLine
let binexp = BinaryExp::new(
tokens.pop().unwrap(),
tokens.pop().unwrap(),
tokens.pop().unwrap()
);
assert_eq!(binexp.execute(), Ok(20)); // Checking the execution result
```
[`Token::Name`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.Name
[`Token::Number`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.Number
[`Token::String`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.String
[`Token::OP`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.OP
[`Token::Indent`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.Indent
[`Token::Dedent`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.Dedent
[`Token::Comment`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.Comment
[`Token::NewLine`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.NewLine
[`Token::NL`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.NL
[`Token::EndMarker`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#variant.EndMarker
[`Token`]: https://docs.rs/tokenizer_py/latest/tokenizer_py/enum.Token.html#