<div align="center">
# litext
[](https://doc.rust-lang.org/edition-guide/rust-2024/)
[](https://crates.io/crates/litext)
[](https://docs.rs/litext)
[](https://codeberg.org/razkar/litext/src/branch/main/LICENSE-MIT)
[](https://codeberg.org/razkar/litext/src/branch/main/LICENSE-APACHE)
[](https://crates.io/crates/litext)
[](https://deps.rs/repo/codeberg/razkar/litext)
[](https://codeberg.org/razkar/litext)
Literal extraction for proc-macro authors. Pull typed values out of `TokenStream` input without the boilerplate.
```rust
let value: String = litext!(input);
let count: u32 = litext!(input as u32);
let (s, n): (String, i32) = litext!(input as (String , i32));
```
</div>
## What is this?
When you write a procedural macro, you often receive a `TokenStream` that contains a single literal and need to pull out its value. Doing this correctly means handling every literal kind, every radix, escape sequences, raw strings, byte strings, C strings, and all the edge cases. `litext` does all that for you.
The library supports every Rust literal kind, span-aware wrapper types for precise diagnostics, a `ToTokens` round-trip trait, a `FromLit` extension point for custom types, and a variadic multi-extraction form for pulling several values out of one `TokenStream` at once.
## Installation
```sh
cargo add litext
```
Your proc-macro entry point needs to bridge `proc_macro` and `proc_macro2`:
```rust
#[proc_macro]
pub fn my_macro(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
my_macro_inner(input.into()).into()
}
fn my_macro_inner(input: proc_macro2::TokenStream) -> proc_macro2::TokenStream {
let s: String = litext!(input);
// ...
}
```
## Quick Start
Extract a string literal:
```rust
use litext::{litext, TokenStream};
fn my_macro(input: TokenStream) -> TokenStream {
let text: String = litext!(input);
// Returns early with a compile error if extraction fails.
// Use `litext!(input as String)` for the same result, explicitly typed.
}
```
Extract other literal types:
```rust
let num: i32 = litext!(input as i32);
let val: f64 = litext!(input as f64);
let ch: char = litext!(input as char);
let flag: bool = litext!(input as bool);
```
Keep the `Result` instead of returning early:
```rust
let result: Result<String, TokenStream> = litext!(try input);
match result {
Ok(text) => { /* use text */ }
Err(e) => return e,
}
// Same with an explicit type:
let result: Result<i32, TokenStream> = litext!(try input as i32);
```
## Extracting Multiple Literals at Once
`litext!` can extract several values from one `TokenStream` in a single call. Wrap the types in a tuple and put the separator token between them:
```rust
// Input TokenStream: "hello" , 42
let (s, n): (String, i32) = litext!(input as (String , i32));
// Three values separated by commas:
// Input: "hello" , 3.14 , true
let (s, f, b): (String, f64, bool) = litext!(input as (String , f64 , bool));
// Semicolon separator:
// Input: "key" ; 100
let (key, val): (String, u32) = litext!(input as (String ; u32));
```
The `try` form also works for tuples:
```rust
let result: Result<(String, i32), _> = litext!(try input as (String , i32));
```
Up to 12 values can be extracted in one call.
**Separator rules:** The separator written in the macro call and the separator in the actual `TokenStream` must be the same single punctuation character. Any single ASCII punctuation character works: `,`, `;`, `|`, `&`, `+`, `-`, `*`, `/`, `%`, `^`, `!`, `?`, `<`, `>`, `=`, `.`, `:`, `@`, `~`.
Multi-character punctuation sequences such as `->`, `=>`, and `::` do not work as separators. Rust tokenizes them as multiple tokens, so only the first character would be consumed and the extraction would fail. Grouping delimiters (`(`, `)`, `[`, `]`, `{`, `}`) and `$` are also not valid separators.
**Note:** Type arguments in tuple position must be single-token identifiers. Generic types like `LitInt<u8>` span multiple tokens and cannot appear in this position. Use the default forms (`LitInt`, `LitFloat`) which carry their default type parameters.
## Span-Aware Types
Every literal kind has a span-aware wrapper that bundles the parsed value with its source location. Use these when you need to emit a compiler diagnostic pointing at the exact literal in the user's code.
| `String` | `LitStr` | `"..."`, `r#"..."#` |
| `i8` to `i128`, `isize` | `LitInt<T>` (default: `i32`) | `42`, `0xFF`, `0b1010` |
| `u8` to `u128`, `usize` | `LitInt<T>` (default: `i32`) | same as above |
| `f32`, `f64` | `LitFloat<T>` (default: `f64`) | `3.14`, `1e10` |
| `bool` | `LitBool` | `true`, `false` |
| `char` | `LitChar` | `'a'`, `'\n'` |
| `u8` | `LitByte` | `b'a'`, `b'\xff'` |
| `Vec<u8>` | `LitByteStr` | `b"..."`, `br#"..."#` |
| `CString` | `LitCStr` | `c"..."`, `cr#"..."#` |
```rust
use litext::{litext, TokenStream};
use litext::literal::LitStr;
fn my_macro(input: TokenStream) -> TokenStream {
let lit: LitStr = litext!(input as LitStr);
if lit.value().is_empty() {
return comperr::error(lit.span(), "string cannot be empty");
}
// Use lit.value() for the content, lit.span() for diagnostics.
}
```
```rust
use litext::literal::LitInt;
let lit: LitInt<u8> = litext!(input as LitInt<u8>);
let value: u8 = *lit.value();
if value == 0 {
return comperr::error(lit.span(), "value cannot be zero");
}
```
All span-aware types expose:
- `.value()` - the parsed value
- `.span()` - the source location
- `.suffix()` - the explicit type suffix if present (`"i32"` for `42i32`, `None` for `42`)
## Round-Tripping with ToTokens
All span-aware types implement `ToTokens`, which converts them back into a `TokenStream`. This enables an extract, validate, emit pattern:
```rust
use litext::literal::{LitStr, ToTokens};
let lit: LitStr = litext!(input as LitStr);
if lit.value().starts_with("_") {
return comperr::error(lit.span(), "string cannot start with underscore");
}
// Emit the literal back into the output with its original span preserved.
lit.to_token_stream()
```
## Custom Literal Types
Implement `FromLit` to make `litext!(input as MyType)` work for your own types:
```rust
use litext::literal::FromLit;
use proc_macro2::{Literal, TokenStream};
pub struct NonEmpty(String);
impl FromLit for NonEmpty {
fn from_lit(lit: Literal) -> Result<Self, TokenStream> {
let s = String::from_lit(lit)?;
if s.is_empty() {
return Err(comperr::error(
proc_macro2::Span::call_site(),
"string cannot be empty",
));
}
Ok(NonEmpty(s))
}
}
// Then in your macro:
let val: NonEmpty = litext!(input as NonEmpty);
```
Override `from_ident` if your type is represented as an identifier token (like `bool`):
```rust
fn from_ident(ident: proc_macro2::Ident) -> Result<Self, TokenStream> {
// handle true/false/etc.
}
```
## What Is Supported
### Strings
| Regular | `"hello world"` |
| Escape sequences | `"\n"`, `"\t"`, `"\\"`, `"\""`, `"\0"`, `"\x41"`, `"\u{1F600}"` |
| Line continuation | `"hello \` + newline + `world"` |
| Raw | `r#"no escapes here"#` |
| Raw (multiple hashes) | `r##"can contain #"##` |
### Integers
| Decimal | `42` |
| Hexadecimal | `0xFF`, `0XFF` |
| Octal | `0o77`, `0O77` |
| Binary | `0b1010`, `0B1010` |
| Underscore separators | `1_000_000`, `0xFF_FF` |
| Type suffix | `42i32`, `255u8`, `100usize` |
All integer types are supported: `i8` to `i128`, `isize`, `u8` to `u128`, `usize`. Overflow returns a compile error.
### Floats
| Standard | `3.14` |
| Scientific | `1e10`, `2.5e-3`, `1E10` |
| Underscore separators | `1_000.5` |
| Type suffix | `3.14f32`, `1e10f64` |
### Characters
Full escape support: `'a'`, `'\n'`, `'\t'`, `'\\'`, `'\''`, `'\"'`, `'\0'`, `'\x41'`, `'\u{1F600}'`.
### Booleans
`true` and `false` are parsed as identifier tokens, not literals. All `litext` extraction handles this transparently.
### Byte Literals
`b'a'`, `b'\n'`, `b'\xff'`. The full 0x00..=0xFF range is valid for `\x` escapes in byte literals.
### Byte Strings
`b"hello"`, `b"\xff\x80"`, `br#"raw bytes"#`. The full byte range is supported for `\x` escapes.
### C Strings
`c"hello"`, `cr#"raw"#`. Interior null bytes are rejected with a compile error.
## Error Handling
All errors are returned as `TokenStream` values containing `compile_error!` invocations. When returned from a proc-macro entry point, the compiler displays the error at the correct source location.
Negative numbers (`-42`) are two tokens in Rust's token stream: a `-` punctuation token followed by a positive literal. `litext` does not support them. Parse the sign and magnitude separately if needed.
## Planned Features
## Planned
* Feature flag gates to `FromLit` impls for standard types, makes `litext` lighter (2.0)
* Multi-character separators if possible (<=2.0)
* Negative numbers maybe (<=2.0)
## Requirements
- Rust 2024 edition (Rust 1.85)
- `proc_macro2` as a direct dependency in your crate
## License
Licensed under either of:
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
- MIT license ([LICENSE-MIT](LICENSE-MIT) or <http://opensource.org/licenses/MIT>)
at your option.
Cheers, RazkarStudio
© 2026 RazkarStudio. All rights reserved.