Crate moxy_token 

Moxy tokens

Token types, lexing, spans, and compiler-token conversions for Rust syntax tooling. TokenStream is an owned sequence of TokenTree values that preserves token text, punctuation spacing, and source spans.

Quick start

Parse source into a token stream, or construct individual token types with the provided macros:

use moxy::token::{ident, TokenStream};
use moxy::Token;

let stream: TokenStream = "fn generated() {}".parse().unwrap();
let name = ident!(generated);
let semi: Token![;] = Default::default();
assert_eq!(name.to_string(), "generated");
assert_eq!(semi.to_string(), ";");

Integrations

Enable serde to serialize supported token types. Enable proc-macro2 for conversions with proc_macro2;

Modules

source

Source files, locations, and source maps used by spans.

Macros

Token

Map a Rust punctuation or keyword symbol to its crate token type.

ident

Builds an Ident from one or more segments, lexing the result and panicking on invalid identifier syntax.

Structs

And

The Rust punctuation token &.

As

The Rust keyword token as.

Async

The Rust keyword token async.

At

The Rust punctuation token @.

Auto

The Rust keyword token auto.

Await

The Rust keyword token await.

Become

The Rust keyword token become.

Box

The Rust keyword token box.

Break

The Rust keyword token break.

Caret

The Rust punctuation token ^.

Colon

The Rust punctuation token :.

Comma

The Rust punctuation token ,.

Const

The Rust keyword token const.

Continue

The Rust keyword token continue.

Crate

The Rust keyword token crate.

Cursor

Zero-copy immutable cursor over source text. Each parse step returns a new advanced cursor. A copyable cursor over source text during lexing.

Default

The Rust keyword token default.

DelimSpan

The opening and closing spans of a delimited token group.

Do

The Rust keyword token do.

Dollar

The Rust punctuation token $.

Dot

The Rust punctuation token ..

Dyn

The Rust keyword token dyn.

Else

The Rust keyword token else.

Enum

The Rust keyword token enum.

Eq

The Rust punctuation token =.

Extern

The Rust keyword token extern.

Final

The Rust keyword token final.

Fn

The Rust keyword token fn.

For

The Rust keyword token for.

Group

A delimited sequence of token trees.

Gt

The Rust punctuation token >.

Ident

A Rust identifier token, including its source span and raw-identifier form.

If

The Rust keyword token if.

Impl

The Rust keyword token impl.

In

The Rust keyword token in.

Let

The Rust keyword token let.

LexError

An error returned while lexing Rust source into tokens.

LitBool

A parsed Rust bool literal token.

LitByte

A parsed Rust byte literal token.

LitByteStr

A parsed Rust byte str literal token.

LitCStr

A parsed Rust cstr literal token.

LitChar

A parsed Rust char literal token.

LitF32

A parsed Rust f32 literal token.

LitF64

A parsed Rust f64 literal token.

LitInt

A parsed Rust int literal token.

LitStr

A parsed Rust str literal token.

LitVerbatim

A parsed Rust verbatim literal token.

Location

0 indexed char based location A byte, line, and column location within a source file.

Loop

The Rust keyword token loop.

Lt

The Rust punctuation token <.

Macro

The Rust keyword token macro.

MacroRules

The Rust keyword token macro_rules.

Match

The Rust keyword token match.

Minus

The Rust punctuation token -.

Mod

The Rust keyword token mod.

Move

The Rust keyword token move.

Mut

The Rust keyword token mut.

Not

The Rust punctuation token !.

Or

The Rust punctuation token |.

Override

The Rust keyword token override.

Percent

The Rust punctuation token %.

Plus

The Rust punctuation token +.

Pound

The Rust punctuation token #.

Priv

The Rust keyword token priv.

Pub

The Rust keyword token pub.

Question

The Rust punctuation token ?.

Quote

The Rust punctuation token '.

RangeSpan

A span covering a byte range in a source file.

Raw

The Rust keyword token raw.

Ref

The Rust keyword token ref.

Return

The Rust keyword token return.

Safe

The Rust keyword token safe.

SelfType

The Rust keyword token Self.

SelfValue

The Rust keyword token self.

Semi

The Rust punctuation token ;.

Slash

The Rust punctuation token /.

Star

The Rust punctuation token *.

Static

The Rust keyword token static.

Struct

The Rust keyword token struct.

Super

The Rust keyword token super.

Tilde

The Rust punctuation token ~.

TokenStream

An ordered, owned sequence of Rust token trees.

Trait

The Rust keyword token trait.

Try

The Rust keyword token try.

Type

The Rust keyword token type.

Typeof

The Rust keyword token typeof.

Underscore

The Rust punctuation token _.

Union

The Rust keyword token union.

Unsafe

The Rust keyword token unsafe.

Unsized

The Rust keyword token unsized.

Use

The Rust keyword token use.

Virtual

The Rust keyword token virtual.

Where

The Rust keyword token where.

While

The Rust keyword token while.

Yield

The Rust keyword token yield.

Enums

Delim

The delimiter kind of a grouped token sequence.

IntSuffix

The optional primitive-integer suffix of an integer literal.

Keyword

A Rust keyword token.

Lit

A parsed Rust literal token.

LitFloat

A parsed Rust float literal token.

Punct

A Rust punctuation token.

Radix

The radix used by an integer literal.

Spacing

Whether a Punct is followed immediately by another Punct or followed by another token or whitespace. Whether adjacent punctuation is joint or separated.

Span

A compiler or fallback source span.

TokenTree

One Rust token: a group, identifier, literal, punctuation mark, or keyword.

Traits

Scan

Parses one token value from the beginning of a lexer cursor.

Spanner

Provides the source span occupied by a value.

ToTokenStream

Convenience methods for values that can emit a TokenStream. A public Rust token API type for to token stream:.

ToTokens

Emits a value as tokens into a destination token stream. A public Rust token API type for to tokens<t.

TryIntoTokenStream

Fallibly converts a parsing input into a TokenStream.