Struct SyntaxToken

Source

pub struct SyntaxToken<S: Syntax, D: 'static = ()> { /* private fields */ }

Expand description

Syntax tree token.

Implementations§

Source §

impl<S: Syntax, D> SyntaxToken<S, D>

Source

pub fn write_debug<R>(&self, resolver: &R, target: &mut impl Write) -> Result
where R: Resolver<TokenKey> + ?Sized,

Writes this token’s Debug representation into the given target.

Source

pub fn debug<R>(&self, resolver: &R) -> String
where R: Resolver<TokenKey> + ?Sized,

Returns this token’s Debug representation as a string.

To avoid allocating for every token, see write_debug.

Source

pub fn write_display<R>(&self, resolver: &R, target: &mut impl Write) -> Result
where R: Resolver<TokenKey> + ?Sized,

Writes this token’s Display representation into the given target.

Source

pub fn display<R>(&self, resolver: &R) -> String
where R: Resolver<TokenKey> + ?Sized,

Returns this token’s Display representation as a string.

To avoid allocating for every token, see write_display.

Source

pub fn resolver(&self) -> Option<&StdArc<dyn Resolver<TokenKey>>>

If there is a resolver associated with this tree, returns it.

Source

pub fn try_resolved(&self) -> Option<&ResolvedToken<S, D>>

Turns this token into a ResolvedToken, but only if there is a resolver associated with this tree.

Source

pub fn resolved(&self) -> &ResolvedToken<S, D>

Turns this token into a ResolvedToken.

§Panics

If there is no resolver associated with this tree.

Source §

impl<S: Syntax, D> SyntaxToken<S, D>

Source

pub fn replace_with(&self, replacement: GreenToken) -> GreenNode

Returns a green tree, equal to the green tree this token belongs two, except with this token substitute. The complexity of operation is proportional to the depth of the tree

Source

pub fn syntax_kind(&self) -> RawSyntaxKind

The internal representation of the kind of this token.

Source

pub fn kind(&self) -> S

The kind of this token in terms of your language.

Source

pub fn text_range(&self) -> TextRange

The range this token covers in the source text, in bytes.

Source

pub fn resolve_text<'i, I>(&self, resolver: &'i I) -> &'i str
where I: Resolver<TokenKey> + ?Sized,

Uses the provided resolver to return the source text of this token.

If no text is explicitly associated with the token, returns its static_text instead.

Source

pub fn static_text(&self) -> Option<&'static str>

If the syntax kind of this token always represents the same text, returns that text.

§Examples

If there is a syntax kind Plus that represents just the + operator and we implement Syntax::static_text for it, we can retrieve this text in the resulting syntax tree.

let mut builder: GreenNodeBuilder<MySyntax> = GreenNodeBuilder::new();
let tree = parse(&mut builder, "x + 3");
let plus = tree
    .children_with_tokens()
    .nth(2) // `x`, then a space, then `+`
    .unwrap()
    .into_token()
    .unwrap();
assert_eq!(plus.static_text(), Some("+"));

Source

pub fn text_eq(&self, other: &Self) -> bool

Returns true if self and other represent equal source text.

This method is different from the PartialEq and Eq implementations in that it compares only the token text and not its source position. It is more efficient than comparing the result of resolve_text because it compares the tokens’ interned text_keys (if their text is not static) or their kind (if it is). Therefore, it also does not require a Resolver.

Note that the result of the comparison may be wrong when comparing two tokens from different trees that use different interners.

§Examples

let mut builder: GreenNodeBuilder<MySyntax> = GreenNodeBuilder::new();
let tree = parse(&mut builder, "x + x + 3");
let mut tokens = tree.children_with_tokens();
let tokens = tokens.by_ref();
let first_x = tokens.next().unwrap().into_token().unwrap();

// For the other tokens, skip over the whitespace between them
let first_plus = tokens.skip(1).next().unwrap().into_token().unwrap();
let second_x = tokens.skip(1).next().unwrap().into_token().unwrap();
let second_plus = tokens.skip(1).next().unwrap().into_token().unwrap();
assert!(first_x.text_eq(&second_x));
assert!(first_plus.text_eq(&second_plus));

Source

pub fn text_key(&self) -> Option<TokenKey>

Returns the interned key of text covered by this token, if any. This key may be used for comparisons with other keys of strings interned by the same interner.

§Examples

If you intern strings inside of your application, like inside a compiler, you can use token’s text keys to cross-reference between the syntax tree and the rest of your implementation by re-using the interner in both.

use cstree::interning::{TokenInterner, TokenKey, new_interner};
struct TypeTable {
    // ...
}
impl TypeTable {
    fn type_of(&self, ident: TokenKey) -> &str {
        // ...
    }
}
let interner = new_interner();
let mut state = State {
    interner,
    type_table: TypeTable{ /* stuff */},
};
let mut builder: GreenNodeBuilder<MySyntax, TokenInterner> =
    GreenNodeBuilder::with_interner(&mut state.interner);
let tree = parse(&mut builder, "x");
let type_table = &state.type_table;
let ident = tree
    .children_with_tokens()
    .next()
    .unwrap()
    .into_token()
    .unwrap();
let typ = type_table.type_of(ident.text_key().unwrap());

Source

pub fn green(&self) -> &GreenToken

Returns the unterlying green tree token of this token.

Source

pub fn parent(&self) -> &SyntaxNode<S, D>

The parent node of this token.

Source

pub fn ancestors(&self) -> impl Iterator<Item = &SyntaxNode<S, D>>

Returns an iterator along the chain of parents of this token.

Source

pub fn next_sibling_or_token(&self) -> Option<SyntaxElementRef<'_, S, D>>

The tree element to the right of this one, i.e. the next child of this token’s parent after this token.

Source

pub fn prev_sibling_or_token(&self) -> Option<SyntaxElementRef<'_, S, D>>

The tree element to the left of this one, i.e. the previous child of this token’s parent after this token.

Source

pub fn siblings_with_tokens( &self, direction: Direction, ) -> impl Iterator<Item = SyntaxElementRef<'_, S, D>>

Returns an iterator over all siblings of this token in the given direction, i.e. all of this token’s parent’s children from this token on to the left or the right. The first item in the iterator will always be this token.

Source