Struct Tokens 

pub struct Tokens {
    pub tokens: TokenList,
    pub identifiers: Identifiers,
    /* private fields */
}

Tokens is a compound storage for list of tokens with their identifiers.

A TokenList is invalid without Identifiers, therefore this struct keeps them always together. Furthermore a lookup table is kept, so creating new identifier is a relatively quick task.

Fields

tokens: TokenList

List of tokens

identifiers: Identifiers

Identifier strings used in Token::Identifier

Implementations

impl Tokens

pub fn new(i_cap: Option<usize>) -> Tokens

Creates a new token storage with optional capacity of identifiers.

pub fn clear(&mut self)

Clears tokens list, however underlying identifiers vector and its lookup cache is not cleared, as identifiers may be reused.

pub fn push(&mut self, position: usize, token: Token)

Pushes a token (with its position) to the end of token list.

Examples

let mut tokens = Tokens::new(None);

tokens.push(0, Token::Number(1.0));
assert_eq!(tokens.tokens.len(), 1);

tokens.push(1, Token::Number(2.0));
assert_eq!(tokens.tokens.len(), 2);

pub fn push_identifier(&mut self, position: usize, value: &String)

Pushes a identifier (with its position) to the end of token list.

Using a lookup cache, an offset in identifier table is obtained. If identifier does not yet exist, it is added both to the identifier vector and to the lookup cache. Identifiers are case-agnostic. A Token::Identifier with appropriate offset is pushed to the end of the token list.

Examples

let mut tokens = Tokens::new(None);

tokens.push_identifier(0, &String::from("FOO"));
assert_eq!(tokens[0], (0, Token::Identifier(0)));
tokens.push_identifier(4, &String::from("foo"));
assert_eq!(tokens[1], (4, Token::Identifier(0)));
tokens.push_identifier(8, &String::from("bar"));
assert_eq!(tokens[2], (8, Token::Identifier(1)));
assert_eq!(tokens.identifiers, ["foo", "bar"]);

Trait Implementations

impl Debug for Tokens

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Index<usize> for Tokens

Returns position and token with given index.

One should note that token position is indepentent from its index.

Panics

It will panic when the index is greater than number of tokens stored in the list.

Examples

let mut tokens = Tokens::new(None);

tokens.push(0, Token::Number(1.0));
tokens.push(8, Token::Number(2.0));

assert_eq!(tokens[0], (0, Token::Number(1.0)));
assert_eq!(tokens[1], (8, Token::Number(2.0)));

type Output = (usize, Token)

The returned type after indexing.

fn index(&self, idx: usize) -> &(usize, Token)

Performs the indexing (container[index]) operation.