Struct xxcalc::Tokens
[−]
[src]
pub struct Tokens { pub tokens: TokenList, pub identifiers: Identifiers, // some fields omitted }
Tokens is a compound storage for list of tokens with their identifiers.
A TokenList is invalid without Identifiers, therefore this struct keeps them always together. Furthermore a lookup table is kept, so creating new identifier is a relatively quick task.
Fields
tokens: TokenList
List of tokens
identifiers: Identifiers
Identifier strings used in Token::Identifier
Methods
impl Tokens
[src]
fn new(i_cap: Option<usize>) -> Tokens
Creates a new token storage with optional capacity of identifiers.
fn clear(&mut self)
Clears tokens list, however underlying identifiers vector and its lookup cache is not cleared, as identifiers may be reused.
fn push(&mut self, position: usize, token: Token)
Pushes a token (with its position) to the end of token list.
Examples
let mut tokens = Tokens::new(None); tokens.push(0, Token::Number(1.0)); assert_eq!(tokens.tokens.len(), 1); tokens.push(1, Token::Number(2.0)); assert_eq!(tokens.tokens.len(), 2);
fn push_identifier(&mut self, position: usize, value: &String)
Pushes a identifier (with its position) to the end of token list.
Using a lookup cache, an offset in identifier table is obtained. If identifier does not yet exist, it is added both to the identifier vector and to the lookup cache. Identifiers are case-agnostic. A Token::Identifier with appropriate offset is pushed to the end of the token list.
Examples
let mut tokens = Tokens::new(None); tokens.push_identifier(0, &String::from("FOO")); assert_eq!(tokens[0], (0, Token::Identifier(0))); tokens.push_identifier(4, &String::from("foo")); assert_eq!(tokens[1], (4, Token::Identifier(0))); tokens.push_identifier(8, &String::from("bar")); assert_eq!(tokens[2], (8, Token::Identifier(1))); assert_eq!(tokens.identifiers, ["foo", "bar"]);
Trait Implementations
impl Debug for Tokens
[src]
impl Index<usize> for Tokens
[src]
Returns position and token with given index.
One should note that token position is indepentent from its index.
Panics
It will panic when the index is greater than number of tokens stored in the list.
Examples
let mut tokens = Tokens::new(None); tokens.push(0, Token::Number(1.0)); tokens.push(8, Token::Number(2.0)); assert_eq!(tokens[0], (0, Token::Number(1.0))); assert_eq!(tokens[1], (8, Token::Number(2.0)));