Struct TokenizerConfig

pub struct TokenizerConfig { /* private fields */ }

Config for the tokenizer, for flags

Implementations

impl TokenizerConfig

pub fn comment(self, enabled: bool) -> Self

Support comment in the output of the tokenizer or filter them away

pub fn braces(self, enabled: bool) -> Self

Support braces group in the output of the tokenizer or filter them away

pub fn bracket(self, enabled: bool) -> Self

Support bracket group in the output of the tokenizer or filter them away

pub fn support_bytes(self, supported: bool) -> Self

Support the bytes atom in the output of the tokenizer

Trait Implementations

impl Clone for TokenizerConfig

fn clone(&self) -> TokenizerConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerConfig

fn default() -> Self

Returns the “default value” for a type.