Struct TokenizeOptions 

#[non_exhaustive]
pub struct TokenizeOptions {
    pub time_limit_millis: u64,
    pub max_line_length: usize,
    pub include_token_type: bool,
    pub include_scopes: bool,
    pub preserve_scope_boundaries: bool,
}

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

time_limit_millis: u64

Best-effort per-line budget in milliseconds. Ferriki checks it between tokenizer matching passes and after each synchronous pass returns; a regex search cannot be interrupted and may run past the budget. 0 disables the time limit.

max_line_length: usize

include_token_type: bool

include_scopes: bool

preserve_scope_boundaries: bool

Retains grammar boundaries even when adjacent tokens have equal styles.

Implementations

impl TokenizeOptions

pub fn with_time_limit_millis(self, value: u64) -> Self

Sets time_limit_millis.

pub fn with_max_line_length(self, value: usize) -> Self

Sets max_line_length.

pub fn with_include_token_type(self, value: bool) -> Self

Sets include_token_type.

pub fn with_preserve_scope_boundaries(self, value: bool) -> Self

Retains full scope paths and theme-independent grammar boundaries.

pub fn with_include_scopes(self, value: bool) -> Self

Sets include_scopes.

Trait Implementations

impl Clone for TokenizeOptions

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizeOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizeOptions

fn default() -> Self

Returns the “default value” for a type.

impl Eq for TokenizeOptions

impl PartialEq for TokenizeOptions

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenizeOptions