Struct BatchGrammarMatcher 

pub struct BatchGrammarMatcher { /* private fields */ }

A batch version of GrammarMatcher that can fill the next token bitmask for multiple matchers in parallel. It utilizes multiple threads to speed up the computation. It is especially useful when the batch size is large.

Implementations

impl BatchGrammarMatcher

pub fn new(max_threads: i32) -> Self

Construct the batch grammar matcher.

Parameters

• max_threads: The maximum number of threads to use for parallel processing. Use -1 for automatic thread count (hardware_concurrency / 2).

pub fn new_auto() -> Self

Create a batch grammar matcher with automatic thread count.

pub fn batch_fill_next_token_bitmask( &mut self, matchers: &[GrammarMatcher], bitmask: &mut DLTensor, indices: Option<&[i32]>, debug_print: bool, )

Fill the next token bitmask for multiple matchers.

Parameters

• matchers: The list of matchers to fill the bitmask for.
• bitmask: Must be a 2-dimensional int32 tensor with shape (bitmask_batch_size, bitmask_size). Bitmask_batch_size could be larger than the actual batch size to allow padding. Bitmask_size equals to ceil(vocab_size/32).
• indices: A list of indices to specify which rows in the bitmask to fill. If None, fill the bitmask [0..matchers.len()).
• debug_print: Whether to print information about generated bitmask (default: false).

pub fn batch_accept_token( matchers: &[GrammarMatcher], tokens: &[i32], debug_print: bool, ) -> Box<[bool]>

Accept a batch of tokens for multiple matchers.

Parameters

• matchers: The list of matchers to accept tokens for.
• tokens: The list of tokens to accept.
• debug_print: Whether to print information about generated bitmask (default: false).

Returns

A boxed slice of booleans indicating whether each token was accepted by its corresponding matcher.

pub fn batch_accept_string( matchers: &[GrammarMatcher], strings: &[impl AsRef<str>], debug_print: bool, ) -> Box<[bool]>

Accept a batch of strings for multiple matchers.

Parameters

• matchers: The list of matchers to accept tokens for.
• strings: The list of strings to accept.
• debug_print: Whether to print information about generated bitmask (default: false).

Returns

A boxed slice of booleans indicating whether each string was accepted by its corresponding matcher.