Module tokenizers::utils::parallelism

This module defines helpers to allow optional Rayon usage.

Constants

• ENV_VARIABLE

Traits

• MaybeParallelBridge
  Converts any serial iterator into a CondIterator, that can either run parallelly or serially.
• MaybeParallelIterator
  Allows to convert into an iterator that can be executed either parallelly or serially.
• MaybeParallelRefIterator
  Shared reference version of MaybeParallelIterator, works the same but returns an iterator over references, does not consume self
• MaybeParallelRefMutIterator
  Exclusive reference version of MaybeParallelIterator, works the same but returns an iterator over mutable references, does not consume self
• MaybeParallelSlice
  Allows to convert into chunks that can be executed either parallelly or serially.

Functions

• current_num_threads
  Returns the number of threads in the current registry. If this code is executing within a Rayon thread-pool, then this will be the number of threads for the thread-pool of the current thread. Otherwise, it will be the number of threads for the global thread-pool.
• get_parallelism
  Get the currently set value for TOKENIZERS_PARALLELISM env variable
• has_parallelism_been_used
  Check if at some point we used a parallel iterator
• is_parallelism_configured
  Check if the TOKENIZERS_PARALLELISM env variable has been explicitly set
• set_parallelism
  Set the value for TOKENIZERS_PARALLELISM for the current process