Module parallelism

This module defines helpers to allow optional Rayon usage.

Constants

ENV_VARIABLE

Traits

MaybeParallelBridge

Converts any serial iterator into a CondIterator, that can either run parallelly or serially.

MaybeParallelIterator

Allows to convert into an iterator that can be executed either parallelly or serially.

MaybeParallelRefIterator

Shared reference version of MaybeParallelIterator, works the same but returns an iterator over references, does not consume self

MaybeParallelRefMutIterator

Exclusive reference version of MaybeParallelIterator, works the same but returns an iterator over mutable references, does not consume self

MaybeParallelSlice

Allows to convert into chunks that can be executed either parallelly or serially.

Functions

current_num_threads

Returns the number of threads in the current registry. If this code is executing within a Rayon thread-pool, then this will be the number of threads for the thread-pool of the current thread. Otherwise, it will be the number of threads for the global thread-pool.

get_parallelism

has_parallelism_been_used

Check if at some point we used a parallel iterator

is_parallelism_configured

Check if the TOKENIZERS_PARALLELISM env variable has been explicitly set

set_parallelism

Set the value for TOKENIZERS_PARALLELISM for the current process