Module tokenizers::utils::parallelism

This module defines helpers to allow optional Rayon usage.

Constants

ENV_VARIABLE

Traits

MaybeParallelBridge

Converts any serial iterator into a CondIterator, that can either run parallelly or serially.

MaybeParallelIterator

Allows to convert into an iterator that can be executed either parallelly or serially.

MaybeParallelRefIterator

Shared reference version of MaybeParallelIterator, works the same but returns an iterator over references, does not consume self

MaybeParallelRefMutIterator

Exclusive reference version of MaybeParallelIterator, works the same but returns an iterator over mutable references, does not consume self

Functions

get_parallelism

Get the currently set value for TOKENIZERS_PARALLELISM env variable

has_parallelism_been_used

Check if at some point we used a parallel iterator

is_parallelism_configured

Check if the TOKENIZERS_PARALLELISM env variable has been explicitly set

set_parallelism

Set the value for TOKENIZERS_PARALLELISM for the current process