Trait tokenizers::utils::parallelism::MaybeParallelIterator
source · [−]pub trait MaybeParallelIterator<P, S> where
P: ParallelIterator,
S: Iterator<Item = P::Item>, {
fn into_maybe_par_iter(self) -> CondIterator<P, S>;
fn into_maybe_par_iter_cond(self, cond: bool) -> CondIterator<P, S>;
}
Expand description
Allows to convert into an iterator that can be executed either parallelly or serially.
The choice is made according to the currently set TOKENIZERS_PARALLELISM
environment variable.
This variable can have one of the following values
- False => “” (empty value), “false”, “f”, “off”, “no”, “n”, “0”
- True => Any other value
Required methods
fn into_maybe_par_iter(self) -> CondIterator<P, S>
fn into_maybe_par_iter(self) -> CondIterator<P, S>
Convert ourself in a CondIterator, that will be executed either in parallel or serially,
based solely on the TOKENIZERS_PARALLELISM
environment variable
fn into_maybe_par_iter_cond(self, cond: bool) -> CondIterator<P, S>
fn into_maybe_par_iter_cond(self, cond: bool) -> CondIterator<P, S>
Convert ourself in a CondIterator, that will be executed either in parallel or serially,
based on both the TOKENIZERS_PARALLELISM
environment variable and the provided bool.
Both must be true to run with parallelism activated.