pub struct Rationer { /* private fields */ }Expand description
Handle that owns long-lived clustering resources (Metal device, IOPM power assertion, thread QoS
boost). Construct once per process; share &Rationer across rayon workers.
On macOS with feature = "gpu", Rationer::new() lazily acquires a Metal device + power
assertion. On platforms without Metal (or with the gpu feature disabled at compile time) the
struct is essentially empty — methods still work and produce the same answers, just always on CPU.
Implementations§
Source§impl Rationer
impl Rationer
Sourcepub fn builder() -> RationerBuilder
pub fn builder() -> RationerBuilder
Start with sensible defaults: GPU+CPU concurrency, rayon’s global thread pool.
Sourcepub fn new() -> Self
pub fn new() -> Self
Create a Rationer with default settings (Concurrency::GpuPlusCpu, global rayon pool,
delta = 0.0 = exact RO). Never fails — if Metal is unavailable, the handle quietly
degrades to CPU-only.
Equivalent to Rationer::builder().build().
Sourcepub fn concurrency(&self) -> Concurrency
pub fn concurrency(&self) -> Concurrency
The active backend (after construction-time fallback). A Rationer built with
Concurrency::Gpu on a non-Metal platform reports Cpu here.
Sourcepub fn ratio(&self, a: &str, b: &str) -> f64
pub fn ratio(&self, a: &str, b: &str) -> f64
Single-pair Ratcliff–Obershelp ratio. Always CPU (one pair offers no GPU win). Same
output as the free function crate::gestalt_ratio.
Sourcepub fn ratio_many<S1, S2>(&self, pairs: &[(S1, S2)]) -> Vec<f64>
pub fn ratio_many<S1, S2>(&self, pairs: &[(S1, S2)]) -> Vec<f64>
Batched ratio over a list of (a, b) string pairs, in parallel across cores.
CPU by default. Benchmarking (mypy/django/sympy/ha/transformers, 61k–404k pairs) showed
the GPU matching_stats offload loses here — 0.82–0.93× CPU across every size measured —
because the CPU rayon path (intern uniques, prebuild each SAM once, gestalt_ratio_prebuilt
per pair) is already efficient and the GPU’s corpus-build + dispatch overhead isn’t amortized
by the relatively light per-pair matching_stats walk. So ratio_many stays on CPU even
under Concurrency::Gpu. The GPU path is retained behind DFGPU_RATIO_MANY_THRESHOLD=<n>
(default off) for experimentation / other hardware: set it to engage GPU at pairs.len() >= n.
Output is the bit-identical Ratcliff–Obershelp ratio for each pair, in input order. On non-ASCII pairs the GPU path routes the affected pairs to the CPU per-pair fallback.
Sourcepub fn prepare<S: AsRef<str>>(&self, strings: &[S]) -> PreparedRationer<'_>
pub fn prepare<S: AsRef<str>>(&self, strings: &[S]) -> PreparedRationer<'_>
Build a reusable corpus over strings: parses chars, builds SAMs, and (when GPU is
active) uploads the [CorpusGpu] arena. The returned PreparedRationer borrows the
Rationer and lets you issue multiple ratio_many_idx(pairs) calls that amortize the
SAM-build + GPU-upload cost over an arbitrary number of pair queries.
Use this when the same string set is queried repeatedly (e.g. iterative refinement,
dedup pipelines, find-dup-defs over a fixed file list). For one-shot queries the regular
Rationer::ratio_many does the same work internally and is fine.
All strings should be ASCII for the GPU path to engage; non-ASCII strings are kept in the
SAM pool but their pair queries automatically fall back to a CPU per-pair compute on the
host (same semantics as ratio_many).
Sourcepub fn cluster_canonicals_chars(
&self,
chars: &[Vec<char>],
threshold: f64,
) -> Vec<(Vec<usize>, f64)>
pub fn cluster_canonicals_chars( &self, chars: &[Vec<char>], threshold: f64, ) -> Vec<(Vec<usize>, f64)>
Exact single-linkage clustering at threshold, identical to the free-standing
[crate::cluster_canonicals_chars] in behaviour and output. Routes through the GPU on
macOS when the group is large enough to amortize dispatch overhead; small groups stay on
CPU because the GPU dispatch fixed cost (~5–50 ms) exceeds CPU rayon’s full run for them.
Routing: if the handle’s Concurrency includes GPU and the group is big + all-ASCII,
dispatch through Metal; else stay on the CPU. The size cutoff is set so the GPU path’s
CorpusGpu build + dispatch (~10–30 ms) is amortized over enough verified pairs to win.
Override with DFGPU_CLUSTER_THRESHOLD=<n> env var; default 300.
Sourcepub fn cluster_canonicals(
&self,
canonicals: &[String],
threshold: f64,
) -> Vec<(Vec<usize>, f64)>
pub fn cluster_canonicals( &self, canonicals: &[String], threshold: f64, ) -> Vec<(Vec<usize>, f64)>
String-input convenience, mirrors crate::cluster_canonicals.
Sourcepub fn cluster_canonicals_multi(
&self,
groups: &[Vec<String>],
threshold: f64,
) -> Vec<Vec<(Vec<usize>, f64)>>
pub fn cluster_canonicals_multi( &self, groups: &[Vec<String>], threshold: f64, ) -> Vec<Vec<(Vec<usize>, f64)>>
Batched-across-groups cluster_canonicals.
Run K independent clustering jobs as ONE GPU dispatch — concatenate every group’s strings
into a single Metal corpus arena, run filters per-group on the CPU, submit ALL surviving
candidate pairs (from every group) in one batched matching_stats kernel call, then split
results back to per-group gestalt_edge_with_ms + assemble on the CPU.
The single-dispatch idea was meant to be find-dup-defs’s win condition (thousands of
same-name groups, each too small for per-call GPU overhead, batched into one dispatch).
CPU by default, though: benchmarking showed the batched GPU path loses on that very shape
— 0.70× CPU at 44 groups of 50, only reaching break-even (~0.99×) at a handful of large
groups — the corpus-build + dispatch overhead isn’t amortized when each group’s surviving-pair
count is small. So cluster_canonicals_multi runs per-group on CPU (rayon across groups)
unless DFGPU_MULTI_THRESHOLD=<total strings> is set to opt the batched GPU path back in.
Returns one cluster list per input group in the input order. Each list has identical
semantics to [cluster_canonicals] called on that group alone.
Trait Implementations§
Auto Trait Implementations§
impl Freeze for Rationer
impl !RefUnwindSafe for Rationer
impl Send for Rationer
impl Sync for Rationer
impl Unpin for Rationer
impl UnsafeUnpin for Rationer
impl !UnwindSafe for Rationer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more