Struct EmbedderConfig 

pub struct EmbedderConfig {
    pub model_dir: PathBuf,
    pub runtime_path: PathBuf,
    pub provider: ExecutionProvider,
    pub truncate_dim: usize,
    pub max_tokens: usize,
    pub max_batch: usize,
    pub intra_threads: usize,
    pub fallback_model_dir: Option<PathBuf>,
    pub coreml_cache_dir: Option<PathBuf>,
    pub parity_cache_dir: Option<PathBuf>,
    pub prewarm: Option<bool>,
    pub coreml_units: Option<CoreMlUnits>,
    /* private fields */
}

Fields

model_dir: PathBuf

runtime_path: PathBuf

provider: ExecutionProvider

truncate_dim: usize

max_tokens: usize

max_batch: usize

intra_threads: usize

fallback_model_dir: Option<PathBuf>

A Q4 bundle directory served on CPU when an E40 bundle in model_dir cannot be served on its accelerator under Auto (see Self::with_fallback_model_dir).

coreml_cache_dir: Option<PathBuf>

Where a Core ML bundle’s compiled model is cached (see Self::with_coreml_cache_dir).

parity_cache_dir: Option<PathBuf>

Where an E40 bundle’s parity verdict is cached (see Self::with_parity_cache_dir).

prewarm: Option<bool>

Build and run every kept E40 graph or Core ML function at load (see Self::with_prewarm). None is the backend default: on for Core ML, off for E40.

coreml_units: Option<CoreMlUnits>

Core ML compute units per length; None is CoreMlUnits::per_length_default (see Self::with_coreml_units).

Implementations

impl EmbedderConfig

pub fn new( model_dir: impl Into<PathBuf>, runtime_path: impl Into<PathBuf>, ) -> Self

pub fn with_prewarm(self, prewarm: bool) -> Self

Pre-warm an E40 or Core ML bundle served on an accelerator. The default is on for Core ML and off for E40: measured on Windows, E40 pre-warm added 6-18 s to a 2 s cached load, while a lazily built graph costs about 1-2 s on its first query. After the parity check passes (or its cached verdict is used) every graph that is kept has its session built and one dummy run executed during load, so no query pays a graph build. The graphs the parity check built are kept instead of dropped. Costs load time (recorded in LoadTimings::prewarm) and keeps all five sessions or functions resident; false restores the earlier behaviour: graphs are dropped after the check and built on first use. For Core ML the first call of a function on cpu_and_gpu was measured at up to 51 s cold (shader build), so with false the first query of that length pays it. Ignored for CPU and Q4.

pub fn with_coreml_units(self, units: impl Into<CoreMlUnits>) -> Self

Core ML compute units of each function (macOS, Core ML bundle). The default is CoreMlUnits::per_length_default: cpu_and_ne up to 128 tokens, cpu_and_gpu for 256 and 512. Pass a crate::CoreMlComputeUnit to force one unit for every length (all, cpu_and_ne, cpu_and_gpu or cpu_only), or a CoreMlUnits (for example parsed from "256=cpu_only") for a custom map. Applies when the provider is Auto or CoreMl; Cpu always runs cpu_only. A map that is cpu_only everywhere is served as "cpu" with no parity check. Ignored for every other bundle.

pub fn with_fallback_model_dir(self, dir: impl Into<PathBuf>) -> Self

Q4 bundle directory (EmbeddingGemma 2 Q4 or 300M Q4) to serve on CPU instead when model_dir holds an E40 or Core ML bundle and Auto cannot admit its accelerator: the probe or the vector parity check failed, or the CPU reference could not be built (for Core ML also: macOS older than 15, or the package would not compile). Without it Auto serves the E40 graphs on CPU (Core ML: the same package on cpuOnly). Never used for an explicit DirectMl / CoreMl, which fail instead, and ignored when model_dir is not an E40 or Core ML bundle. The embedder then reports the Q4 model’s model_id() and pipeline_id() (vectors differ, so consumers reindex) and says why in fallback_reason().

pub fn with_coreml_cache_dir(self, dir: impl Into<PathBuf>) -> Self

Directory for the compiled Core ML model of a Core ML bundle (macOS). Default <model_dir>/.coreml-compiled. The compile is keyed on the package file hashes and the macOS version, reused when present, and written atomically; stale entries of older keys are not removed. Ignored for every other bundle.

pub fn with_parity_cache_dir(self, dir: impl Into<PathBuf>) -> Self

Opt-in directory for the E40 load-time parity verdict. There is no default location and nothing is written without it. The verdict (lowest cosine, pass or fail, timestamp) is stored as one small JSON file per model directory, keyed on the bundle files’ names, sizes and modification times (not their content hash, which would cost about as much as the check), the ONNX Runtime library (path, size, mtime, the sibling DirectML.dll and ORT’s build string) and the provider name. ORT exposes no cheap GPU adapter or driver identity, and neither does std (the DXGI or registry route needs a new dependency or a child process), so the key does not see a GPU or driver change. A stored verdict, pass or fail, is therefore trusted for 7 days only; an older one is a miss and the check re-runs.

A stored pass skips the CPU reference sessions: the load builds no CPU graph. A stored failure still serves CPU (Auto) or returns AcceleratorParity (explicit DirectMl). A missing, corrupt or non-matching file runs the check and rewrites the file. A failure caused by an inference error is not stored. Delete the file to force a re-check. Applies to E40 bundles only; ignored for every other bundle.

pub fn with_provider(self, provider: impl Into<ExecutionProvider>) -> Self

pub fn with_truncate_dim(self, dim: usize) -> Self

pub fn with_max_tokens(self, max_tokens: usize) -> Self

pub fn with_max_batch(self, max_batch: usize) -> Self

Rows per ONNX call for the batch methods (default 16, minimum 1).

pub fn with_intra_threads(self, threads: usize) -> Self

Intra-op threads for the session (default 1, minimum 1).

Trait Implementations

impl Clone for EmbedderConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EmbedderConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.