onnxruntime-ep-mlx

Run ONNX models on Apple Silicon GPUs with MLX and ONNX Runtime. This execution provider is a plugin: no ONNX Runtime fork is needed. Supported parts of your model run in MLX; unsupported operators stay on ONNX Runtime's CPU provider.

Install

Requires macOS 14+ on Apple Silicon, Python 3.11+, and ONNX Runtime 1.31+. The wheel bundles the MLX runtime and installs ONNX Runtime as a dependency; no source-build tools are needed.

python -m pip install -U onnxruntime-ep-mlx

Use

Register the plugin once per process, before creating your session:

import onnxruntime as ort
import onnxruntime_ep_mlx

onnxruntime_ep_mlx.register_execution_provider_library()
session = ort.InferenceSession(
    "model.onnx",
    providers=["MLXExecutionProvider", "CPUExecutionProvider"],
)

# Supply your model's input names and NumPy arrays.
feeds = {"input_name": input_array}
outputs = session.run(None, feeds)

Replace model.onnx, input_name, and input_array with your model and inputs. Use session.get_inputs() to inspect the required input names, shapes, and types.

Before using it

• Coverage is conditional. Operator attributes, dtypes, and shapes determine what runs in MLX. See the operator support guide. CPU fallback requires a matching ONNX Runtime CPU kernel.
• Speed depends on the model. Compilation adds startup cost; CPU fallback and small workloads can outweigh GPU gains. Measure your own workload with the benchmark guide.
• Use one session per thread. Do not run a shared session concurrently from multiple threads; MLX evaluation is thread-affine.
• Results are not bit-identical to CPU. Floating-point differences can change long greedy generations when logits are nearly tied.

Documentation

See shared immutable initializers for cross-session weight ownership, lifecycle tests and the diagnostic switch.