cyo_mimalloc/lib.rs
1//! [mimalloc](https://github.com/microsoft/mimalloc) v3 as a Rust global
2//! allocator.
3//!
4//! To use mimalloc for every Rust allocation, declare it as the global
5//! allocator:
6//!
7//! ```rust
8//! use cyo_mimalloc::MiMalloc;
9//!
10//! #[global_allocator]
11//! static GLOBAL: MiMalloc = MiMalloc;
12//! ```
13//!
14//! The crate is `no_std` and does not use `alloc`. Functions that produce text,
15//! such as [`MiMalloc::stats_print`], write it to any [`core::fmt::Write`]. A
16//! `String` works, and so do a fixed buffer and a logger that allocates
17//! nothing.
18//!
19//! The global allocator only covers Rust allocations. A C library linked into
20//! the same program, for example through a `-sys` crate, still calls the
21//! system `malloc`.
22//!
23//! # Build configuration
24//!
25//! This crate has no Cargo features. Cargo merges features across the whole
26//! dependency tree, so any library could switch on a mode that changes the
27//! allocator for the entire program. Instead, you configure mimalloc through
28//! environment variables of the build that produces the final binary or shared
29//! library. Put them in that project's `.cargo/config.toml`, or in the
30//! environment of its Nix derivation or CI job. Cargo does not read the
31//! `.cargo/config.toml` of a dependency. Changing any of the variables rebuilds
32//! mimalloc.
33//!
34//! | Variable | Values | Effect |
35//! |---|---|---|
36//! | `MI_SECURE` | 0 to 4 (default 0) | Secure mode. Higher levels add more protections, such as guard pages, encoded free lists, randomized allocation and double-free detection. Each protection costs some speed. |
37//! | `MI_DEBUG` | 0 to 3 (default 0) | Internal assertions (1), plus consistency checks (2), plus expensive checks (3). |
38//! | `MI_NO_THP` | 0 or 1 (default 0) | 1 compiles out mimalloc's requests for transparent huge pages. To decide at runtime instead, use the `allow_thp` option. |
39//! | `CYO_MIMALLOC_TLS_MODEL` | `initial-exec` (default), `local-dynamic`, `global-dynamic`, `local-exec` | How mimalloc finds its thread-local state. `initial-exec` is the fastest. A shared library that is loaded with `dlopen`, such as a Python extension, needs `local-dynamic`. A build for MSVC checks the value but does not use it. |
40//! | `MI_DEFAULT_<NAME>` | see [Build-time defaults](#build-time-defaults) | The default value of a runtime option. |
41//!
42//! An invalid value stops the build with an error.
43//!
44//! On MSVC, the build compiles mimalloc with the C runtime that the rest of the
45//! build uses. That is `/MT` under `-C target-feature=+crt-static`, and `/MD`
46//! otherwise. A statically linked program therefore needs neither
47//! `vcruntime140.dll` nor `ucrtbase.dll`. No build variable of this crate
48//! selects the runtime, because mixing the two runtimes in one binary is a link
49//! error.
50//!
51//! ```toml
52//! # .cargo/config.toml of the application
53//! [env]
54//! MI_SECURE = "4"
55//! CYO_MIMALLOC_TLS_MODEL = "local-dynamic"
56//! ```
57//!
58//! A library that depends on this crate should leave this configuration to the
59//! application. It should also not declare a `#[global_allocator]` or call
60//! [`MiMalloc::option_set`], because both affect the whole program. For memory
61//! of its own, a library can use a [`heap::Heap`], optionally in an exclusive
62//! arena from [`heap::reserve_os_memory`].
63//!
64//! # Options
65//!
66//! mimalloc has runtime options for things like how quickly unused memory is
67//! returned to the OS and whether huge pages are used. [`mi_option_t`] lists
68//! them all, with their defaults and when mimalloc reads each one. An option
69//! gets its value from, in increasing order of precedence:
70//!
71//! 1. mimalloc's built-in default;
72//! 2. a default chosen when your program is built;
73//! 3. an environment variable when your program runs;
74//! 4. a call from your code.
75//!
76//! An application can therefore ship its own defaults, and whoever runs it can
77//! still change them.
78//!
79//! ## Build-time defaults
80//!
81//! Like the rest of the [build configuration](#build-configuration), every
82//! `MI_DEFAULT_<NAME>` variable in the build environment becomes the default
83//! for the option `<name>`:
84//!
85//! ```toml
86//! [env]
87//! MI_DEFAULT_ALLOW_THP = "0"
88//! MI_DEFAULT_ARENA_EAGER_COMMIT = "1"
89//! MI_DEFAULT_RESERVE_OS_MEMORY = "1048576" # KiB, so 1 GiB
90//! ```
91//!
92//! Only the options whose [`mi_option_t`] entry names a `MI_DEFAULT_*`
93//! variable accept one. The build also accepts
94//! `MI_DEFAULT_PHYSICAL_MEMORY_IN_KIB`, which sets the amount of physical memory
95//! that mimalloc assumes until it has detected the real amount. The build passes
96//! each value to the C compiler unchanged, and sizes are in KiB. The build
97//! ignores any other `MI_DEFAULT_*` variable, with a warning.
98//!
99//! ## Environment variables
100//!
101//! When the program starts, mimalloc reads `MIMALLOC_<NAME>` for each option,
102//! for example `MIMALLOC_PURGE_DELAY=0` or `MIMALLOC_ALLOW_THP=0`. The value
103//! overrides the build-time default. Sizes accept a unit here:
104//! `MIMALLOC_ARENA_RESERVE=4GiB`. Set `MIMALLOC_VERBOSE=1` to print every
105//! option's value at startup.
106//!
107//! ## From code
108//!
109//! [`MiMalloc::option_set`], [`option_enable`](MiMalloc::option_enable) and
110//! [`option_disable`](MiMalloc::option_disable) override every other source,
111//! from the moment you call them:
112//!
113//! ```rust
114//! use cyo_mimalloc::{MiMalloc, mi_option_t};
115//!
116//! MiMalloc::option_set(mi_option_t::mi_option_purge_delay, 100);
117//! MiMalloc::option_enable(mi_option_t::mi_option_purge_decommits);
118//! ```
119//!
120//! mimalloc starts before any Rust code runs, including `main`. The options that
121//! it only reads at startup are fixed by then, and setting them from code does
122//! nothing. [`mi_option_t`] marks them "read at startup". Set them with a
123//! build-time default or an environment variable, or call the function that
124//! does the same at run time, where there is one:
125//!
126//! | Option | In code |
127//! |---|---|
128//! | `reserve_os_memory` | [`heap::reserve_os_memory`] |
129//! | `reserve_huge_os_pages`, `reserve_huge_os_pages_at` | [`heap::reserve_huge_os_pages_interleave`], [`heap::reserve_huge_os_pages_at`] |
130//! | `allow_thp` = 0 | `prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0)`, plus setting the option to 0 |
131//! | `use_numa_nodes`, `max_vabits`, `pagemap_commit`, `max_errors`, `max_warnings` | none |
132//!
133//! An option that mimalloc reads "per thread" applies to the threads that
134//! start after you change it.
135
136#![no_std]
137#![warn(missing_docs)]
138#![warn(clippy::undocumented_unsafe_blocks)]
139#![warn(clippy::missing_errors_doc, clippy::missing_safety_doc)]
140#![cfg_attr(test, allow(clippy::undocumented_unsafe_blocks))]
141
142#[cfg(test)]
143extern crate std;
144
145mod api;
146mod ffi;
147
148pub mod heap;
149
150pub use api::{ProcessInfo, set_current_thread_in_threadpool};
151pub use cyo_mimalloc_sys::mi_option_t;
152
153use core::alloc::{GlobalAlloc, Layout};
154use core::ffi::c_void;
155
156/// The mimalloc global allocator.
157///
158/// Every allocation goes through an aligned mimalloc function, such as
159/// `mi_malloc_aligned`, so the block meets the alignment of its [`Layout`].
160#[derive(Debug, Clone, Copy, Default)]
161pub struct MiMalloc;
162
163// ── GlobalAlloc ─────────────────────────────────────────────────────────────
164
165// SAFETY: each method returns a block from mimalloc that meets the size and
166// alignment of `layout`, or null. mimalloc frees and resizes only blocks that
167// it allocated, and `GlobalAlloc` passes only those.
168unsafe impl GlobalAlloc for MiMalloc {
169 #[inline(always)]
170 unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
171 // SAFETY: `mi_malloc_aligned` has no preconditions. `Layout`
172 // guarantees that the alignment is a power of two.
173 unsafe { cyo_mimalloc_sys::mi_malloc_aligned(layout.size(), layout.align()) as *mut u8 }
174 }
175
176 #[inline(always)]
177 unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
178 // SAFETY: as in `alloc`.
179 unsafe { cyo_mimalloc_sys::mi_zalloc_aligned(layout.size(), layout.align()) as *mut u8 }
180 }
181
182 #[inline(always)]
183 unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) {
184 // SAFETY: the caller of `dealloc` guarantees that `ptr` came from
185 // this allocator and is still allocated.
186 unsafe { cyo_mimalloc_sys::mi_free(ptr as *mut c_void) };
187 }
188
189 #[inline(always)]
190 unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
191 // SAFETY: the caller of `realloc` guarantees that `ptr` came from
192 // this allocator with `layout` and is still allocated.
193 unsafe {
194 cyo_mimalloc_sys::mi_realloc_aligned(ptr as *mut c_void, new_size, layout.align())
195 as *mut u8
196 }
197 }
198}
199
200// ── Tests ───────────────────────────────────────────────────────────────────
201
202#[cfg(test)]
203mod tests {
204 use super::*;
205 use std::alloc::{GlobalAlloc, Layout};
206 use std::boxed::Box;
207 use std::vec::Vec;
208
209 #[global_allocator]
210 static GLOBAL: MiMalloc = MiMalloc;
211
212 #[test]
213 fn alloc_dealloc_roundtrip() {
214 let layout = Layout::from_size_align(64, 8).unwrap();
215 unsafe {
216 let ptr = GLOBAL.alloc(layout);
217 assert!(!ptr.is_null());
218 GLOBAL.dealloc(ptr, layout);
219 }
220 }
221
222 #[test]
223 fn alloc_zeroed_is_zero() {
224 let layout = Layout::from_size_align(256, 16).unwrap();
225 unsafe {
226 let ptr = GLOBAL.alloc_zeroed(layout);
227 assert!(!ptr.is_null());
228 assert!((0..256).all(|i| *ptr.add(i) == 0));
229 GLOBAL.dealloc(ptr, layout);
230 }
231 }
232
233 #[test]
234 fn realloc_preserves_content() {
235 let layout = Layout::from_size_align(64, 8).unwrap();
236 unsafe {
237 let ptr = GLOBAL.alloc(layout);
238 core::ptr::write_bytes(ptr, 0xAB, 64);
239 let new_ptr = GLOBAL.realloc(ptr, layout, 128);
240 assert!(!new_ptr.is_null());
241 assert!((0..64).all(|i| *new_ptr.add(i) == 0xAB));
242 GLOBAL.dealloc(new_ptr, Layout::from_size_align(128, 8).unwrap());
243 }
244 }
245
246 #[test]
247 fn alignment_respected() {
248 for align_pow in 0..=12 {
249 let align = 1usize << align_pow;
250 let layout = Layout::from_size_align(64, align).unwrap();
251 unsafe {
252 let ptr = GLOBAL.alloc(layout);
253 assert!(!ptr.is_null(), "align={align}");
254 assert_eq!(ptr as usize % align, 0, "align={align}");
255 GLOBAL.dealloc(ptr, layout);
256 }
257 }
258 }
259
260 #[test]
261 fn large_alignment_page_size() {
262 // Page-sized alignment has crashed allocators that skip the aligned
263 // mimalloc functions for alignments they assume are already met.
264 let layout = Layout::from_size_align(4096, 4096).unwrap();
265 for _ in 0..100 {
266 unsafe {
267 let ptr = GLOBAL.alloc(layout);
268 assert!(!ptr.is_null());
269 assert_eq!(ptr as usize % 4096, 0);
270 GLOBAL.dealloc(ptr, layout);
271 }
272 }
273 }
274
275 #[test]
276 fn vec_push_smoke() {
277 let v: Vec<i32> = (0..10_000).collect();
278 assert_eq!(v.len(), 10_000);
279 assert_eq!(v[9999], 9999);
280 }
281
282 #[test]
283 fn box_smoke() {
284 let b = Box::new(42u64);
285 assert_eq!(*b, 42);
286 }
287
288 #[test]
289 fn concurrent_alloc_free() {
290 use std::sync::{Arc, Barrier};
291 use std::thread;
292
293 let n = 8;
294 let barrier = Arc::new(Barrier::new(n));
295 let handles: Vec<_> = (0..n)
296 .map(|_| {
297 let b = barrier.clone();
298 thread::spawn(move || {
299 b.wait();
300 let layout = Layout::from_size_align(128, 16).unwrap();
301 for _ in 0..1000 {
302 unsafe {
303 let ptr = GLOBAL.alloc(layout);
304 assert!(!ptr.is_null());
305 core::ptr::write_bytes(ptr, 0xCD, 128);
306 GLOBAL.dealloc(ptr, layout);
307 }
308 }
309 })
310 })
311 .collect();
312 for h in handles {
313 h.join().unwrap();
314 }
315 }
316}