1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
//! The OS boundary: anonymous mapping, unmapping, and returning pages.
//!
//! Three hand-declared `extern "C"` symbols, no `libc` crate — the house
//! rule for OS boundaries. Linux and macOS only; elsewhere every entry
//! point reports failure and the allocator is simply unavailable.
//!
//! # Why not `kevy-madvise`
//!
//! That crate already binds `mmap`/`munmap`/`madvise`, so reusing it was
//! the first choice. It does not fit: it is Linux-only by construction
//! and its contract *is* huge-page advice — every mapping it hands out
//! has `MADV_HUGEPAGE` applied. An allocator needs mappings on macOS too
//! (that is where this is developed), and it must be able to *return*
//! pages, which is the property the whole experiment rests on. Widening
//! a crate whose name is its contract costs more than three extern
//! declarations, so the boundary lives here — which is also why
//! `kevy-alloc` carries its own `unsafe` extern block.
//!
//! # Examples
//!
//! ```
//! use kevy_alloc::os::{PAGE, map_aligned, unmap};
//! if let Some(p) = map_aligned(PAGE, PAGE) {
//! // SAFETY: `p`/PAGE is the mapping just made, and nothing refers into it.
//! unsafe { unmap(p, PAGE) };
//! }
//! ```
use c_void;
use NonNull;
unsafe extern "C"
/// `_SC_PAGESIZE`. Linux says 30, macOS says 29 — the one constant in
/// this file that is not shared, which is itself why it is worth asking
/// the system rather than assuming.
const SC_PAGESIZE: i32 = 30;
const SC_PAGESIZE: i32 = 29;
const PROT_READ: i32 = 0x1;
const PROT_WRITE: i32 = 0x2;
const MAP_PRIVATE: i32 = 0x2;
const MAP_ANONYMOUS: i32 = 0x20;
const MAP_ANONYMOUS: i32 = 0x1000;
/// Discard the contents of a resident range and return the physical
/// pages to the OS, keeping the mapping addressable.
///
/// Linux `MADV_DONTNEED` (4) drops the pages outright: RSS falls and a
/// later touch faults in a zero page. macOS has no equivalent that
/// *guarantees* the drop — `MADV_FREE` (5) marks pages reclaimable and
/// the kernel takes them under pressure, so RSS may not move promptly.
/// The difference is why M4 (reclaim proven directly) is asserted on
/// Linux and reported as informational on macOS rather than being
/// quietly assumed to hold on both.
const MADV_DISCARD: i32 = 4;
const MADV_DISCARD: i32 = 5;
/// The system page size this module assumes for rounding.
///
/// It is a constant because the geometry is: `PAGES_PER_SPAN` is
/// `SPAN_BYTES / PAGE`, and `SpanMeta::discarded` is exactly a `u16` for
/// the sixteen pages that gives. What is NOT safe is assuming the
/// system agrees — see [`page_size_matches`].
///
/// # Examples
///
/// ```
/// use kevy_alloc::class::SPAN_BYTES;
/// use kevy_alloc::os::PAGE;
/// // sixteen pages per span, one bit each in a `u16`
/// assert_eq!(SPAN_BYTES / PAGE, 16);
/// ```
pub const PAGE: usize = 4096;
/// Whether the running system's page size is the one the geometry above
/// was built for.
///
/// This is not pedantry. The development machine for this crate reports
/// 16384 and the bench box reports 4096, and the reclaim path issues
/// `madvise` on ranges computed at 4096 granularity. On a 16 KiB-page
/// system those ranges are not page-aligned; macOS answers 0 anyway and
/// reclaims nothing, and the three callers in `reclaim.rs` discard the
/// return value — so the allocator marks the pages discarded, counts
/// them in `returned`, lowers `predicted_resident()`, and the kernel
/// hands back nothing at all.
///
/// A measuring device that fails in the shape of data. The README's
/// headline "plus returning the pages = 29.3 ns/op" was taken on the
/// 16384 machine, which means it timed a run of `madvise` calls that
/// could not do what the line says they did.
///
/// Answered once and cached: `sysconf` is a call, and this sits under
/// the reclaim tick.
///
/// ```
/// // Stable across calls — it is answered once and cached, and an
/// // answer that flapped would be worse than either value.
/// let a = kevy_alloc::os::page_size_matches();
/// assert_eq!(a, kevy_alloc::os::page_size_matches());
/// ```
/// The memo's three states, as a function of the stored byte.
///
/// Every machine takes exactly one of `1` and `2` forever, so whichever
/// it is not is unreachable code there — a permanently dead region on
/// any single platform's coverage run. Splitting the decode out makes
/// all three answerable from a test anywhere.
const
/// The inverse of [`decode_memo`], kept beside it so the two cannot
/// drift into disagreeing about which byte means what.
const
/// The decision, separated from the syscall that supplies it.
///
/// Every machine gives one answer, so the other branch cannot be
/// executed where it runs — which is how a coverage ratchet ends up
/// holding a permanently dead region, and how the interesting half (a
/// mismatch: the one that makes reclaim inert) stays untested on
/// exactly the machines where it is false. Taking the measurement as an
/// argument makes both answers reachable from a test anywhere.
pub
/// Non-Unix has no reclaim path at all, so nothing can be misreported.
/// Round `n` up to a multiple of `align`, which must be a power of two.
///
/// # Examples
///
/// ```
/// use kevy_alloc::os::{PAGE, round_up};
/// assert_eq!(round_up(1, PAGE), PAGE);
/// assert_eq!(round_up(PAGE, PAGE), PAGE);
/// assert_eq!(round_up(PAGE + 1, PAGE), 2 * PAGE);
/// ```
pub const
/// Map `len` bytes anonymously with the returned address aligned to
/// `align` bytes.
///
/// `align` must be a power of two and a multiple of [`PAGE`]; `len` must
/// be a non-zero multiple of `align`. Over-allocates by one alignment
/// unit and trims both sides, because `mmap` only promises page
/// alignment. Returns `None` on failure — never panics, because an
/// allocator that panics on OOM is worse than one that reports it.
///
/// # Examples
///
/// ```
/// use kevy_alloc::os::{available, map_aligned, unmap};
/// const LEN: usize = 64 * 1024;
/// if let Some(p) = map_aligned(LEN, LEN) {
/// assert_eq!(p.as_ptr() as usize % LEN, 0);
/// // SAFETY: the mapping is fresh and read/write; `LEN` bytes are ours.
/// unsafe { p.as_ptr().write(7) };
/// // SAFETY: `p`/`LEN` is exactly the mapping made above, now unused.
/// unsafe { unmap(p, LEN) };
/// } else {
/// assert!(!available());
/// }
/// // a length that is not a multiple of the alignment is refused
/// assert!(map_aligned(4096 * 3, 8192).is_none());
/// ```
/// Trim an over-allocated mapping down to `len` bytes starting at the
/// first `align`-aligned address inside it, unmapping both offcuts.
/// Unmap `len` bytes at `ptr`.
///
/// # Safety
/// `ptr`/`len` must describe a live mapping produced by [`map_aligned`]
/// (or a whole sub-range of one that is no longer referenced).
///
/// # Examples
///
/// ```
/// use kevy_alloc::os::{map_aligned, unmap};
/// if let Some(p) = map_aligned(8192, 4096) {
/// // SAFETY: `p`/8192 is the whole mapping from `map_aligned`, and
/// // nothing refers into it any more.
/// unsafe { unmap(p, 8192) };
/// }
/// ```
pub unsafe
/// Return the physical pages backing `len` bytes at `ptr` to the OS
/// while keeping the range mapped and addressable.
///
/// The range must be page-aligned and a whole number of pages. Contents
/// are discarded: a later read sees zeroes, which is why only spans with
/// no live slots are ever passed here.
///
/// # Safety
/// `ptr`/`len` must lie inside a live mapping from [`map_aligned`], and
/// no live data may remain in the range.
///
/// # Examples
///
/// ```
/// use kevy_alloc::os::{PAGE, discard, map_aligned, unmap};
/// if let Some(p) = map_aligned(4 * PAGE, PAGE) {
/// // SAFETY: fresh read/write mapping of 4 pages.
/// unsafe { p.as_ptr().write(42) };
/// // SAFETY: the range is inside the mapping and holds no live data.
/// let _returned: bool = unsafe { discard(p, 4 * PAGE) };
/// // the range stays mapped: writing again is fine (contents were
/// // dropped on Linux, and may or may not be on macOS)
/// // SAFETY: still inside the live read/write mapping.
/// unsafe { p.as_ptr().write(1) };
/// // SAFETY: the whole mapping, no longer referenced.
/// unsafe { unmap(p, 4 * PAGE) };
/// }
/// ```
pub unsafe
/// Whether this target can map memory at all. Used by tests and by the
/// heap's construction path to fail fast rather than mysteriously.
///
/// # Examples
///
/// ```
/// use kevy_alloc::os::{available, map_aligned};
/// if !available() {
/// assert!(map_aligned(4096, 4096).is_none());
/// }
/// ```
pub const