rusty_rtos_port_host/lib.rs
1//! The Kairos **host** port: a task gets a real stack from the operating
2//! system.
3//!
4//! # Why this exists
5//!
6//! The Kairos kernel is stackless by design. A blocking call keeps its
7//! locals in the TCB and answers `Wait::Blocked`, meaning "call again when
8//! this task next runs" — which is what lets one kernel run the
9//! conformance corpus on a host, on a simulator and on three different
10//! chips without a line of assembly in between.
11//!
12//! A task written in **C** cannot work that way. `vTaskDelay` has to return
13//! *later*, on the line after, with its locals intact, and locals in C live
14//! on a stack. So the C ABI needs a port that supplies one — and until this
15//! crate there were only two kinds: three bare-metal ports, which need an
16//! emulator or a chip, and the sim port, which is stackless. That is why
17//! K6's kill test ran on QEMU Cortex-M3 before it ran on a host: there was
18//! nowhere on a host for a C task to keep its locals.
19//!
20//! # The shape is FreeRTOS's own
21//!
22//! `portable/ThirdParty/GCC/Posix` runs one pthread per task and lets
23//! exactly one of them run at a time; the MSVC-MinGW port does the same
24//! with Win32 threads. This is that: one OS thread per task, a single
25//! **run permit**, and a tick thread standing in for the timer interrupt.
26//! The OS supplies the stacks and switches them; we supply the policy.
27//!
28//! # The two ways a switch happens
29//!
30//! | | who decides | how the outgoing task stops |
31//! |---|---|---|
32//! | voluntary | the task, through [`pend_switch`] | it waits on its own permit |
33//! | preemptive | the tick thread | the OS freezes the thread where it stands |
34//!
35//! The voluntary path is all a cooperative kernel needs. The preemptive
36//! path is not optional: `integer.c` and `flop.c` in the standard demos
37//! **never block and never yield** — with `configUSE_PREEMPTION` set they
38//! rely on being taken off the CPU — so a host port without it starves
39//! every task below them and their neighbours' checkers report failure. A
40//! cooperative host port would pass the demos that block and quietly fail
41//! the ones that do not, which is the worst of both.
42//!
43//! # Freezing a thread is safe here, and the reason is narrow
44//!
45//! Suspending an arbitrary thread is a famous way to deadlock: freeze it
46//! inside `malloc` and the next thread that allocates waits forever.
47//!
48//! It is safe here because of the ORDER in [`Ticker::interrupt`]: the tick
49//! thread takes the critical-section lock **first**, and every kernel call
50//! and every heap call is inside that lock. A thread that is not holding it
51//! is not inside the kernel and not inside our heap, so there is no lock
52//! for it to be holding when it freezes.
53//!
54//! What that argument does NOT cover is a lock we do not own: the C
55//! runtime's allocator, or `stdout`. A task that calls `printf` or the
56//! system `malloc` directly can still be frozen inside one. The demo task
57//! files do neither — they allocate through `pvPortMalloc`, which is the
58//! Kairos heap behind our own critical section. A cell that prints from a
59//! task must do it inside [`Port::enter_critical`].
60//!
61//! # Unsafe
62//!
63//! The workspace denies `unsafe_code`. Every use here is fenced with
64//! `#[expect(unsafe_code)]` and a justification, and they are all of one
65//! kind: freezing and thawing an OS thread. See `UNSAFE.md`.
66
67use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicUsize, Ordering};
68use std::sync::{Condvar, Mutex, OnceLock};
69use std::time::Duration;
70
71use rusty_rtos_core::isr::Woken;
72use rusty_rtos_core::port::Port;
73
74/// Crate version, for manifests and logs.
75pub const VERSION: &str = env!("CARGO_PKG_VERSION");
76
77/// The most tasks a host cell may have.
78///
79/// Fixed rather than grown, so the slot table is a `static` and a slot's
80/// address never moves — a task thread holds a reference to its own slot
81/// for its whole life.
82pub const MAX_TASKS: usize = 128;
83
84// ------------------------------------------------------ the run permit --
85
86/// One task's slot: its permit, and how it is currently stopped.
87///
88/// A task runs only when it holds the single run permit. It can be stopped
89/// in two different ways and the resume path has to know which, because
90/// thawing a thread that is waiting on a condvar does nothing and
91/// signalling a condvar that a frozen thread is not waiting on does nothing
92/// either. Getting this wrong is a hang, not an error.
93#[derive(Debug)]
94struct Slot {
95 /// Set when this task may run. The task's own thread waits for it.
96 granted: Mutex<bool>,
97 /// Signalled when `granted` becomes true.
98 wake: Condvar,
99 /// The OS thread, once it exists, in whatever form the platform needs
100 /// to freeze it. Zero means "no thread yet".
101 os_thread: AtomicU64,
102 /// True while this task's thread is frozen by the tick thread rather
103 /// than parked on its own condvar.
104 frozen: AtomicBool,
105 /// Which occupant of this slot the thread belongs to.
106 ///
107 /// A slot outlives its tasks -- `death.c` deletes and recreates them
108 /// forever and the kernel hands the freed index straight back -- and a
109 /// deleted task's thread is parked inside the C function it was
110 /// running, from which it will never return. So the slot takes a NEW
111 /// thread and this counter is how the old one knows it is no longer
112 /// the occupant: it wakes with the others, sees a generation that is
113 /// not its own, and parks again forever.
114 ///
115 /// Without it the orphan competes for the run permit with the task
116 /// that replaced it, and both run.
117 generation: AtomicU64,
118}
119
120impl Slot {
121 const fn new() -> Self {
122 Self {
123 granted: Mutex::new(false),
124 wake: Condvar::new(),
125 os_thread: AtomicU64::new(0),
126 frozen: AtomicBool::new(false),
127 generation: AtomicU64::new(0),
128 }
129 }
130}
131
132/// The slot table. One entry per task index, and the index is the
133/// kernel's `TaskHandle::index()`, exactly as on the Cortex-M port.
134fn slots() -> &'static [Slot] {
135 static SLOTS: OnceLock<Vec<Slot>> = OnceLock::new();
136 SLOTS.get_or_init(|| (0..MAX_TASKS).map(|_| Slot::new()).collect())
137}
138
139/// Which task index currently holds the run permit.
140///
141/// This is the host's `pxCurrentTCB`. `usize::MAX` means "nothing is
142/// running yet", the state before the first task starts.
143pub static CURRENT: AtomicUsize = AtomicUsize::new(usize::MAX);
144
145/// Nobody is running.
146pub const NO_TASK: usize = usize::MAX;
147
148// ---------------------------------------------------- the critical lock --
149
150/// The critical section, as a lock the tick thread also takes.
151///
152/// On a chip `cpsid i` masks the timer interrupt. Here there is no
153/// interrupt to mask: the "interrupt" is another OS thread, and the only
154/// way to hold it off is a lock it agrees to take. That agreement is the
155/// whole of this port's interrupt model, and [`Ticker::interrupt`] is the
156/// one place the other side of it lives.
157///
158/// It is a spin lock rather than a `Mutex` because it must be re-entrant
159/// (the kernel nests critical sections freely) and must be releasable from
160/// a different call frame than the one that took it, neither of which a
161/// `MutexGuard` will do.
162#[derive(Debug)]
163struct CriticalLock {
164 /// The thread that holds it, as its OS id. Zero is free.
165 owner: AtomicU64,
166 /// How many times that thread has entered without leaving.
167 nesting: AtomicU32,
168}
169
170static CRITICAL: CriticalLock = CriticalLock {
171 owner: AtomicU64::new(0),
172 nesting: AtomicU32::new(0),
173};
174
175impl CriticalLock {
176 /// Take the lock, or count another level if this thread already has it.
177 fn enter(&self, me: u64) {
178 if self.owner.load(Ordering::Acquire) == me {
179 self.nesting.fetch_add(1, Ordering::Relaxed);
180 return;
181 }
182 while self
183 .owner
184 .compare_exchange_weak(0, me, Ordering::AcqRel, Ordering::Relaxed)
185 .is_err()
186 {
187 // The holder is either running on another core or about to be
188 // scheduled. Either way there is nothing useful to do here.
189 std::thread::yield_now();
190 }
191 self.nesting.store(1, Ordering::Relaxed);
192 }
193
194 /// Give up one level, releasing the lock at zero. `true` when it was
195 /// actually released.
196 fn exit(&self, me: u64) -> bool {
197 if self.owner.load(Ordering::Acquire) != me {
198 // Not ours to release. A port that "restored" a lock it never
199 // took would let the tick thread in halfway through somebody
200 // else's critical section.
201 return false;
202 }
203 let left = self.nesting.load(Ordering::Relaxed).saturating_sub(1);
204 self.nesting.store(left, Ordering::Relaxed);
205 if left == 0 {
206 self.owner.store(0, Ordering::Release);
207 return true;
208 }
209 false
210 }
211
212 /// Give the lock up entirely, whatever depth this thread was at, and
213 /// answer that depth.
214 ///
215 /// A critical section belongs to the TASK, not to the CPU. FreeRTOS's
216 /// own Win32 port saves and restores `uxCriticalNesting` across a
217 /// switch for exactly this reason: the kernel yields from inside its
218 /// own critical sections, and a task that parked while still holding
219 /// this lock would deadlock the task it just handed the CPU to.
220 fn release_all(&self, me: u64) -> u32 {
221 if self.owner.load(Ordering::Acquire) != me {
222 return 0;
223 }
224 let depth = self.nesting.load(Ordering::Relaxed);
225 self.nesting.store(0, Ordering::Relaxed);
226 self.owner.store(0, Ordering::Release);
227 depth
228 }
229
230 /// Take it again at the depth [`release_all`](Self::release_all) gave
231 /// back. Nothing to do for a depth of zero.
232 fn reacquire(&self, me: u64, depth: u32) {
233 if depth == 0 {
234 return;
235 }
236 self.enter(me);
237 self.nesting.store(depth, Ordering::Relaxed);
238 }
239}
240
241// ------------------------------------------------------ the OS backend --
242
243#[cfg(windows)]
244mod backend {
245 //! Windows: `SuspendThread` / `ResumeThread` freeze a thread where it
246 //! stands, which is what makes preemption possible at all.
247
248 use windows_sys::Win32::Foundation::{CloseHandle, HANDLE};
249 use windows_sys::Win32::System::Threading::{ResumeThread, SuspendThread};
250
251 /// Is preemption available on this platform?
252 pub const PREEMPTIVE: bool = true;
253
254 /// The OS handle a `JoinHandle` owns.
255 ///
256 /// Taken from the spawner so the thread has a handle before it has
257 /// run. Borrowed, not owned: the `JoinHandle` must outlive every use,
258 /// which is why the caller keeps it.
259 #[must_use]
260 pub fn thread_handle_of(join: &std::thread::JoinHandle<()>) -> u64 {
261 use std::os::windows::io::AsRawHandle as _;
262 join.as_raw_handle() as u64
263 }
264
265 /// Freeze the thread. `false` if it could not be.
266 #[expect(unsafe_code, reason = "suspending a thread is a syscall")]
267 pub fn freeze(handle: u64) -> bool {
268 if handle == 0 {
269 return false;
270 }
271 // SAFETY: `handle` came from `current_thread` above and is a real
272 // thread handle this process owns.
273 let previous = unsafe { SuspendThread(handle as HANDLE) };
274 previous != u32::MAX
275 }
276
277 /// Thaw the thread. `false` if it could not be.
278 #[expect(unsafe_code, reason = "resuming a thread is a syscall")]
279 pub fn thaw(handle: u64) -> bool {
280 if handle == 0 {
281 return false;
282 }
283 // SAFETY: as `freeze`.
284 let previous = unsafe { ResumeThread(handle as HANDLE) };
285 previous != u32::MAX
286 }
287
288 /// Give the handle back to the OS.
289 #[expect(unsafe_code, reason = "closing a handle is a syscall")]
290 pub fn release(handle: u64) {
291 if handle != 0 {
292 // SAFETY: as `freeze`; the handle is not used again.
293 unsafe { CloseHandle(handle as HANDLE) };
294 }
295 }
296}
297
298#[cfg(unix)]
299mod backend {
300 //! Unix: a thread cannot be frozen from outside, so it freezes
301 //! **itself**.
302 //!
303 //! There is no `SuspendThread` here. What Unix gives instead is a
304 //! signal, and a signal handler is the only code that runs in another
305 //! thread's context without that thread's cooperation — which is
306 //! exactly what preemption means. So `freeze` sends `SIGUSR1` and the
307 //! handler, running in the target, parks the target in `sigsuspend`
308 //! until `SIGUSR2` arrives. It is the shape FreeRTOS's own
309 //! `portable/ThirdParty/GCC/Posix` port uses.
310 //!
311 //! # Freezing has to be SYNCHRONOUS, and that is the whole difficulty
312 //!
313 //! `pthread_kill` returns as soon as the signal is *queued*, not when
314 //! it is handled. If `freeze` returned there, the tick thread would
315 //! grant the CPU to a new task while the old one was still running on
316 //! it — two tasks live at once, which is the one thing a single run
317 //! permit exists to prevent. So `freeze` waits until it can observe
318 //! the target parked, and answers `false` if it never does, in which
319 //! case the caller declines the switch rather than taking it unsafely.
320 //!
321 //! # The lost-wakeup race, and why `SIGUSR2` is blocked in the handler
322 //!
323 //! The handler marks itself `PARKED` and then calls `sigsuspend`. A
324 //! `thaw` landing in the gap between those two would be delivered to a
325 //! thread that is not yet waiting, and the thread would then sleep for
326 //! ever on a wake-up that already happened.
327 //!
328 //! `SIGUSR2` is therefore in the suspend handler's `sa_mask`, so it is
329 //! BLOCKED for the whole handler. A `thaw` arriving in that gap stays
330 //! *pending*; `sigsuspend` atomically unblocks it, which delivers it
331 //! immediately and returns. Nothing is lost, and the loop re-checks
332 //! the state rather than trusting the wake-up — a signal handler that
333 //! trusts a single wake-up is a signal handler with a bug.
334 //!
335 //! # What runs inside a handler
336 //!
337 //! Only async-signal-safe calls: `pthread_self`, `sigfillset`,
338 //! `sigdelset`, `sigsuspend`, and atomic loads over a fixed static
339 //! table. No allocation, no locks, no thread-locals — a Rust
340 //! thread-local can allocate on first touch, which is not safe here.
341
342 use std::sync::Once;
343 use std::sync::atomic::{AtomicU32, AtomicUsize, Ordering};
344
345 /// Is preemption available on this platform?
346 pub const PREEMPTIVE: bool = true;
347
348 /// How many task threads can be registered at once.
349 ///
350 /// An entry is freed by [`release`], which the port calls when a task
351 /// slot is reused or a task is deleted, so this bounds LIVE threads
352 /// and not threads over the run. The C ABI cell runs 60.
353 const MAX_THREADS: usize = 256;
354
355 /// Neither asked to park nor parked.
356 const RUNNING: u32 = 0;
357 /// Signalled, but not yet seen to be parked.
358 const ASKED: u32 = 1;
359 /// Sitting in `sigsuspend`, inside the handler.
360 const PARKED: u32 = 2;
361
362 /// How many times [`freeze`] looks for the target to park before it
363 /// gives up and answers `false`.
364 ///
365 /// This is a bound on a handoff between two runnable threads, not a
366 /// wait for work, so it is generous rather than tuned: the cost of
367 /// being wrong in one direction is a declined switch, and in the other
368 /// it is two tasks running at once.
369 const PARK_SPINS: u32 = 200_000;
370
371 struct Entry {
372 /// The `pthread_t`, widened to a `usize`.
373 ///
374 /// Zero means the entry is free. No Unix gives out a null
375 /// `pthread_t` — on Linux it is a pointer to the thread's own
376 /// descriptor — so zero is safe to spend as the empty marker.
377 tid: AtomicUsize,
378 /// One of [`RUNNING`], [`ASKED`], [`PARKED`].
379 state: AtomicU32,
380 }
381
382 static TABLE: [Entry; MAX_THREADS] = [const {
383 Entry {
384 tid: AtomicUsize::new(0),
385 state: AtomicU32::new(RUNNING),
386 }
387 }; MAX_THREADS];
388
389 /// The entry a handle names, or `None`.
390 ///
391 /// The handle is the table index plus one, so that a handle of zero
392 /// keeps meaning "no thread" the way the Windows backend's does.
393 fn entry_at(handle: u64) -> Option<&'static Entry> {
394 let index = usize::try_from(handle).ok()?.checked_sub(1)?;
395 TABLE.get(index)
396 }
397
398 /// The entry belonging to a `pthread_t`, or `None`.
399 ///
400 /// A linear scan of atomic loads, which is what makes it callable from
401 /// a signal handler. 256 relaxed loads is nothing next to the context
402 /// switch that is about to happen.
403 fn entry_of(tid: usize) -> Option<&'static Entry> {
404 TABLE.iter().find(|e| e.tid.load(Ordering::Acquire) == tid)
405 }
406
407 /// A `pthread_t` as the table's `usize`. It is `usize` on Apple and `u64`
408 /// on Linux, so the cast is needed on one and a no-op on the other;
409 /// clippy's `unnecessary_cast` sees only the platform it is run on.
410 #[allow(
411 clippy::unnecessary_cast,
412 reason = "pthread_t is usize on Apple, u64 on Linux"
413 )]
414 fn tid_of(t: libc::pthread_t) -> usize {
415 t as usize
416 }
417
418 /// Park this thread until someone thaws it. `SIGUSR1`.
419 #[expect(
420 unsafe_code,
421 reason = "a signal handler is the only code that runs in another thread without its cooperation"
422 )]
423 extern "C" fn on_suspend(_signal: core::ffi::c_int) {
424 // First, before anything can touch it: `sigsuspend` below always
425 // sets `errno` (to `EINTR`), and the code this handler interrupted
426 // may be between a failing syscall and reading its `errno`. Put back
427 // on every return by the guard's `Drop`.
428 let _errno = SavedErrno::take();
429 // SAFETY: `pthread_self` reads the calling thread's own id and is
430 // async-signal-safe.
431 let me = tid_of(unsafe { libc::pthread_self() });
432 let Some(entry) = entry_of(me) else {
433 // Not a task thread, or its entry was released while the
434 // signal was in flight. Either way there is nobody to wait
435 // for a thaw that will never come.
436 return;
437 };
438 if entry
439 .state
440 .compare_exchange(ASKED, PARKED, Ordering::AcqRel, Ordering::Acquire)
441 .is_err()
442 {
443 // `freeze` gave up before this handler ran and put the state
444 // back. Parking now would be parking with nobody left to thaw.
445 return;
446 }
447
448 // Everything blocked except the thaw signal, so `sigsuspend` can
449 // only be woken by the thing that is meant to wake it.
450 //
451 // SAFETY: `mask` is a real `sigset_t` about to be initialised by
452 // `sigfillset`; all four calls are async-signal-safe.
453 unsafe {
454 let mut mask: libc::sigset_t = core::mem::zeroed();
455 libc::sigfillset(&raw mut mask);
456 libc::sigdelset(&raw mut mask, libc::SIGUSR2);
457 while entry.state.load(Ordering::Acquire) == PARKED {
458 libc::sigsuspend(&raw const mask);
459 }
460 }
461 }
462
463 /// `errno` as the interrupted code left it, restored when the signal
464 /// handler returns.
465 ///
466 /// A signal handler that calls anything able to set `errno` must put it
467 /// back, or the code it interrupted can read a value it did not cause --
468 /// a failed `read` reporting `EINTR`, or a success reporting an error.
469 /// `on_suspend` calls `sigsuspend`, which always sets it. ThreadSanitizer
470 /// reported exactly this ("signal handler spoils errno") on the host
471 /// port's tests during the hardening audit of 2026-10-01.
472 struct SavedErrno(Option<core::ffi::c_int>);
473
474 impl SavedErrno {
475 #[expect(
476 unsafe_code,
477 reason = "reading errno through the C runtime's per-thread pointer"
478 )]
479 fn take() -> Self {
480 // SAFETY: the pointer is the calling thread's own errno slot,
481 // valid for the life of the thread; reading it is what `errno`
482 // means, and is async-signal-safe.
483 Self(errno_location().map(|p| unsafe { *p }))
484 }
485 }
486
487 impl Drop for SavedErrno {
488 #[expect(
489 unsafe_code,
490 reason = "writing errno through the C runtime's per-thread pointer"
491 )]
492 fn drop(&mut self) {
493 if let (Some(p), Some(value)) = (errno_location(), self.0) {
494 // SAFETY: as `take`: this thread's own errno slot.
495 unsafe { *p = value };
496 }
497 }
498 }
499
500 /// The calling thread's `errno` slot, where the C runtime names it.
501 /// `None` on a Unix this does not know, which then keeps the old
502 /// behaviour rather than failing to build.
503 #[expect(unsafe_code, reason = "the C runtime's errno accessor is an extern fn")]
504 fn errno_location() -> Option<*mut core::ffi::c_int> {
505 #[cfg(any(target_os = "linux", target_os = "android"))]
506 // SAFETY: takes no arguments, returns this thread's errno slot, and
507 // is async-signal-safe.
508 return Some(unsafe { libc::__errno_location() });
509 #[cfg(any(
510 target_os = "macos",
511 target_os = "ios",
512 target_os = "freebsd",
513 target_os = "dragonfly"
514 ))]
515 // SAFETY: as above; this platform's name for the same accessor.
516 return Some(unsafe { libc::__error() });
517 #[cfg(any(target_os = "openbsd", target_os = "netbsd"))]
518 // SAFETY: as above; this platform's name for the same accessor.
519 return Some(unsafe { libc::__errno() });
520 #[allow(unreachable_code)]
521 None
522 }
523
524 /// Wake a parked thread. `SIGUSR2`.
525 ///
526 /// Deliberately empty: the work is done by `sigsuspend` RETURNING. A
527 /// handler that did anything here would be doing it in the wrong
528 /// thread's context for no reason.
529 extern "C" fn on_resume(_signal: core::ffi::c_int) {}
530
531 /// Install both handlers, once for the process.
532 #[expect(unsafe_code, reason = "installing a signal handler is a syscall")]
533 fn install_handlers() {
534 static ONCE: Once = Once::new();
535 ONCE.call_once(|| {
536 // SAFETY: both structures are zeroed, then filled with a real
537 // handler and a real mask before `sigaction` reads them, and
538 // the handlers have C ABI and the right signature.
539 unsafe {
540 let mut suspend: libc::sigaction = core::mem::zeroed();
541 suspend.sa_sigaction = on_suspend as *const () as libc::sighandler_t;
542 libc::sigemptyset(&raw mut suspend.sa_mask);
543 // The lost-wakeup guard: see this module's header.
544 libc::sigaddset(&raw mut suspend.sa_mask, libc::SIGUSR2);
545 // A task frozen mid-`read` should resume the read, not see
546 // `EINTR` it never asked about.
547 suspend.sa_flags = libc::SA_RESTART;
548 libc::sigaction(libc::SIGUSR1, &raw const suspend, core::ptr::null_mut());
549
550 let mut resume: libc::sigaction = core::mem::zeroed();
551 resume.sa_sigaction = on_resume as *const () as libc::sighandler_t;
552 libc::sigemptyset(&raw mut resume.sa_mask);
553 resume.sa_flags = libc::SA_RESTART;
554 libc::sigaction(libc::SIGUSR2, &raw const resume, core::ptr::null_mut());
555 }
556 });
557 }
558
559 /// The OS handle a `JoinHandle` owns, registered so it can be frozen.
560 ///
561 /// Taken from the spawner so the thread has a handle before it has
562 /// run, exactly as on Windows: a thread that registered itself would
563 /// have a window in which it holds the run permit and cannot be
564 /// frozen.
565 #[must_use]
566 pub fn thread_handle_of(join: &std::thread::JoinHandle<()>) -> u64 {
567 use std::os::unix::thread::JoinHandleExt as _;
568
569 install_handlers();
570 let tid = tid_of(join.as_pthread_t());
571 if tid == 0 {
572 return 0;
573 }
574 for (index, entry) in TABLE.iter().enumerate() {
575 // Claim the entry with the store that publishes it, so two
576 // threads spawning at once cannot take the same one.
577 if entry
578 .tid
579 .compare_exchange(0, tid, Ordering::AcqRel, Ordering::Acquire)
580 .is_ok()
581 {
582 entry.state.store(RUNNING, Ordering::Release);
583 return index.saturating_add(1) as u64;
584 }
585 }
586 // Full. Zero is the port's "no thread", so it declines to freeze
587 // this one rather than freezing somebody else.
588 0
589 }
590
591 /// Freeze the thread, and do not answer until it really is frozen.
592 #[expect(unsafe_code, reason = "signalling a thread is a syscall")]
593 pub fn freeze(handle: u64) -> bool {
594 let Some(entry) = entry_at(handle) else {
595 return false;
596 };
597 let tid = entry.tid.load(Ordering::Acquire);
598 if tid == 0 {
599 return false;
600 }
601 if entry
602 .state
603 .compare_exchange(RUNNING, ASKED, Ordering::AcqRel, Ordering::Acquire)
604 .is_err()
605 {
606 // Already asked or already parked. Not this caller's to take.
607 return false;
608 }
609 // SAFETY: `tid` was registered by `thread_handle_of` from a live
610 // `JoinHandle` the port still owns, and cleared by `release`
611 // before that thread's slot is reused.
612 if unsafe { libc::pthread_kill(tid as libc::pthread_t, libc::SIGUSR1) } != 0 {
613 entry.state.store(RUNNING, Ordering::Release);
614 return false;
615 }
616 for _ in 0..PARK_SPINS {
617 if entry.state.load(Ordering::Acquire) == PARKED {
618 return true;
619 }
620 std::thread::yield_now();
621 }
622 // It never parked. Put the state back: the handler, if it runs at
623 // all now, will find its `ASKED -> PARKED` swap refused and return
624 // without parking, which is the unwind this depends on.
625 entry.state.store(RUNNING, Ordering::Release);
626 false
627 }
628
629 /// Thaw the thread. `false` if it was not parked.
630 #[expect(unsafe_code, reason = "signalling a thread is a syscall")]
631 pub fn thaw(handle: u64) -> bool {
632 let Some(entry) = entry_at(handle) else {
633 return false;
634 };
635 let tid = entry.tid.load(Ordering::Acquire);
636 if tid == 0 {
637 return false;
638 }
639 // The state goes back FIRST, so the handler's loop sees `RUNNING`
640 // when the signal wakes it. The other order is a thread that wakes
641 // up, re-reads `PARKED`, and suspends itself again for ever.
642 if entry.state.swap(RUNNING, Ordering::AcqRel) != PARKED {
643 return false;
644 }
645 // SAFETY: as `freeze`.
646 unsafe { libc::pthread_kill(tid as libc::pthread_t, libc::SIGUSR2) == 0 }
647 }
648
649 /// Give the entry back. There is no OS handle to close on Unix.
650 pub fn release(handle: u64) {
651 let Some(entry) = entry_at(handle) else {
652 return;
653 };
654 entry.state.store(RUNNING, Ordering::Release);
655 entry.tid.store(0, Ordering::Release);
656 }
657}
658
659#[cfg(not(any(windows, unix)))]
660mod backend {
661 //! Everywhere else: **cooperative only**, and it says so rather than
662 //! pretending.
663 //!
664 //! Windows and Unix both have a way to stop a thread that is not
665 //! cooperating. A platform that is neither does not necessarily, and
666 //! guessing at one would be worse than reporting the truth: on this
667 //! platform a task that never blocks and never yields will not be
668 //! taken off the CPU, and the demos that rely on preemption
669 //! (`integer.c`, `flop.c`) will starve their neighbours.
670 //!
671 //! [`super::PREEMPTIVE`] is the honest report of that, and a cell
672 //! should check it rather than assume.
673
674 /// Is preemption available on this platform?
675 pub const PREEMPTIVE: bool = false;
676
677 /// No handle to take.
678 #[must_use]
679 pub fn thread_handle_of(_join: &std::thread::JoinHandle<()>) -> u64 {
680 0
681 }
682
683 /// Cannot freeze a thread here.
684 pub fn freeze(_handle: u64) -> bool {
685 false
686 }
687
688 /// Nothing was frozen, so nothing thaws.
689 pub fn thaw(_handle: u64) -> bool {
690 false
691 }
692
693 /// Nothing to release.
694 pub fn release(_handle: u64) {}
695}
696
697/// Whether this platform can take a task off the CPU that does not
698/// cooperate.
699///
700/// A cell that runs the standard demo files should refuse to report a pass
701/// when this is false and `integer`-shaped tasks are in the mix: they
702/// never block, so without preemption everything below them starves and
703/// the failure looks like somebody else's bug.
704pub const PREEMPTIVE: bool = backend::PREEMPTIVE;
705
706// --------------------------------------------------------- the switcher --
707
708/// What the port calls to choose the next task.
709///
710/// It must set [`CURRENT`] to the incoming task's index; whatever it
711/// leaves there is what runs next. The same contract as the Cortex-M
712/// port's `CURRENT_SP_SLOT`, one level up: there the scheduler names a
713/// stack pointer, here it names a thread.
714pub type Scheduler = extern "C" fn();
715
716static SCHEDULER: AtomicUsize = AtomicUsize::new(0);
717
718/// Install the function the port calls to pick the next task.
719pub fn set_scheduler(f: Scheduler) {
720 SCHEDULER.store(f as usize, Ordering::SeqCst);
721}
722
723fn ask_scheduler() {
724 let f = SCHEDULER.load(Ordering::SeqCst);
725 if f != 0 {
726 // SAFETY: `f` was stored by `set_scheduler` from a `Scheduler`,
727 // and nothing else ever writes this.
728 #[expect(unsafe_code, reason = "the installed scheduler is a fn pointer")]
729 let f: Scheduler = unsafe { core::mem::transmute::<usize, Scheduler>(f) };
730 f();
731 }
732}
733
734/// Permits handed to each slot, so a cell can ask who actually got the CPU.
735#[allow(clippy::declare_interior_mutable_const)]
736const NO_GRANTS: AtomicU64 = AtomicU64::new(0);
737static GRANTS: [AtomicU64; MAX_TASKS] = [NO_GRANTS; MAX_TASKS];
738
739/// How many times slot `index` was handed the run permit.
740#[must_use]
741pub fn grants_to(index: usize) -> u64 {
742 GRANTS.get(index).map_or(0, |c| c.load(Ordering::Relaxed))
743}
744
745/// Give `index` the run permit.
746fn grant(index: usize) {
747 if let Some(c) = GRANTS.get(index) {
748 c.fetch_add(1, Ordering::Relaxed);
749 }
750 let Some(slot) = slots().get(index) else {
751 return;
752 };
753 if slot.frozen.swap(false, Ordering::SeqCst) {
754 // It was taken off the CPU by the tick thread, not parked, so it
755 // is not waiting on anything and must be thawed instead.
756 backend::thaw(slot.os_thread.load(Ordering::SeqCst));
757 return;
758 }
759 let Ok(mut granted) = slot.granted.lock() else {
760 return;
761 };
762 *granted = true;
763 // `notify_all`, not `notify_one`: an orphaned thread from an earlier
764 // occupant of this slot may be waiting on the same condvar, and
765 // waking only one of them could wake the wrong one. Each checks its
766 // own generation and the loser goes back to sleep for good.
767 slot.wake.notify_all();
768}
769
770/// Wait until this task has the run permit, or forever if this thread is
771/// no longer the slot's occupant.
772fn await_permit(index: usize, generation: u64) {
773 let Some(slot) = slots().get(index) else {
774 return;
775 };
776 let Ok(granted) = slot.granted.lock() else {
777 return;
778 };
779 take_permit(slot, generation, granted);
780}
781
782/// The waiting half, given a guard the caller already holds.
783///
784/// `pend_switch` must take this lock BEFORE it lets anything else run, so
785/// the wait cannot be a self-contained function -- it has to be handed the
786/// guard.
787fn take_permit(slot: &Slot, generation: u64, mut granted: std::sync::MutexGuard<'_, bool>) {
788 loop {
789 if slot.generation.load(Ordering::SeqCst) != generation {
790 // A later task has this slot. This thread belongs to a task
791 // that no longer exists and must never run again.
792 let Ok(next) = slot.wake.wait(granted) else {
793 return;
794 };
795 granted = next;
796 continue;
797 }
798 if *granted {
799 *granted = false;
800 return;
801 }
802 let Ok(next) = slot.wake.wait(granted) else {
803 return;
804 };
805 granted = next;
806 }
807}
808
809/// `portYIELD()`: ask the scheduler who is next and hand the CPU over.
810///
811/// Returns when this task is next given the permit, which is what makes a
812/// blocking C call possible: the locals are on this thread's own stack and
813/// are still there when it comes back.
814pub fn pend_switch() {
815 let who = me();
816 if CRITICAL.owner.load(Ordering::Acquire) == who {
817 // We are inside a critical section, which on this port means
818 // inside a kernel call. This is `PendSV` being PENDED, not taken:
819 // record it, and `exit_critical` runs it on the way out.
820 //
821 // Taking it here instead re-enters the kernel while an outer call
822 // still holds a `&mut` to it -- and the kernel's idea of the
823 // current task then moves under the feet of the call in progress,
824 // so every API that means "the calling task" names the wrong one.
825 // `GenQTest` ends a test with `vTaskPrioritySet( NULL,
826 // genqMUTEX_LOW_PRIORITY )`, and the task set to low priority was
827 // the HIGH priority one.
828 PENDING.store(true, Ordering::SeqCst);
829 return;
830 }
831 switch_now(who);
832}
833
834/// Set when a switch has been asked for and not yet taken. One flag,
835/// because there is one CPU: this is `ICSR.PENDSVSET`.
836static PENDING: AtomicBool = AtomicBool::new(false);
837
838/// Take the switch. The caller must NOT hold the critical section.
839fn switch_now(who: u64) {
840 // The decision and the handover happen with the tick thread locked
841 // out. Without this the two interleave: a task asks the scheduler, the
842 // tick thread freezes it mid-answer and hands the CPU to somebody
843 // else, and when the frozen task is later thawed it finishes a
844 // handover whose answer is now stale -- granting the permit to a task
845 // that should not have it, so two run at once.
846 //
847 // That race is not theoretical and it is not loud. It shows up as
848 // `eTaskGetState( x ) == eBlocked` failing three statements after a
849 // `vTaskResume`, which reads as a scheduler bug rather than a port
850 // one.
851 CRITICAL.enter(who);
852 let mine = CURRENT.load(Ordering::SeqCst);
853 ask_scheduler();
854 let next = CURRENT.load(Ordering::SeqCst);
855 if next == mine || next == NO_TASK || mine == NO_TASK {
856 CRITICAL.exit(who);
857 return;
858 }
859 let Some(slot) = slots().get(mine) else {
860 CRITICAL.exit(who);
861 return;
862 };
863 // Our own permit lock, taken BEFORE anything else can run, so a grant
864 // aimed at us cannot land between the handover and the wait.
865 let Ok(granted) = slot.granted.lock() else {
866 CRITICAL.exit(who);
867 return;
868 };
869 let generation = slot.generation.load(Ordering::SeqCst);
870 grant(next);
871
872 // Put the critical section down completely while parked, and pick it
873 // up at the same depth on the way back. See `release_all`.
874 let depth = CRITICAL.release_all(who);
875
876 take_permit(slot, generation, granted);
877
878 CRITICAL.reacquire(who, depth);
879 // ...and undo the level this function took on the way in.
880 CRITICAL.exit(who);
881}
882
883/// Give task `index` a thread, parked until it is granted the permit.
884///
885/// `entry` is handed the task index, exactly as `init_stack` does on the
886/// Cortex-M port, so one trampoline can serve every task.
887pub fn init_task(index: usize, entry: extern "C" fn(usize) -> !) {
888 let Some(slot) = slots().get(index) else {
889 return;
890 };
891 // A slot that already had a thread gets a NEW one, and the old one is
892 // orphaned rather than reused: it is parked inside the C function of a
893 // task that no longer exists, and there is no safe way to bring it
894 // back to the top. Bumping the generation is what stops it competing
895 // for the permit.
896 let generation = slot
897 .generation
898 .fetch_add(1, Ordering::SeqCst)
899 .wrapping_add(1);
900 if generation > 1 {
901 ORPHANS.fetch_add(1, Ordering::Relaxed);
902 // Wake the outgoing occupant so it can see its generation has
903 // passed and settle; and give its handle back.
904 //
905 // And clear the two flags that describe the OLD thread, which a new
906 // occupant must not inherit. The tick can freeze a thread in the gap
907 // between its grant and its waking to take it; if that task is then
908 // deleted, the slot still says both "granted" and "frozen":
909 //
910 // * a standing GRANT lets the new thread run the moment it starts,
911 // uninvited, while the kernel has somebody else current -- on
912 // Linux, "this thread is task 77, the kernel believes 43";
913 // * a standing FREEZE makes the new occupant's first grant try to
914 // THAW a thread that is not frozen -- a no-op -- instead of
915 // handing it the permit, so the kernel's current task runs nowhere
916 // until the next tick, and every tick it is chosen is lost
917 // (`TimerDemo`'s exact-tick ISR checks fail).
918 //
919 // Clearing one alone produced the other's failure; both were
920 // measured, with `death.c` deleting a SUICID task the tick had just
921 // interrupted. The counter says how often a reuse found either.
922 if let Ok(mut granted) = slot.granted.lock() {
923 let stale_grant = core::mem::replace(&mut *granted, false);
924 let stale_freeze = slot.frozen.swap(false, Ordering::SeqCst);
925 if stale_grant || stale_freeze {
926 STALE_GRANTS.fetch_add(1, Ordering::Relaxed);
927 }
928 drop(granted);
929 slot.wake.notify_all();
930 }
931 backend::release(slot.os_thread.swap(0, Ordering::SeqCst));
932 }
933 let spawned = std::thread::Builder::new()
934 .name(format!("kairos-task-{index}.{generation}"))
935 .stack_size(STACK_BYTES)
936 .spawn(move || {
937 MY_INDEX.set(Some(index));
938 await_permit(index, generation);
939 entry(index);
940 });
941
942 // Register the thread from HERE, not from inside it.
943 //
944 // A thread that names itself has a window in which it exists, may hold
945 // the run permit, and has no handle -- and a tick landing in that
946 // window cannot freeze it. That is not a missed tick: it is the one
947 // case where the port would have to decline a switch, which is the
948 // path that used to leave the kernel and the port disagreeing about
949 // who was running.
950 //
951 // The `JoinHandle` owns the OS thread handle, so it is kept: dropping
952 // it would close the handle and every later freeze would fail.
953 if let Ok(handle) = spawned {
954 if let Some(slot) = slots().get(index) {
955 slot.os_thread
956 .store(backend::thread_handle_of(&handle), Ordering::SeqCst);
957 }
958 if let Ok(mut live) = threads().lock() {
959 live.push(handle);
960 }
961 }
962}
963
964thread_local! {
965 /// Which task this thread IS, as opposed to which task the kernel
966 /// believes is running.
967 ///
968 /// On a chip those cannot differ: there is one CPU and the kernel's
969 /// `current` is definitionally who is on it. Here they are two
970 /// different facts kept in two places, so they CAN differ -- and when
971 /// they do, every API that means "the calling task" quietly names
972 /// somebody else. Having the thread able to state its own identity is
973 /// what makes that checkable rather than arguable.
974 static MY_INDEX: core::cell::Cell<Option<usize>> = const { core::cell::Cell::new(None) };
975}
976
977/// Is a context switch outstanding but not yet taken? (`ICSR.PENDSVSET`.)
978#[must_use]
979pub fn switch_pending() -> bool {
980 PENDING.load(Ordering::SeqCst)
981}
982
983/// How deep the calling thread is inside critical sections, and whether it
984/// is the holder at all.
985#[must_use]
986pub fn critical_depth() -> (bool, u32) {
987 let mine = CRITICAL.owner.load(Ordering::Acquire) == me();
988 (mine, CRITICAL.nesting.load(Ordering::Relaxed))
989}
990
991/// Which task the CALLING THREAD is, if it is a task thread at all.
992#[must_use]
993pub fn my_index() -> Option<usize> {
994 MY_INDEX.with(core::cell::Cell::get)
995}
996
997/// What the tick thread did, tick by tick.
998///
999/// A port that declines to switch reports the same thing as a port that
1000/// was never asked, and those are different bugs -- so it counts both.
1001static TICKS_SEEN: AtomicU64 = AtomicU64::new(0);
1002static TICKS_WANTED_SWITCH: AtomicU64 = AtomicU64::new(0);
1003static TICKS_COULD_NOT_FREEZE: AtomicU64 = AtomicU64::new(0);
1004static TICKS_SWITCHED: AtomicU64 = AtomicU64::new(0);
1005
1006/// `(ticks, wanted a switch, could not freeze, actually switched)`.
1007#[must_use]
1008pub fn tick_stats() -> (u64, u64, u64, u64) {
1009 (
1010 TICKS_SEEN.load(Ordering::Relaxed),
1011 TICKS_WANTED_SWITCH.load(Ordering::Relaxed),
1012 TICKS_COULD_NOT_FREEZE.load(Ordering::Relaxed),
1013 TICKS_SWITCHED.load(Ordering::Relaxed),
1014 )
1015}
1016
1017/// Bytes of OS stack per task thread.
1018///
1019/// Every orphaned thread keeps one until the process ends, and `death.c`
1020/// makes orphans on purpose, so this is not purely a depth question. The
1021/// deepest a demo task reached on the Cortex-M3 cell, measured by stack
1022/// painting, was 122 words; a host frame is wider but not by three orders
1023/// of magnitude.
1024const STACK_BYTES: usize = 128 * 1024;
1025
1026/// How many task threads have been orphaned by a slot being reused.
1027///
1028/// Reported rather than hidden: each one holds [`STACK_BYTES`] of address
1029/// space for the life of the process, and a cell that makes thousands of
1030/// them should be able to see that it is doing so. Terminating them
1031/// instead is what FreeRTOS's Posix port does with `pthread_cancel`, and
1032/// is the obvious next step for this port.
1033static ORPHANS: AtomicU64 = AtomicU64::new(0);
1034
1035/// Slots handed to a new occupant while a grant to, or a freeze of, the old
1036/// one was still standing (see `init_task`).
1037static STALE_GRANTS: AtomicU64 = AtomicU64::new(0);
1038
1039/// How many slots `init_task` found holding a stale grant or freeze.
1040#[must_use]
1041pub fn stale_grants() -> u64 {
1042 STALE_GRANTS.load(Ordering::Relaxed)
1043}
1044
1045/// How many task threads have been orphaned by a slot being reused.
1046#[must_use]
1047pub fn orphaned_threads() -> u64 {
1048 ORPHANS.load(Ordering::Relaxed)
1049}
1050
1051/// The spawned task threads, kept alive so their OS handles stay valid.
1052fn threads() -> &'static Mutex<Vec<std::thread::JoinHandle<()>>> {
1053 static THREADS: OnceLock<Mutex<Vec<std::thread::JoinHandle<()>>>> = OnceLock::new();
1054 THREADS.get_or_init(|| Mutex::new(Vec::new()))
1055}
1056
1057/// Start the first task and never come back.
1058///
1059/// The calling thread becomes the idle of last resort: it parks, because
1060/// every task now has its own thread and this one has nothing left to do.
1061pub fn start_first_task(index: usize) -> ! {
1062 CURRENT.store(index, Ordering::SeqCst);
1063 grant(index);
1064 loop {
1065 // Long enough that this thread costs nothing, short enough that a
1066 // cell which exits can do so promptly.
1067 std::thread::sleep(Duration::from_millis(50));
1068 }
1069}
1070
1071// ------------------------------------------------------------ the tick --
1072
1073/// What the tick thread calls once per period.
1074///
1075/// It stands where `SysTick` does on a chip: it is called with the
1076/// critical-section lock HELD and the running task frozen, so it may
1077/// touch the kernel freely. `true` asks for a context switch, exactly as
1078/// `xTaskIncrementTick` answering `pdTRUE` does.
1079pub type TickFn = extern "C" fn() -> bool;
1080
1081/// The tick thread's own handle on the system.
1082#[derive(Debug)]
1083pub struct Ticker {
1084 period: Duration,
1085 on_tick: TickFn,
1086}
1087
1088impl Ticker {
1089 /// A ticker that calls `on_tick` every `period`.
1090 #[must_use]
1091 pub const fn new(period: Duration, on_tick: TickFn) -> Self {
1092 Self { period, on_tick }
1093 }
1094
1095 /// Run the ticker on its own thread, forever.
1096 pub fn spawn(self) {
1097 let _ = std::thread::Builder::new()
1098 .name("kairos-tick".to_owned())
1099 .spawn(move || {
1100 loop {
1101 std::thread::sleep(self.period);
1102 self.interrupt();
1103 }
1104 });
1105 }
1106
1107 /// One tick, in the order that makes freezing a thread safe.
1108 ///
1109 /// 1. Take the critical-section lock. This WAITS for the running task
1110 /// to leave any critical section, so when it is held the running
1111 /// task is outside the kernel and outside our heap.
1112 /// 2. Freeze the running task. It cannot be holding a lock of ours,
1113 /// by step 1.
1114 /// 3. Do the kernel's tick work and ask who is next.
1115 /// 4. Release the lock, then thaw or grant the incoming task.
1116 ///
1117 /// Steps 1 and 2 in the other order is the classic deadlock: freeze a
1118 /// task inside `pvPortMalloc` and the next task to allocate waits for
1119 /// a lock whose holder is never going to run again.
1120 fn interrupt(&self) {
1121 let me = me();
1122 CRITICAL.enter(me);
1123 IN_TICK.store(true, Ordering::SeqCst);
1124
1125 TICKS_SEEN.fetch_add(1, Ordering::Relaxed);
1126 let running = CURRENT.load(Ordering::SeqCst);
1127 let frozen = self.freeze_running(running);
1128 if !frozen {
1129 TICKS_COULD_NOT_FREEZE.fetch_add(1, Ordering::Relaxed);
1130 }
1131
1132 let want_switch = (self.on_tick)();
1133 if want_switch {
1134 TICKS_WANTED_SWITCH.fetch_add(1, Ordering::Relaxed);
1135 }
1136
1137 // Ask the scheduler ONLY when its answer can be honoured.
1138 //
1139 // The scheduler is not a query: `switch_context` MOVES the
1140 // kernel's idea of the current task. Asking it and then declining
1141 // to switch -- because the running thread could not be frozen --
1142 // leaves the kernel and this port disagreeing about who is
1143 // running, and the running task keeps executing under somebody
1144 // else's identity. Every API that means "the calling task" then
1145 // names the wrong one: `vTaskPrioritySet( NULL, x )` sets a task
1146 // that is not the caller, `uxTaskPriorityGet( NULL )` answers for
1147 // a task that is not asking.
1148 //
1149 // `GenQTest` catches it precisely, because it ends a test with
1150 // `vTaskPrioritySet( NULL, genqMUTEX_LOW_PRIORITY )` -- and the
1151 // task that was wrongly set to low priority was the HIGH priority
1152 // one, three hundred lines from the assertion that noticed.
1153 let next = if want_switch && frozen {
1154 ask_scheduler();
1155 CURRENT.load(Ordering::SeqCst)
1156 } else {
1157 running
1158 };
1159
1160 IN_TICK.store(false, Ordering::SeqCst);
1161 // A switch the kernel asked for during the tick has been taken
1162 // here, or declined here; either way it is not still outstanding.
1163 PENDING.store(false, Ordering::SeqCst);
1164 CRITICAL.exit(me);
1165
1166 if next == running || next == NO_TASK {
1167 // Nothing to change. Put the task back exactly as it was.
1168 //
1169 // A switch the tick wanted and could not take is not lost, only
1170 // deferred: the running task's next kernel call asks the
1171 // scheduler again, and the ready higher-priority task is still
1172 // ready.
1173 if frozen {
1174 if let Some(slot) = slots().get(running) {
1175 slot.frozen.store(false, Ordering::SeqCst);
1176 backend::thaw(slot.os_thread.load(Ordering::SeqCst));
1177 }
1178 }
1179 return;
1180 }
1181 TICKS_SWITCHED.fetch_add(1, Ordering::Relaxed);
1182 grant(next);
1183 }
1184
1185 /// Freeze the running task, and record that it was frozen rather than
1186 /// parked so [`grant`] thaws it instead of signalling a condvar
1187 /// nobody is waiting on.
1188 fn freeze_running(&self, running: usize) -> bool {
1189 if running == NO_TASK || !preemption_enabled() {
1190 return false;
1191 }
1192 let Some(slot) = slots().get(running) else {
1193 return false;
1194 };
1195 let handle = slot.os_thread.load(Ordering::SeqCst);
1196 if !backend::freeze(handle) {
1197 return false;
1198 }
1199 slot.frozen.store(true, Ordering::SeqCst);
1200 true
1201 }
1202}
1203
1204/// Turn preemption off at runtime, with `KAIROS_HOST_NO_PREEMPT=1`.
1205///
1206/// Two uses, and both are about not having to take this port's word for
1207/// anything:
1208///
1209/// * **The poison test.** A cell that claims "a task that never yields was
1210/// taken off the CPU" has to be able to show the claim FAILING when the
1211/// mechanism is removed. Without that, the check might be passing for
1212/// some other reason and nobody would know.
1213/// * **Standing in for the platforms that do not have it.** Windows and
1214/// Unix both preempt; a platform that is neither reports `PREEMPTIVE`
1215/// false, and this is how that is reproduced on the machine you have
1216/// rather than discovered on the machine you do not.
1217fn preemption_enabled() -> bool {
1218 static ENABLED: OnceLock<bool> = OnceLock::new();
1219 *ENABLED.get_or_init(|| std::env::var("KAIROS_HOST_NO_PREEMPT").as_deref() != Ok("1"))
1220}
1221
1222/// True while the tick thread is inside [`Ticker::interrupt`].
1223///
1224/// This port's only interrupt context. It is a flag rather than a magic
1225/// owner id in the critical lock because the lock's identity has to be the
1226/// SAME function everywhere -- `pend_switch` compares against it to decide
1227/// whether it is being pended or taken, and a tick thread that held the
1228/// lock under a different name would take a switch from inside the tick.
1229static IN_TICK: AtomicBool = AtomicBool::new(false);
1230
1231/// The OS thread backing task `index`, or zero if it has none yet.
1232///
1233/// A cell reports this the way a bare-metal cell reports a saved stack
1234/// pointer: a slot with an entry point and no thread is a task that is
1235/// SCHEDULABLE WITH NOWHERE TO RUN, which is a real bug shape and one that
1236/// presents a long way from its cause.
1237#[must_use]
1238pub fn thread_of(index: usize) -> u64 {
1239 slots()
1240 .get(index)
1241 .map_or(0, |s| s.os_thread.load(Ordering::SeqCst))
1242}
1243
1244/// Hand a task slot's OS handle back. For a cell that deletes tasks.
1245pub fn release_task(index: usize) {
1246 if let Some(slot) = slots().get(index) {
1247 backend::release(slot.os_thread.swap(0, Ordering::SeqCst));
1248 }
1249}
1250
1251// ------------------------------------------------------ virtual cores --
1252
1253/// Which kernel core the code holding the run permit is running as.
1254///
1255/// This port runs ONE task thread at a time, whatever the kernel's core
1256/// count. A two-core cell keeps that, and takes turns: it tells the port
1257/// which core the permit currently stands for, and the kernel reads it back
1258/// through [`Port::core_id`] -- `portGET_CORE_ID()`. Two cores this way are
1259/// virtual: every two-core path in the kernel is exercised (per-core
1260/// current tasks, cross-core yields, selection that skips a task the other
1261/// core holds), while the kernel is still never entered by two threads at
1262/// once. The policy of WHEN a turn passes belongs to the cell; this is only
1263/// where the answer is kept.
1264static CORE: core::sync::atomic::AtomicU8 = core::sync::atomic::AtomicU8::new(0);
1265
1266/// Make the run permit stand for kernel core `core` from now on.
1267pub fn set_core(core: u8) {
1268 CORE.store(core, Ordering::SeqCst);
1269}
1270
1271/// The kernel core the run permit currently stands for.
1272#[must_use]
1273pub fn core() -> u8 {
1274 CORE.load(Ordering::SeqCst)
1275}
1276
1277// -------------------------------------------------------------- the port --
1278
1279/// The host [`Port`].
1280#[derive(Debug, Default)]
1281pub struct HostPort {
1282 yields: AtomicU64,
1283 ticks: AtomicU64,
1284}
1285
1286impl HostPort {
1287 /// A port with nothing counted yet.
1288 #[must_use]
1289 pub const fn new() -> Self {
1290 Self {
1291 yields: AtomicU64::new(0),
1292 ticks: AtomicU64::new(0),
1293 }
1294 }
1295
1296 /// Ticks delivered so far.
1297 #[must_use]
1298 pub fn tick_count(&self) -> u64 {
1299 self.ticks.load(Ordering::Relaxed)
1300 }
1301
1302 /// `portYIELD()` calls so far.
1303 #[must_use]
1304 pub fn yield_count(&self) -> u64 {
1305 self.yields.load(Ordering::Relaxed)
1306 }
1307
1308 /// Count a tick the ticker delivered.
1309 pub fn note_tick(&self) {
1310 self.ticks.fetch_add(1, Ordering::Relaxed);
1311 }
1312}
1313
1314/// The identity the critical lock uses for the calling thread.
1315///
1316/// A thread id, not a handle: it only has to be unique and non-zero, and
1317/// `ThreadId` is neither a number nor stable across runs, so it is hashed
1318/// into one.
1319fn me() -> u64 {
1320 use std::hash::{Hash as _, Hasher as _};
1321 let mut h = std::collections::hash_map::DefaultHasher::new();
1322 std::thread::current().id().hash(&mut h);
1323 // Zero means "the lock is free", so it is the one value that may not
1324 // be handed out.
1325 match h.finish() {
1326 0 => 1,
1327 other => other,
1328 }
1329}
1330
1331impl Port for HostPort {
1332 /// This port COMMITS the switch.
1333 ///
1334 /// `pend_switch` does not merely note a request: it asks the scheduler,
1335 /// hands the run permit over and parks the caller, so the decision and
1336 /// the change of running task are one step -- exactly what every
1337 /// bare-metal port here declares, for the same reason.
1338 ///
1339 /// Leaving it `false` was a real defect and a quiet one. The kernel
1340 /// then treats itself as STACKLESS and moves `current` inside
1341 /// `port_yield` by calling `switch_context` directly; the port then
1342 /// switches AGAIN on the way out. Two selections per yield, and the
1343 /// round robin advances twice -- which on a ready list of two is the
1344 /// same task every time, for ever.
1345 ///
1346 /// That is how `EventGroups` and `dynamic` stopped dead in the C ABI
1347 /// host cell while a busier demo's equal-priority tasks took a
1348 /// thousandfold share, and how one of two never-yielding tasks in
1349 /// `firmware/host-kernel` got 13.5 million laps against ZERO.
1350 ///
1351 /// The trait's own doc names the other half of the cost: between the
1352 /// kernel moving `current` and the port catching up, "code runs as a
1353 /// task the kernel no longer thinks is current".
1354 const COMMITS_SWITCH: bool = true;
1355
1356 fn yield_now(&self) {
1357 self.yields.fetch_add(1, Ordering::Relaxed);
1358 pend_switch();
1359 }
1360
1361 fn yield_from_isr(&self, woken: Woken) {
1362 if woken.needed() {
1363 self.yields.fetch_add(1, Ordering::Relaxed);
1364 pend_switch();
1365 }
1366 }
1367
1368 fn enter_critical(&self) {
1369 CRITICAL.enter(me());
1370 }
1371
1372 fn exit_critical(&self) {
1373 let who = me();
1374 // Released, not merely counted down: this is the outermost exit,
1375 // which on a chip is where `cpsie i` lets the pending `PendSV`
1376 // finally run. Same place, same reason.
1377 if CRITICAL.exit(who)
1378 && !IN_TICK.load(Ordering::SeqCst)
1379 && PENDING.swap(false, Ordering::SeqCst)
1380 {
1381 switch_now(who);
1382 }
1383 }
1384
1385 fn set_interrupt_mask_from_isr(&self) -> u32 {
1386 let who = me();
1387 let already = CRITICAL.owner.load(Ordering::Acquire) == who;
1388 if !already {
1389 CRITICAL.enter(who);
1390 }
1391 u32::from(already)
1392 }
1393
1394 fn clear_interrupt_mask_from_isr(&self, saved: u32) {
1395 // Restore, do not release: a mask taken inside somebody else's
1396 // critical section must leave that section standing. The Cortex-M
1397 // port learned this the hard way -- an unconditional release there
1398 // ended a caller's critical section halfway through and let a
1399 // half-created task be scheduled.
1400 if saved == 0 {
1401 CRITICAL.exit(me());
1402 }
1403 }
1404
1405 fn in_isr(&self) -> bool {
1406 // The tick thread is this port's only interrupt context, and only
1407 // one context runs at a time -- so if the tick is inside, the
1408 // caller IS the tick.
1409 IN_TICK.load(Ordering::SeqCst)
1410 }
1411
1412 /// `portGET_CORE_ID()`: the core the run permit stands for -- always 0
1413 /// unless a two-core cell has passed a turn ([`set_core`]).
1414 fn core_id(&self) -> u8 {
1415 core()
1416 }
1417
1418 fn count_tick(&self) {
1419 self.note_tick();
1420 }
1421
1422 fn count_yield(&self) {
1423 self.yields.fetch_add(1, Ordering::Relaxed);
1424 }
1425}
1426
1427#[cfg(test)]
1428#[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
1429mod tests {
1430 use super::*;
1431
1432 #[test]
1433 fn a_critical_section_nests_and_only_the_outermost_exit_releases() {
1434 let who = 7;
1435 CRITICAL.enter(who);
1436 assert_ne!(CRITICAL.owner.load(Ordering::Acquire), 0);
1437 CRITICAL.enter(who);
1438 assert!(!CRITICAL.exit(who), "the inner exit must not release");
1439 assert_ne!(CRITICAL.owner.load(Ordering::Acquire), 0);
1440 assert!(CRITICAL.exit(who), "the outer exit releases");
1441 assert_eq!(CRITICAL.owner.load(Ordering::Acquire), 0);
1442 }
1443
1444 #[test]
1445 fn a_thread_cannot_release_a_critical_section_it_does_not_hold() {
1446 CRITICAL.enter(11);
1447 assert!(!CRITICAL.exit(12), "12 does not hold it");
1448 assert_ne!(CRITICAL.owner.load(Ordering::Acquire), 0, "11 still does");
1449 assert!(CRITICAL.exit(11));
1450 }
1451
1452 #[test]
1453 fn the_thread_identity_is_never_the_free_marker() {
1454 assert_ne!(me(), 0, "zero means the lock is free");
1455 }
1456
1457 /// A yield asked for from inside a critical section must be PENDED,
1458 /// not taken -- that is the whole of what `PendSV` being the
1459 /// lowest-priority exception buys on a chip.
1460 #[test]
1461 fn a_yield_inside_a_critical_section_is_pended_rather_than_taken() {
1462 let who = me();
1463 PENDING.store(false, Ordering::SeqCst);
1464 CRITICAL.enter(who);
1465 pend_switch();
1466 assert!(
1467 PENDING.load(Ordering::SeqCst),
1468 "the switch should be outstanding, not taken"
1469 );
1470 PENDING.store(false, Ordering::SeqCst);
1471 assert!(CRITICAL.exit(who));
1472 }
1473
1474 #[test]
1475 fn the_slot_table_is_the_size_it_says() {
1476 assert_eq!(slots().len(), MAX_TASKS);
1477 }
1478
1479 /// The port must not claim preemption it does not have. A cell reads
1480 /// this to decide whether a demo that never yields can be trusted.
1481 #[test]
1482 fn preemption_is_reported_honestly_for_this_platform() {
1483 assert_eq!(PREEMPTIVE, cfg!(any(windows, unix)));
1484 }
1485
1486 /// Freezing a thread that is **running** is the whole of preemption,
1487 /// so it gets a test that needs no kernel, no cell and no emulator.
1488 ///
1489 /// The subject does nothing but increment a counter, which is the
1490 /// point: it never blocks, never yields and never takes a lock, so
1491 /// the ONLY thing that can stop it is the port. If the counter stops
1492 /// moving while it is frozen and moves again after the thaw, the
1493 /// mechanism works; if `freeze` returned `true` without actually
1494 /// stopping it, this fails — and that is the failure worth catching,
1495 /// because a `freeze` that answers before the thread has stopped puts
1496 /// two tasks on the CPU at once and the damage shows up somewhere
1497 /// else entirely.
1498 #[test]
1499 fn a_running_thread_is_frozen_where_it_stands_and_thaws_again() {
1500 if !PREEMPTIVE {
1501 // Nothing to assert on a platform that says it cannot.
1502 return;
1503 }
1504 if cfg!(miri) {
1505 // Miri interprets Rust; it does not run `SuspendThread` and it
1506 // does not deliver signals. This test IS the foreign call, so
1507 // under Miri there is nothing here it can check -- and the
1508 // package's Miri row is a real gate, not one to break with a
1509 // test that was never going to run under it.
1510 return;
1511 }
1512 static SPINS: AtomicU64 = AtomicU64::new(0);
1513 static STOP: AtomicBool = AtomicBool::new(false);
1514
1515 SPINS.store(0, Ordering::SeqCst);
1516 STOP.store(false, Ordering::SeqCst);
1517
1518 let spinner = std::thread::spawn(|| {
1519 while !STOP.load(Ordering::Relaxed) {
1520 SPINS.fetch_add(1, Ordering::Relaxed);
1521 std::hint::spin_loop();
1522 }
1523 });
1524 let handle = backend::thread_handle_of(&spinner);
1525 assert_ne!(handle, 0, "a spawned thread must have a usable handle");
1526
1527 // Wait for it to be genuinely running, so that what gets frozen
1528 // is a thread in its loop rather than one that has not started.
1529 let mut started = false;
1530 for _ in 0..1_000 {
1531 if SPINS.load(Ordering::SeqCst) > 0 {
1532 started = true;
1533 break;
1534 }
1535 std::thread::sleep(Duration::from_millis(1));
1536 }
1537 assert!(started, "the spinner never ran");
1538
1539 assert!(backend::freeze(handle), "freeze refused a running thread");
1540 let at_freeze = SPINS.load(Ordering::SeqCst);
1541 // Long enough that a thread still on the CPU would move the
1542 // counter by millions, so this is not a close-run thing.
1543 std::thread::sleep(Duration::from_millis(50));
1544 assert_eq!(
1545 SPINS.load(Ordering::SeqCst),
1546 at_freeze,
1547 "a frozen thread kept running"
1548 );
1549
1550 assert!(backend::thaw(handle), "thaw refused a frozen thread");
1551 let mut resumed = false;
1552 for _ in 0..1_000 {
1553 if SPINS.load(Ordering::SeqCst) > at_freeze {
1554 resumed = true;
1555 break;
1556 }
1557 std::thread::sleep(Duration::from_millis(1));
1558 }
1559 assert!(resumed, "a thawed thread never ran again");
1560
1561 STOP.store(true, Ordering::SeqCst);
1562 spinner.join().expect("the spinner should end");
1563 backend::release(handle);
1564 }
1565}