1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
//! Moving bytes into and out of the ring.
//!
//! This is the crate's concurrency boundary, and the one place where the C++ original
//! and this crate part ways.
//!
//! In the original, the reader `memcpy`s a frame that the writer may be overwriting at
//! the same moment, and detects the collision afterwards through the overrun check. That
//! works on real hardware. It is also a data race in the abstract machine — the talk says
//! so on its own slide — and a data race is undefined behaviour in Rust exactly as in C++.
//!
//! So every byte of the ring is read and written through `AtomicU64` with `Relaxed`
//! ordering. Concurrent atomic accesses are never a data race, and the acquire/release on
//! the counters orders them. Correct by construction, at one atomic per 8 bytes — which on
//! x86-64 is one `mov` per 8 bytes.
//!
//! The `memcpy` version was implemented, benchmarked and removed. It was *slower* at every
//! reader count measured — 0.65x the throughput of this one with a single reader, 0.80x
//! with twelve — because at these payload sizes `copy_nonoverlapping` calls a generic
//! `memcpy` that dispatches on length, while this loop inlines to a handful of plain
//! `mov`s: a `Relaxed` load or store of a `u64` on x86-64 is exactly that. The race bought
//! nothing, so there is no trade to offer. The README has the numbers.
use ;
/// Copy `src` into the ring at `dst`, which must be 8-aligned with `src.len()` rounded up
/// to 8 available.
///
/// # Safety
/// `dst` must point into a live mapping with at least `align_up(src.len(), 8)` bytes, and
/// the caller must be the queue's single producer.
pub unsafe
/// Copy `dst.len()` bytes out of the ring at `src`, which must be 8-aligned.
///
/// # Safety
/// `src` must point into a live mapping with at least `align_up(dst.len(), 8)` bytes.
/// The bytes may be concurrently overwritten by the producer; the caller must validate
/// the read afterwards with the overrun check and discard `dst` on failure.
pub unsafe
/// Write one 8-byte header word.
///
/// # Safety
/// `dst` must be an 8-aligned pointer into a live mapping.
pub unsafe
/// Read one 8-byte header word.
///
/// # Safety
/// `src` must be an 8-aligned pointer into a live mapping.
pub unsafe