1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
//! NPU backend infrastructure.
//!
//! Vendor NPU runtimes (Qualcomm QNN/Hexagon, Huawei CANN, Core ML bridges,
//! …) are proprietary C/C++ stacks that cannot be part of Joshua's pure-Rust
//! core. This module contains them behind three safety layers:
//!
//! 1. **Trait boundary** ([`NpuBackend`] / [`NpuSession`]): the engine only
//! knows an object-safe token-in / logits-out interface. The default
//! build contains no vendor code, and generation transparently falls back
//! to the candle CPU/GPU path when a backend is missing, fails to load,
//! or starts erroring (a circuit breaker disables it after repeated
//! failures).
//!
//! 2. **Plugin ABI, loaded at runtime** ([`InProcessBackend`]): vendor support ships as
//! a shared library exporting the tiny `joshua_npu_*` C ABI below. Joshua
//! `dlopen`s it with `libloading` — nothing is linked at build time, the
//! `unsafe` surface is confined to the `vendor` module, and a missing library is a
//! clean fallback, not a build or startup failure.
//! [`InProcessBackend`] runs a plugin this way for minimum overhead —
//! accepting that a buggy vendor runtime shares the process.
//!
//! 3. **Process isolation** ([`ShimBackend`]): the same plugin is loaded by
//! the small `joshua-npu-shim` subprocess instead. Control messages run
//! over pipes (NDJSON), tensors over a shared memory-mapped file, and the
//! host enforces timeouts and kills the child on any protocol violation.
//! If the vendor runtime crashes or hangs, one request fails and the
//! server keeps running; the next request starts a fresh shim. This is
//! the recommended mode for production.
//!
//! # The plugin ABI
//!
//! A vendor plugin is any `cdylib` exporting:
//!
//! ```c
//! int32_t joshua_npu_init(const char *model_path, uint32_t n_ctx,
//! uint32_t *out_vocab, void **out_handle);
//! int32_t joshua_npu_forward(void *handle, const uint32_t *tokens,
//! uint32_t n_tokens, uint32_t pos,
//! float *out_logits);
//! int32_t joshua_npu_reset(void *handle);
//! void joshua_npu_free(void *handle);
//! ```
//!
//! Semantics: `init` loads/compiles whatever artifact the vendor needs for
//! `model_path` and reports the vocabulary size. `forward` appends
//! `n_tokens` tokens starting at absolute position `pos` (which always
//! equals the number of tokens fed so far — a KV-cache contract) and writes
//! `out_vocab` logits for the last fed token. `reset` clears the internal
//! state for reuse with an unrelated prompt. All functions return `0` on
//! success.
//!
//! `crates/joshua-mock-npu` is a pure-Rust reference plugin used by the test
//! suite; real vendor plugins wrap their SDK behind the same four symbols.
pub use ShimBackend;
pub use InProcessBackend;
// Internals shared with the `joshua-npu-shim` binary.
use Path;
/// A source of NPU-backed generation sessions.
///
/// Implementations must be cheap to share (`Send + Sync`); session creation
/// may be expensive (model compilation, subprocess start).
/// A stateful generation session on an NPU.
///
/// Mirrors the KV-cache contract of the candle models: `forward(tokens, pos)`
/// appends `tokens` at absolute position `pos` (== number of tokens fed so
/// far) and returns the logits for the last token.
/// The media marker plugins must recognise in multimodal prompts (matches
/// llama.cpp's `mtmd` default marker).
pub const MEDIA_MARKER: &str = "<__media__>";