1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
//! Local AI Inference Backends
//!
//! Backends that do not require a hosted API account.
//!
//! # The built-in engine
//!
//! **IronWorks is ironmon's only built-in inference engine** — the engine ironmon ships
//! against, and the default everywhere. See [`ironworks`].
//!
//! Everything else here is an **external provider**: software you install, start, or
//! sign in to separately. They are fully supported, but IronMonitor does not embed them.
//! [`crate::agent::backend::BackendType::is_builtin_engine`] draws the line in code.
//!
//! # External providers
//!
//! Self-hosted servers:
//!
//! - **Ollama**: popular local LLM server with easy model management
//! - **vLLM**: high-performance inference server with OpenAI-compatible API
//! - **TensorRT-LLM**: NVIDIA's optimized inference engine
//! - **LM Studio**: user-friendly local model server
//! - **llama.cpp**: GGUF models, driven through the `llama-cli` executable
//!
//! Command-line tools ([`cli`]), driven as subprocesses:
//!
//! - **`ollama`**, **`claude`**, **`codex`**, **`gemini`**
//!
//! # Local process is not local inference
//!
//! A CLI tool runs on your machine, but that says nothing about where inference
//! happens. Only `ollama` runs the model locally; `claude`, `codex` and `gemini`
//! relay the prompt — including any hardware telemetry in it — to their vendor's API.
//!
//! [`crate::agent::backend::BackendType::runs_on_host`] is the predicate to check
//! when the question is "does this leave the machine", and it is what orders
//! [`crate::agent::backend::BackendDiscovery::recommended`].
//!
//! # Feature Flags
//!
//! IronWorks, Ollama and the CLI providers are always compiled; the rest are opt-in.
//!
//! - `local-llamacpp`: Enable the llama.cpp subprocess client
//! - `local-vllm`: Enable vLLM client (HTTP-based)
//! - `local-tensorrt`: Enable TensorRT-LLM support
//!
//! # Example - Ollama
//!
//! ```no_run
//! use ironmonitor::agent::local::{OllamaClient, LocalInferenceClient, InferenceRequest};
//!
//! # async fn example() -> Result<(), Box<dyn std::error::Error>> {
//! let client = OllamaClient::new("http://localhost:11434")?;
//!
//! // Check available models
//! let models = client.list_models().await?;
//! println!("Available models: {:?}", models);
//!
//! // Run inference
//! let request = InferenceRequest {
//! model: "llama3".to_string(),
//! prompt: "What is the GPU temperature?".to_string(),
//! system: None,
//! max_tokens: Some(256),
//! temperature: Some(0.7),
//! ..Default::default()
//! };
//! let response = client.generate(request).await?;
//! println!("Response: {}", response.text);
//! # Ok(())
//! # }
//! ```
pub use ;
pub use IronWorksClient;
pub use OllamaClient;
pub use LlamaCppClient;
pub use VllmClient;
pub use TensorRtClient;
use crateResult;
use async_trait;
use ;
/// Common inference request parameters
/// Inference response
/// Model information
/// Common trait for local inference clients