1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
// Obligations of the published library surface, kept here rather than in
// `Cargo.toml`'s lint tables for the reason ADR-018 D1 records: `[lints.*]`
// binds every target in the package — library, tests and benches — and
// conventions about what this crate *exports* must not bind the tests and
// benches that are not the published surface. Conventions that are about the
// Rust written in this repository rather than about what it exports live in
// `Cargo.toml` instead.
// A returned value nobody is obliged to look at is a call that silently does
// nothing; these three say so at the definition rather than in review.
// A panic a caller was not told about is an undocumented contract. The crate
// answers the lock-poisoning half of it once, in the section below, so that
// affected call sites cite one decision instead of restating it independently.
// The Rust API Guidelines that hard constraint 6 adopts require both of these,
// and neither restates the signature: a `# Errors` section says under what
// condition the call fails, which is the part a caller has to act on.
//! Reef is an open, provider-neutral set of Rust agent primitives.
//!
//! An agent, here, is a loop: it sends a model everything the conversation has
//! said so far, records what comes back, runs any tools the model asked for,
//! records those too, and goes round again until something ends it. Reef owns
//! that loop and the two things it is made of — the transcript it reads and
//! extends, and the events it publishes while doing so — and it owns the seams
//! that let you change what happens at every boundary.
//!
//! Reef deliberately does not own a provider, a tool admission policy, a sandbox,
//! a persistence format, a compaction strategy, or a configuration file. Those are
//! yours, and they are written against exactly the public API documented here,
//! with no private access.
//!
//! # Start here
//!
//! [`Agent::start`](runtime::agent::Agent::start) is the entry point. Everything
//! before it is assembly, and it happens in one order:
//!
//! 1. **Tools.** Register each one with
//! [`tool::RegistryBuilder`](runtime::tool::RegistryBuilder) and
//! [`build`](runtime::tool::RegistryBuilder::build) an immutable
//! [`tool::Registry`](runtime::tool::Registry).
//! 2. **Hooks.** Register each one with
//! [`hook::RegistryBuilder`](runtime::hook::RegistryBuilder), then pass that
//! builder directly to [`Config::new`](runtime::agent::Config::new).
//! 3. **Configuration.** [`agent::Config::new`](runtime::agent::Config::new)
//! takes the model client, the tool registry and the hook builder, plus any
//! default request material.
//! 4. **The agent.** [`agent::Agent::new`](runtime::agent::Agent::new) freezes
//! that configuration into an [`agent::Revision`](runtime::agent::Revision).
//! Everything in it stays fixed for as long as the agent runs; changing any of
//! it means building a new agent.
//! 5. **A session.** [`session::Session`](runtime::session::Session) owns one
//! conversation's transcript and admits one Run at a time.
//! 6. **The Run.** [`Agent::start_observed`](runtime::agent::Agent::start_observed)
//! admits the Session, records the caller's input, and atomically returns a
//! [`run::Run`](runtime::run::Run) with its unique event stream. Drain the stream,
//! then [`wait`](runtime::run::Run::wait) for the
//! [`run::Terminal`](runtime::run::Terminal).
//!
//! ```
//! use std::future::{Future, ready};
//! use std::pin::Pin;
//! use std::sync::Arc;
//! use std::task::{Context, Poll};
//!
//! use reef_core::cancel;
//! use reef_core::futures_core::Stream;
//! use reef_core::runtime::{agent, event, model, hook, run, session, tool};
//! use reef_core::value::{content, lifecycle};
//!
//! // 1. A provider. Reef ships none: `model::Client` is the adapter you write
//! // around whatever answers your requests. This one answers offline.
//! #[derive(Debug)]
//! struct Offline;
//!
//! impl model::Client for Offline {
//! type Error = std::convert::Infallible;
//! type Stream = Scripted;
//!
//! fn info(&self) -> model::Info {
//! model::Info::new("offline".to_owned(), 8_192)
//! }
//!
//! fn generate(
//! &self,
//! _request: model::Request,
//! ) -> impl Future<Output = Result<Self::Stream, Self::Error>> + Send {
//! ready(Ok(Scripted(vec![
//! model::Event::Done {
//! finish_reason: "stop".to_owned(),
//! completion: model::CompletionReason::Stop,
//! provider_data: Vec::new(),
//! },
//! // A provider that answers in one frame needs no deltas.
//! model::Event::ContentFinal {
//! id: "answer".to_owned(),
//! kind: model::ContentKind::Text,
//! content: "Hello back.".to_owned(),
//! extensions: content::SegmentExtensions::new(String::new(), None, None),
//! },
//! ])))
//! }
//! }
//!
//! /// `futures-core` has no combinators, so a stream is a `poll_next`.
//! #[derive(Debug)]
//! struct Scripted(Vec<model::Event>);
//!
//! impl Stream for Scripted {
//! type Item = Result<model::Event, std::convert::Infallible>;
//!
//! fn poll_next(mut self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll<Option<Self::Item>> {
//! Poll::Ready(self.0.pop().map(Ok))
//! }
//! }
//!
//! # #[tokio::main(flavor = "current_thread")]
//! # async fn main() {
//! // 2-4. Assemble. No tools and no hooks here; both registries are empty.
//! let tools = tool::RegistryBuilder::new().build().unwrap();
//! let hooks = hook::RegistryBuilder::new();
//! let config = agent::Config::new(Arc::new(Offline), tools, hooks);
//! let agent = agent::Agent::new(config).unwrap();
//!
//! // 5-6. Run one conversation.
//! let session = session::Session::new("readme".to_owned(), None);
//! let (run, mut events) = agent
//! .start_observed(
//! session.clone(),
//! vec![content::Part::text("Hello.".to_owned())],
//! cancel::Token::new(),
//! )
//! .await
//! .unwrap();
//!
//! while let Some(delivery) = events.recv().await {
//! // Three decision points are a rendezvous the Run waits on. The type says
//! // which, so a consumer never has to know the list.
//! if delivery.decision_point().is_some() {
//! delivery.accept().unwrap();
//! }
//! }
//!
//! let terminal = run.wait().await;
//! assert_eq!(terminal.outcome(), lifecycle::Outcome::Completed);
//!
//! // The transcript is the conversation, and it outlives the Run.
//! let history = session.transcript().snapshot().unwrap();
//! let spoken: Vec<String> = history.items().iter().map(|item| item.to_plain_text()).collect();
//! assert_eq!(spoken, ["Hello.", "Hello back."]);
//!
//! agent.close().await.unwrap();
//! # }
//! ```
//!
//! # The pieces
//!
//! - [`runtime::agent`], [`runtime::run`], [`runtime::session`] — the lifecycle:
//! an Agent holds frozen configuration, a Session holds one conversation, and a
//! Run is one execution of the second under the first.
//! - [`runtime::model`] — the provider boundary. Implement
//! [`model::Client`](runtime::model::Client); validate what you emit with
//! [`model::StreamGrammar`](runtime::model::StreamGrammar).
//! - [`runtime::tool`] — what the model may call, and who may authorise it.
//! - [`runtime::hook`] — seven runtime families plus [`AgentTeardown`](runtime::hook::AgentTeardown)
//! that observe or change what the loop does at every boundary it has.
//! - [`runtime::event`] — the outward stream. Single-consumer, and three
//! [decision points](runtime::event::DecisionPoint) are acknowledgements the
//! Run blocks on.
//! - [`transcript`], [`value`] — the records a conversation is made of, and the
//! closed value vocabulary inside them.
//! - [`state`] — typed per-Run and per-Session storage for whatever your hooks
//! need to carry.
//! - [`cancel`], [`audit`], [`opaque`] — cancellation, out-of-band facts, and the
//! uninterpreted byte channel providers travel in.
//!
//! # Serialization
//!
//! The optional `serde` feature — **off by default** — adds `Serialize` and
//! `Deserialize`, as `serde_core` spells them and therefore as `serde` spells
//! them too, to 28 owned value types used by session files and lifecycle records:
//! 27 are reachable from [`transcript::store::Snapshot`],
//! [`value::lifecycle::Coord`], [`audit::Event`] or [`runtime::event::Kind`], and
//! [`value::lifecycle::Outcome`] is the remaining standalone lifecycle value. It
//! adds no format: reef depends
//! on `serde_core` and on no JSON, CBOR or postcard crate, and exposes no
//! `to_json`-shaped convenience. Picking a format, an envelope and a destination
//! is the composition layer's job, and enabling the feature changes nothing
//! observable until that layer writes serialization code.
//!
//! What the format guarantees is that nothing is lost on the way through.
//! Serialized values carry no schema version: compatibility is driven by their
//! content, not by which reef version wrote them. A reader refuses unknown fields
//! and unknown enum tags rather than dropping them; every declared field is
//! present and has no default. Thus `None` and `Some(empty)` survive, and so do
//! all three states of [`MetadataPresence`](value::tool_call::MetadataPresence).
//!
//! One bound is worth knowing before you meet it:
//! [`MetadataValue`](value::content::MetadataValue) nests without a limit in
//! memory but carries a documented maximum nesting depth of 128 in both
//! serialization directions, and reports a typed error past it rather than
//! truncating the value or overflowing the stack.
//!
//! ## What a serialized value contains
//!
//! A serialized `reef-core` value contains model-visible and user-visible
//! content, and [`SegmentExtensions`](value::content::SegmentExtensions)
//! additionally contains a provider-issued credential-tier token. Anything that
//! persists or transmits it inherits the confidentiality of the underlying
//! conversation. `reef-core` provides no encryption, no key management and no
//! redacted serialization mode, and will not: a redacted session file cannot be
//! resumed, so it would be a footgun with no valid use. At-rest and in-transit
//! protection are the responsibility of the layer that chose the destination.
//!
//! This is deliberately the opposite of what [`Debug`](std::fmt::Debug) does.
//! `{:?}` is an incidental rendering reached from inside a log line nobody
//! audited, so it sizes content and never quotes it; serialization is reached
//! only by an explicit call at a site that named its destination, and a
//! transcript that dropped the token authenticating a reasoning block is one the
//! provider will reject. Redaction governs incidental rendering, not content
//! transport.
//!
//! # Lock poisoning
//!
//! Every lock in this crate guards an invariant reef maintains itself, so a
//! poisoned lock always means a panic already unwound out of code that was
//! halfway through maintaining one — reef's own, or a callback that broke a
//! documented contract, such as a [`state::session_state::SnapshotValue`] whose `Clone`
//! panicked. Nothing behind such a lock can be assumed intact, so reef never
//! recovers one by carrying on with whatever it finds there. It fails closed,
//! and which of two ways it uses follows from the signature rather than from the
//! lock:
//!
//! - **Reported as data**, where the operation is already fallible for reasons
//! the caller has to handle anyway. The transcript is the only such state:
//! every [`transcript::store`] read and mutation returns
//! [`Poisoned`](transcript::store::Error::Poisoned) and leaves the stored
//! history untouched, because a caller that cannot read or extend a transcript
//! needs that as an outcome rather than as an unwind.
//! - **Propagated as a panic** everywhere else. A poisoned session, mailbox,
//! run-state, session-state, Agent, or Run lock leaves no recovery a caller
//! could perform, so the observing call panics instead of inventing one. This
//! is the rule ADR-014 already applies to core invariant panics, which stay
//! outside the observer catch boundary so that a defect stays visible.
//!
//! A poisoned lock is therefore never an expected outcome of correct use. No API
//! models it as a variant to match on beyond the transcript's, no argument a
//! caller can pass provokes it, and the honest response to one is to treat the
//! process as holding a defect rather than to retry. Every public function that
//! can propagate one says so in its own `# Panics` section and links back here.
//!
//! One lock is deliberately recovered rather than propagated: the session's
//! transcript-transaction lock guards no data of its own, and the reservation
//! controller resumes a reservation callback's unwind while that guard is still
//! alive, having already resolved every reservation it held and either committed
//! in full or not at all. Its poison is an expected consequence of a panic reef
//! deliberately re-raises, and a successor must still be able to mutate the
//! transcript, so the session recovers it where it takes it. That recovery is
//! private, and it is the only one.
/// The `futures-core` this crate's stream bound is written against.
///
/// [`model::Client::Stream`](runtime::model::Client::Stream) is bounded by
/// [`futures_core::Stream`], which is a *different crate*, and the bound renders
/// as `futures_core::stream::Stream` with nothing to say where that comes from.
/// A provider author had to infer the crate name, add it to their own manifest,
/// and pick a version that unifies with the one reef resolved — and when it did
/// not unify, the error was an unsatisfied trait bound naming two types that
/// print identically.
///
/// Implement the bound through this re-export and the version cannot disagree:
///
/// ```
/// use std::pin::Pin;
/// use std::task::{Context, Poll};
///
/// use reef_core::futures_core::Stream;
///
/// struct Empty;
///
/// impl Stream for Empty {
/// type Item = ();
///
/// fn poll_next(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll<Option<Self::Item>> {
/// Poll::Ready(None)
/// }
/// }
/// ```
///
/// This adds nothing to reef's dependency budget: `futures-core` is already one
/// of the three crates the default build depends on, and the bound already made
/// it public in everything but name.
///
/// `futures-core` deliberately carries no combinators — no `stream::iter`, no
/// `StreamExt` — so implementing [`Stream`](futures_core::Stream) by hand is the
/// whole of it. [`model::Client`](runtime::model::Client) carries a worked
/// example of the `poll_next` a scripted provider needs.
pub use futures_core;
// This module also carries the crate's `README.md` doctest hook; the comment
// there says why it is not in this file.
pub
pub
// Declared here rather than from `redact.rs` so that the test target which
// compiles that file through `#[path]` — `tests/miri_core.rs` — does not pull
// these tests into the Miri baseline `scripts/miri-tests.txt` pins. This is the
// same reason `state/mod.rs` declares its two sibling test modules.
pub