ramvamp_server/toolcall.rs
1//! Recovering structured `tool_calls` from the reply text the model wrote.
2//!
3//! The template tells the model to answer with
4//!
5//! ```text
6//! <tool_call>
7//! {"name": <function-name>, "arguments": <args-json-object>}
8//! </tool_call>
9//! ```
10//!
11//! which arrives here as ordinary generated text. A client cannot act on that;
12//! it acts on `choices[0].message.tool_calls`. This module is the translation,
13//! and it is pure: text in, [`Extracted`] out, no clock, no randomness, no I/O.
14//! That is what lets every rule below be pinned by a unit test.
15//!
16//! # Finding the block boundaries
17//!
18//! The obvious implementation — `find("<tool_call>")`, then
19//! `find("</tool_call>")` — has a bug that shows up the first time a model
20//! quotes the protocol back at you:
21//!
22//! ```text
23//! <tool_call>
24//! {"name": "reply", "arguments": {"text": "wrap it in </tool_call> tags"}}
25//! </tool_call>
26//! ```
27//!
28//! The naive scan closes the block inside the JSON string and produces a
29//! truncated, unparseable body. [`find_close`] therefore walks the bytes
30//! tracking whether it is inside a JSON string literal (honouring `\`
31//! escapes) and only accepts a `</tool_call>` found *outside* one.
32//!
33//! That tracking is itself a risk: a body that is not JSON at all can carry an
34//! unbalanced `"`, which would make the scan believe the rest of the reply is
35//! one long string and lose a terminator that is plainly there. So the
36//! string-aware scan falls back to a plain search when it reaches the end
37//! without a match. The fallback means the careful scan can only ever find
38//! *more* blocks than the naive one, never fewer.
39//!
40//! Locating `name` and `arguments` inside the body uses the same idea one
41//! level down: a member's value is a *span*, delimited by matching braces or
42//! brackets with strings skipped, so `{"a": {"b": 1}}` and `{"a": "} }"}` both
43//! end where they should. A naive `find('}')` ends both in the wrong place.
44//!
45//! # Bytes are preserved, never re-serialized
46//!
47//! [`ParsedCall::arguments`] is the raw substring the model emitted. It is not
48//! parsed into a `serde_json::Value` and printed back, for two reasons. The
49//! client echoes `arguments` verbatim on the next turn and
50//! [`crate::prompt`] splices that string straight into the prompt as
51//! `ToolArguments::Raw`, so preserving the bytes makes the round trip exact —
52//! the model sees on turn *n+1* the characters it wrote on turn *n*.
53//! Re-serializing would also quietly normalise key order and spacing, and
54//! would turn the everyday case of a model emitting slightly invalid JSON into
55//! a hard failure.
56//!
57//! # Malformed blocks are recovered, not dropped
58//!
59//! Models get this JSON wrong often enough that dropping a malformed block
60//! would be a routine loss of the turn. If a `name` can be recovered the call
61//! is emitted with whatever the `arguments` span held, raw and unchecked;
62//! `FunctionCall::arguments` is a `String` precisely so the client can be
63//! handed exactly what the model said and fail on its own terms. Only a block
64//! with no recoverable name falls back to being ordinary content — and then
65//! the whole block, markers included, is preserved there, so nothing the model
66//! wrote is ever discarded.
67//!
68//! An unterminated `<tool_call>` is content for the same reason: it is what a
69//! reply that ran out of `max_tokens` mid-block looks like, and no call can be
70//! honestly claimed from half a block.
71//!
72//! # Streaming, where the whole reply is not in hand
73//!
74//! [`extract`] needs the finished text. A stream does not have it: the decision
75//! to send a token as content has to be made before the token after it exists,
76//! and once a `<` has gone out as content it cannot be recalled. Scanning the
77//! text as it accumulates would also mean handling a marker split across two
78//! deltas, since nothing about a byte stream promises `</tool_call>` arrives
79//! whole.
80//!
81//! It does arrive whole, and that is the mechanism [`CallStream`] is built on.
82//! Both markers are *added tokens* in the Qwen3 tokenizer —
83//! [`OPEN_TAG_ID`] and [`CLOSE_TAG_ID`] — so each is a single id, and the
84//! decoder does not skip them. Detection is therefore an integer compare
85//! against the id the model sampled, made at the one layer that still has the
86//! id, and a marker cannot be split because a token cannot be. What is left is
87//! bookkeeping: buffer between the two ids, hand the body to the same
88//! [`parse_block`] the buffered path uses, and give the call the same id
89//! [`wire_call`] would have given it there.
90
91use crate::request::{FunctionCall, ToolCall};
92use crate::response::COMPLETION_ID_PREFIX;
93
94/// The marker that opens a call block.
95pub const OPEN_TAG: &str = "<tool_call>";
96
97/// The marker that closes one.
98pub const CLOSE_TAG: &str = "</tool_call>";
99
100/// The single token id [`OPEN_TAG`] encodes to.
101///
102/// An `added_tokens` entry of the Qwen3 tokenizer, `special: false`, so the
103/// streaming decoder emits it as one token carrying the literal text. Pinned as
104/// a constant rather than looked up because the streaming sink has no
105/// tokenizer: it sees ids and text, and this crate is meant to be testable
106/// without an install on disk.
107pub const OPEN_TAG_ID: u32 = 151_657;
108
109/// The single token id [`CLOSE_TAG`] encodes to. See [`OPEN_TAG_ID`].
110pub const CLOSE_TAG_ID: u32 = 151_658;
111
112/// The `type` of every call: OpenAI has defined exactly one.
113pub const CALL_KIND: &str = "function";
114
115/// The prefix of a generated call id.
116pub const CALL_ID_PREFIX: &str = "call_";
117
118/// What a block that named no arguments contributes.
119///
120/// The empty object rather than the empty string: a client's first move is
121/// `json.loads(arguments)`, and `""` breaks it where `{}` is the truth — a
122/// call with no arguments.
123const NO_ARGUMENTS: &str = "{}";
124
125/// One call recovered from a reply.
126#[derive(Debug, Clone, PartialEq, Eq)]
127pub struct ParsedCall {
128 /// The function name, unescaped.
129 pub name: String,
130 /// The arguments, as the exact bytes the model emitted.
131 pub arguments: String,
132}
133
134/// A reply split into what the client should show and what it should run.
135#[derive(Debug, Clone, Default, PartialEq, Eq)]
136pub struct Extracted {
137 /// Everything outside the blocks that became calls.
138 ///
139 /// When no call was extracted this is the reply unchanged, byte for byte;
140 /// a plain answer must not be altered by a parser that found nothing to
141 /// do. When calls *were* extracted the surrounding prose is joined and
142 /// trimmed at the ends, since the whitespace that framed a removed block
143 /// is framing, not text.
144 pub content: String,
145 /// The calls, in the order the model asked for them.
146 pub calls: Vec<ParsedCall>,
147}
148
149impl Extracted {
150 /// The reply as pure content, with nothing parsed out of it.
151 ///
152 /// For a request that declared no tools. The markers only appear because
153 /// the template's tools branch instructed the model to produce them, so
154 /// without tools they are ordinary text: a conversation *about* tool
155 /// calling would otherwise have its example swallowed into a structured
156 /// call the caller never enabled.
157 pub fn none(reply: &str) -> Self {
158 Extracted {
159 content: reply.to_owned(),
160 calls: Vec::new(),
161 }
162 }
163}
164
165/// Split `reply` into content and the calls it asked for.
166///
167/// Never fails and never panics: every malformed shape has a defined outcome,
168/// documented on [the module](self).
169pub fn extract(reply: &str) -> Extracted {
170 let mut content = String::new();
171 let mut calls = Vec::new();
172 let mut cursor = 0;
173
174 while let Some(open) = reply.get(cursor..).and_then(|rest| rest.find(OPEN_TAG)) {
175 let open = cursor + open;
176 let body_start = open + OPEN_TAG.len();
177 // No terminator: this block never finished, so it is text. Leaving
178 // `cursor` where it is puts the opener and everything after it into
179 // content below.
180 let Some(close) = find_close(reply, body_start) else {
181 break;
182 };
183 let body = reply.get(body_start..close).unwrap_or_default();
184 match parse_block(body) {
185 Some(call) => {
186 content.push_str(reply.get(cursor..open).unwrap_or_default());
187 calls.push(call);
188 }
189 // Not a call after all: keep the block verbatim, markers included.
190 None => content.push_str(
191 reply
192 .get(cursor..close + CLOSE_TAG.len())
193 .unwrap_or_default(),
194 ),
195 }
196 cursor = close + CLOSE_TAG.len();
197 }
198 content.push_str(reply.get(cursor..).unwrap_or_default());
199
200 if !calls.is_empty() {
201 content = content.trim().to_owned();
202 }
203 Extracted { content, calls }
204}
205
206/// The id a call is given, derived from the completion's own id.
207///
208/// Deterministic on purpose. The client sends this back as `tool_call_id`, so
209/// it has to be stable within a response, and a test that cannot predict an id
210/// cannot pin the round trip — the same reason `id` and `created` are
211/// parameters everywhere else in this crate rather than reads of a clock.
212/// Uniqueness comes from the completion id, which already cannot repeat.
213pub fn call_id(completion_id: &str, index: usize) -> String {
214 let suffix = completion_id
215 .strip_prefix(COMPLETION_ID_PREFIX)
216 .unwrap_or(completion_id);
217 format!("{CALL_ID_PREFIX}{suffix}_{index}")
218}
219
220/// Give one parsed call its wire shape and its id, at `index`.
221///
222/// The single place a [`ParsedCall`] becomes a [`ToolCall`], which is what
223/// makes the buffered and streaming paths agree by construction rather than by
224/// two implementations that happen to match today: the buffered path indexes a
225/// finished list, the streaming one counts calls as they close, and both arrive
226/// here.
227pub fn wire_call(completion_id: &str, index: usize, call: &ParsedCall) -> ToolCall {
228 ToolCall {
229 id: call_id(completion_id, index),
230 kind: CALL_KIND.to_owned(),
231 function: FunctionCall {
232 name: call.name.clone(),
233 arguments: call.arguments.clone(),
234 },
235 }
236}
237
238/// Give the parsed calls their wire shape and their ids.
239pub fn wire_calls(completion_id: &str, calls: &[ParsedCall]) -> Vec<ToolCall> {
240 calls
241 .iter()
242 .enumerate()
243 .map(|(index, call)| wire_call(completion_id, index, call))
244 .collect()
245}
246
247/// What the sink should do with the token it just fed to a [`CallStream`].
248#[derive(Debug, Clone, PartialEq, Eq)]
249pub enum Step {
250 /// Send nothing: the token is inside a block that has not closed.
251 Nothing,
252 /// Send this as a content delta. Not always the token's own text — a block
253 /// that closed without naming a function comes back here whole, markers
254 /// included, exactly as the buffered path keeps it.
255 Content(String),
256 /// A block closed and named a function.
257 Call {
258 /// Its position in this response's `tool_calls`, which is the `index`
259 /// the delta must carry: it is how a client knows which call a
260 /// fragment belongs to.
261 index: usize,
262 /// The call.
263 call: ParsedCall,
264 },
265}
266
267/// [`extract`]'s rules, applied one token at a time.
268///
269/// Fed every generated token in order, it answers what to put on the wire for
270/// it. The outcome matches what [`extract`] would have produced from the
271/// concatenated reply, with one unavoidable difference: [`extract`] trims the
272/// prose that framed a removed block, and a stream has already sent that
273/// whitespace by the time the block it framed appears.
274///
275/// Pure and clock-free like the rest of the module, so the state machine is
276/// pinned by unit tests rather than by a live model.
277///
278/// The other difference is [`find_close`]'s string awareness, which has no
279/// streaming equivalent and does not need one. A `</tool_call>` written *inside*
280/// an argument string reaches [`extract`] as twelve characters of a JSON
281/// literal; it reaches here as [`CLOSE_TAG_ID`], the id the model chose to
282/// sample, and an id is not text. Closing there is the honest reading of what
283/// the model emitted, and it is also the only reading available: the fallback
284/// that makes the buffered scan safe — search the *rest of the reply* — is a
285/// rest that does not exist yet.
286#[derive(Debug, Clone)]
287pub struct CallStream {
288 /// Whether calls are recovered at all.
289 ///
290 /// False for a request that declared no tools, and then every token is
291 /// content. The model only writes these markers because the tools branch of
292 /// the template told it to, so without tools they are ordinary text a user
293 /// asked for — and swallowing them would delete the answer. The same gate
294 /// the buffered path applies, for the same reason.
295 enabled: bool,
296 /// The body of the block currently open, markers excluded, or `None`
297 /// outside one.
298 open: Option<String>,
299 /// Calls emitted so far, which is both the next `index` and the answer to
300 /// "was this turn a tool call".
301 emitted: usize,
302}
303
304impl CallStream {
305 /// A stream that recovers calls, or one that passes everything through.
306 pub fn new(enabled: bool) -> Self {
307 CallStream {
308 enabled,
309 open: None,
310 emitted: 0,
311 }
312 }
313
314 /// How many calls have been emitted.
315 ///
316 /// Non-zero is what makes `finish_reason` `tool_calls`: the model stopped
317 /// because it wanted a tool, whatever token ended the reply.
318 pub fn emitted(&self) -> usize {
319 self.emitted
320 }
321
322 /// Feed one generated token and its id.
323 pub fn push(&mut self, id: u32, text: &str) -> Step {
324 if !self.enabled {
325 return Step::Content(text.to_owned());
326 }
327 match id {
328 // A second opener inside an open block is body text, matching
329 // `extract`, which looks for the terminator and not for another
330 // opener.
331 OPEN_TAG_ID if self.open.is_none() => {
332 self.open = Some(String::new());
333 Step::Nothing
334 }
335 // A terminator with nothing open closes nothing; it is the stray
336 // marker case, and it is content.
337 CLOSE_TAG_ID if self.open.is_some() => self.close(),
338 _ => match &mut self.open {
339 Some(body) => {
340 body.push_str(text);
341 Step::Nothing
342 }
343 None => Step::Content(text.to_owned()),
344 },
345 }
346 }
347
348 /// Generation ended: give back the text of a block that never closed.
349 ///
350 /// Half a block is not a call — it is what a reply that ran out of
351 /// `max_tokens` mid-block looks like — but it *is* text the model wrote,
352 /// and nothing the model wrote is dropped. Idempotent.
353 pub fn flush(&mut self) -> Option<String> {
354 self.open.take().map(|body| format!("{OPEN_TAG}{body}"))
355 }
356
357 /// Resolve the block that just closed.
358 fn close(&mut self) -> Step {
359 let body = self.open.take().unwrap_or_default();
360 match parse_block(&body) {
361 Some(call) => {
362 let index = self.emitted;
363 self.emitted += 1;
364 Step::Call { index, call }
365 }
366 // No recoverable name, so it was never a call: the block survives
367 // as text, markers and all.
368 None => Step::Content(format!("{OPEN_TAG}{body}{CLOSE_TAG}")),
369 }
370 }
371}
372
373/// Find the `</tool_call>` that closes the block whose body starts at `from`.
374///
375/// String-aware, then naive: see [the module](self) for why it is both.
376fn find_close(reply: &str, from: usize) -> Option<usize> {
377 let bytes = reply.as_bytes();
378 let mut i = from;
379 while i < bytes.len() {
380 match bytes[i] {
381 // A string literal is skipped whole, so a `</tool_call>` written
382 // *inside* one closes nothing.
383 b'"' => i = end_of_string(bytes, i),
384 b'<' if bytes[i..].starts_with(CLOSE_TAG.as_bytes()) => return Some(i),
385 _ => i += 1,
386 }
387 }
388 // An unbalanced quote in a body that was never JSON must not hide a
389 // terminator that is right there in the text.
390 reply
391 .get(from..)
392 .and_then(|rest| rest.find(CLOSE_TAG))
393 .map(|at| from + at)
394}
395
396/// Recover a call from one block body, or decide it is not a call.
397fn parse_block(body: &str) -> Option<ParsedCall> {
398 let open = body.find('{')?;
399 let mut name = None;
400 let mut arguments = None;
401 for (key, value) in members(body, open) {
402 match key.as_str() {
403 // First wins: a duplicated key is malformed, and the first is what
404 // a JSON reader that stops at the first match would have seen.
405 "name" if name.is_none() => name = Some(value),
406 "arguments" if arguments.is_none() => arguments = Some(value),
407 _ => {}
408 }
409 }
410 let name = unquote(name?);
411 if name.is_empty() {
412 return None;
413 }
414 Some(ParsedCall {
415 name,
416 arguments: arguments.map_or_else(|| NO_ARGUMENTS.to_owned(), str::to_owned),
417 })
418}
419
420/// The top-level members of the object starting at `open`, as (key, raw value).
421///
422/// Deliberately not a JSON parser: it recovers what it can and stops at the
423/// first thing it does not understand, keeping every member it read up to
424/// there. A strict parser would return nothing for a body with a trailing
425/// comma, which is exactly the body this has to survive.
426fn members(text: &str, open: usize) -> Vec<(String, &str)> {
427 let bytes = text.as_bytes();
428 let mut out = Vec::new();
429 let mut i = open + 1;
430 loop {
431 i = skip_space(bytes, i);
432 // A key is quoted; anything else ends the walk, `}` included.
433 if bytes.get(i) != Some(&b'"') {
434 return out;
435 }
436 let key_end = end_of_string(bytes, i);
437 let key = unquote(text.get(i..key_end).unwrap_or_default());
438 i = skip_space(bytes, key_end);
439 if bytes.get(i) != Some(&b':') {
440 return out;
441 }
442 i = skip_space(bytes, i + 1);
443 if i >= bytes.len() {
444 return out;
445 }
446 let value_end = end_of_value(bytes, i);
447 out.push((key, text.get(i..value_end).unwrap_or_default()));
448 i = skip_space(bytes, value_end);
449 if bytes.get(i) != Some(&b',') {
450 return out;
451 }
452 i += 1;
453 }
454}
455
456/// The index just past the value starting at `start`.
457fn end_of_value(bytes: &[u8], start: usize) -> usize {
458 match bytes.get(start) {
459 Some(b'"') => end_of_string(bytes, start),
460 Some(b'{') => end_of_nested(bytes, start, b'{', b'}'),
461 Some(b'[') => end_of_nested(bytes, start, b'[', b']'),
462 // A bare token: a number, `true`, `false`, `null`, or whatever the
463 // model put there instead. It ends at the first delimiter.
464 _ => {
465 let mut i = start;
466 while i < bytes.len() && !matches!(bytes[i], b',' | b'}' | b']') && !is_space(bytes[i])
467 {
468 i += 1;
469 }
470 i
471 }
472 }
473}
474
475/// The index just past the balanced `open`/`close` pair starting at `start`.
476///
477/// Strings are skipped whole, so a brace inside one does not count — the
478/// difference between reading `{"path": "a}b"}` and reading half of it.
479/// An unbalanced value ends at the end of the input, which keeps a truncated
480/// block's arguments recoverable instead of empty.
481fn end_of_nested(bytes: &[u8], start: usize, open: u8, close: u8) -> usize {
482 let mut depth = 0usize;
483 let mut i = start;
484 while i < bytes.len() {
485 let byte = bytes[i];
486 if byte == b'"' {
487 i = end_of_string(bytes, i);
488 continue;
489 }
490 if byte == open {
491 depth += 1;
492 } else if byte == close {
493 depth = depth.saturating_sub(1);
494 if depth == 0 {
495 return i + 1;
496 }
497 }
498 i += 1;
499 }
500 bytes.len()
501}
502
503/// The index just past the string literal starting at `start`, which must be
504/// its opening quote. An unterminated literal ends at the end of the input.
505fn end_of_string(bytes: &[u8], start: usize) -> usize {
506 let mut i = start + 1;
507 while i < bytes.len() {
508 match bytes[i] {
509 // A `\` consumes the next byte, so `\"` does not end the string.
510 // Landing mid-character is harmless: this scan only ever compares
511 // against ASCII, and every index it hands out is a boundary.
512 b'\\' => i += 2,
513 b'"' => return i + 1,
514 _ => i += 1,
515 }
516 }
517 bytes.len()
518}
519
520/// JSON whitespace.
521fn is_space(byte: u8) -> bool {
522 matches!(byte, b' ' | b'\t' | b'\n' | b'\r')
523}
524
525fn skip_space(bytes: &[u8], mut i: usize) -> usize {
526 while i < bytes.len() && is_space(bytes[i]) {
527 i += 1;
528 }
529 i
530}
531
532/// The text of a quoted span, with JSON escapes resolved where they can be.
533///
534/// `serde_json` does the unescaping when the span is a well-formed literal, so
535/// a `r` in a name is honoured. When it is not — an unterminated or badly
536/// escaped literal — the quotes come off by hand rather than the value being
537/// lost, which is the module's rule everywhere: recover, do not drop.
538fn unquote(raw: &str) -> String {
539 if let Ok(text) = serde_json::from_str::<String>(raw) {
540 return text;
541 }
542 raw.strip_prefix('"')
543 .map_or(raw, |rest| rest.strip_suffix('"').unwrap_or(rest))
544 .to_owned()
545}
546
547#[cfg(test)]
548mod tests {
549 use super::*;
550
551 /// The block the template asks for, around whatever `arguments` says.
552 fn block(arguments: &str) -> String {
553 format!("<tool_call>\n{{\"name\": \"read\", \"arguments\": {arguments}}}\n</tool_call>")
554 }
555
556 #[test]
557 fn a_reply_with_no_calls_is_returned_byte_for_byte() {
558 for reply in [
559 "",
560 "4",
561 " Sure — the answer is 4.\n\n",
562 "a < b and c > d",
563 "here is a tag: <tool>",
564 ] {
565 let out = extract(reply);
566 assert!(out.calls.is_empty(), "{reply:?} produced {:?}", out.calls);
567 assert_eq!(out.content, reply, "a plain reply must not be rewritten");
568 }
569 }
570
571 #[test]
572 fn one_call_becomes_one_parsed_call_and_empty_content() {
573 let out = extract(&block(r#"{"path": "a.txt"}"#));
574 assert_eq!(out.content, "");
575 assert_eq!(
576 out.calls,
577 vec![ParsedCall {
578 name: "read".to_owned(),
579 arguments: r#"{"path": "a.txt"}"#.to_owned(),
580 }]
581 );
582 }
583
584 #[test]
585 fn two_calls_in_one_turn_are_both_extracted_in_order() {
586 let reply = format!(
587 "{}\n{}",
588 block(r#"{"path": "a"}"#),
589 block(r#"{"path": "b"}"#)
590 );
591 let out = extract(&reply);
592 assert_eq!(out.calls.len(), 2);
593 assert_eq!(out.calls[0].arguments, r#"{"path": "a"}"#);
594 assert_eq!(out.calls[1].arguments, r#"{"path": "b"}"#);
595 assert_eq!(out.content, "");
596 }
597
598 #[test]
599 fn prose_before_between_and_after_survives() {
600 let reply = format!(
601 "I will read both.\n{}\nand the other one\n{}\nThen I will compare them.",
602 block(r#"{"path": "a"}"#),
603 block(r#"{"path": "b"}"#)
604 );
605 let out = extract(&reply);
606 assert_eq!(out.calls.len(), 2);
607 assert_eq!(
608 out.content,
609 "I will read both.\n\nand the other one\n\nThen I will compare them."
610 );
611 }
612
613 #[test]
614 fn an_empty_arguments_object_is_preserved_as_written() {
615 let out = extract(&block("{}"));
616 assert_eq!(out.calls[0].arguments, "{}");
617 }
618
619 /// The absent-`arguments` case is the one place a value is invented, and
620 /// `{}` is invented rather than `""` because the client parses it.
621 #[test]
622 fn a_block_with_no_arguments_key_gets_an_empty_object() {
623 let out = extract("<tool_call>\n{\"name\": \"now\"}\n</tool_call>");
624 assert_eq!(
625 out.calls,
626 vec![ParsedCall {
627 name: "now".to_owned(),
628 arguments: "{}".to_owned(),
629 }]
630 );
631 }
632
633 /// Models emit invalid JSON here routinely. A trailing comma must cost the
634 /// client a parse error it can see, not a silently dropped turn.
635 #[test]
636 fn malformed_json_with_a_recoverable_name_still_becomes_a_call() {
637 let out = extract(&block(r#"{"path": "a.txt",}"#));
638 assert_eq!(out.calls[0].name, "read");
639 assert_eq!(
640 out.calls[0].arguments, r#"{"path": "a.txt",}"#,
641 "the model's bytes reach the client unaltered, invalid or not"
642 );
643 assert_eq!(out.content, "");
644 }
645
646 /// A block truncated inside `arguments` still names a function, so it is
647 /// still a call — with everything that was written of its arguments.
648 #[test]
649 fn a_block_missing_its_closing_brace_keeps_what_arguments_had() {
650 let out = extract(
651 "<tool_call>\n{\"name\": \"read\", \"arguments\": {\"path\": \"a\"\n</tool_call>",
652 );
653 assert_eq!(out.calls[0].name, "read");
654 assert_eq!(out.calls[0].arguments, "{\"path\": \"a\"\n");
655 }
656
657 #[test]
658 fn a_block_with_no_recoverable_name_is_content_markers_and_all() {
659 for body in [
660 "{\"arguments\": {\"path\": \"a\"}}",
661 "{\"name\": \"\"}",
662 "just some words",
663 "",
664 ] {
665 let reply = format!("<tool_call>{body}</tool_call>");
666 let out = extract(&reply);
667 assert!(out.calls.is_empty(), "{body:?} became a call");
668 assert_eq!(out.content, reply, "the block must survive as text");
669 }
670 }
671
672 /// What a reply that hit `max_tokens` mid-block looks like.
673 #[test]
674 fn an_unterminated_block_is_content() {
675 let reply = "Reading it.\n<tool_call>\n{\"name\": \"read\", \"argum";
676 let out = extract(reply);
677 assert!(out.calls.is_empty());
678 assert_eq!(out.content, reply);
679 }
680
681 #[test]
682 fn a_stray_closing_marker_is_content() {
683 let reply = "the tag is </tool_call>, as shown";
684 let out = extract(reply);
685 assert!(out.calls.is_empty());
686 assert_eq!(out.content, reply);
687 }
688
689 /// A naive `find('}')` ends the arguments at the inner object.
690 #[test]
691 fn nested_braces_inside_arguments_are_balanced_not_guessed() {
692 let arguments =
693 r#"{"filter": {"path": "a.txt", "range": {"from": 1, "to": 9}}, "raw": true}"#;
694 let out = extract(&block(arguments));
695 assert_eq!(out.calls[0].arguments, arguments);
696 }
697
698 /// A brace inside a string is text, not structure.
699 #[test]
700 fn a_brace_inside_a_string_argument_does_not_end_the_value() {
701 let arguments = r#"{"pattern": "^\\{.*\\}$", "note": "a } on its own"}"#;
702 let out = extract(&block(arguments));
703 assert_eq!(out.calls[0].arguments, arguments);
704 }
705
706 /// The bug this parser exists to avoid: a terminator quoted inside the
707 /// arguments must not close the block.
708 #[test]
709 fn a_closing_marker_inside_a_string_argument_does_not_close_the_block() {
710 let arguments = r#"{"text": "wrap it in </tool_call> tags"}"#;
711 let out = extract(&block(arguments));
712 assert_eq!(out.calls.len(), 1, "the block closed too early");
713 assert_eq!(out.calls[0].arguments, arguments);
714 assert_eq!(out.content, "");
715 }
716
717 /// The same trap one escape deeper: the quote before the marker is
718 /// escaped, so the string has not ended there.
719 #[test]
720 fn an_escaped_quote_does_not_end_the_string_scan() {
721 let arguments = r#"{"text": "he said \"</tool_call>\" out loud"}"#;
722 let out = extract(&block(arguments));
723 assert_eq!(out.calls.len(), 1);
724 assert_eq!(out.calls[0].arguments, arguments);
725 }
726
727 /// And the reason the string-aware scan has a fallback: an unbalanced
728 /// quote in a body that was never JSON must not swallow the rest of the
729 /// reply.
730 #[test]
731 fn an_unbalanced_quote_does_not_hide_the_terminator() {
732 let reply =
733 "<tool_call>\n{\"name\": \"read\", \"arguments\": {\"path: \"a}\n</tool_call>\nDone.";
734 let out = extract(reply);
735 assert_eq!(out.calls.len(), 1, "the terminator was lost");
736 assert_eq!(out.calls[0].name, "read");
737 assert_eq!(out.content, "Done.");
738 }
739
740 #[test]
741 fn an_escaped_name_is_unescaped_and_its_arguments_are_not() {
742 let reply = r#"<tool_call>{"name": "read", "arguments": {"q": "a\nb"}}</tool_call>"#;
743 let out = extract(reply);
744 assert_eq!(out.calls[0].name, "read");
745 assert_eq!(
746 out.calls[0].arguments, r#"{"q": "a\nb"}"#,
747 "arguments are bytes, not a decoded value"
748 );
749 }
750
751 /// Multibyte text anywhere near the scanner must not split a character.
752 #[test]
753 fn non_ascii_text_around_and_inside_a_block_is_safe() {
754 let reply = format!(
755 "Je vais lire → {}\n…voilà",
756 block(r#"{"path": "café.txt", "note": "naïve"}"#)
757 );
758 let out = extract(&reply);
759 assert_eq!(
760 out.calls[0].arguments,
761 r#"{"path": "café.txt", "note": "naïve"}"#
762 );
763 // Only the ends are trimmed: the space that preceded the block is
764 // prose the model wrote between two words.
765 assert_eq!(out.content, "Je vais lire → \n…voilà");
766 }
767
768 /// The whitespace and key order the model chose reach the client, because
769 /// the client sends them back and they are re-rendered into the prompt.
770 #[test]
771 fn argument_bytes_are_preserved_exactly() {
772 let arguments = "{ \"b\":2,\n \"a\" : 1 }";
773 let out = extract(&block(arguments));
774 assert_eq!(out.calls[0].arguments, arguments);
775 }
776
777 #[test]
778 fn ids_are_derived_from_the_completion_id_and_the_index() {
779 let calls = vec![
780 ParsedCall {
781 name: "read".to_owned(),
782 arguments: "{}".to_owned(),
783 },
784 ParsedCall {
785 name: "write".to_owned(),
786 arguments: r#"{"path":"a"}"#.to_owned(),
787 },
788 ];
789 let wire = wire_calls("chatcmpl-18f3a0001", &calls);
790 assert_eq!(wire[0].id, "call_18f3a0001_0");
791 assert_eq!(wire[1].id, "call_18f3a0001_1");
792 assert!(wire.iter().all(|call| call.kind == "function"));
793 assert_eq!(wire[1].function.name, "write");
794 assert_eq!(wire[1].function.arguments, r#"{"path":"a"}"#);
795 // Same input, same ids: the round trip is assertable.
796 assert_eq!(wire_calls("chatcmpl-18f3a0001", &calls), wire);
797 // An id without the usual prefix still yields a usable one.
798 assert_eq!(call_id("x", 3), "call_x_3");
799 }
800
801 // ---- the streaming half ----
802
803 /// The id of an ordinary token. Any value that is not a marker will do:
804 /// the state machine only ever compares against the two markers.
805 const TEXT: u32 = 7;
806
807 /// Feed a token sequence through a [`CallStream`] and collect what it would
808 /// have put on the wire: the concatenated content and the calls with their
809 /// indices, including whatever the final flush produced.
810 fn drive(enabled: bool, tokens: &[(u32, &str)]) -> (String, Vec<(usize, ParsedCall)>) {
811 let mut stream = CallStream::new(enabled);
812 let mut content = String::new();
813 let mut calls = Vec::new();
814 for (id, text) in tokens {
815 match stream.push(*id, text) {
816 Step::Nothing => {}
817 Step::Content(text) => content.push_str(&text),
818 Step::Call { index, call } => calls.push((index, call)),
819 }
820 }
821 if let Some(rest) = stream.flush() {
822 content.push_str(&rest);
823 }
824 // Flushing twice must not duplicate the tail.
825 assert_eq!(stream.flush(), None, "flush is not idempotent");
826 assert_eq!(stream.emitted(), calls.len());
827 (content, calls)
828 }
829
830 /// The markers as the tokens they actually are, around one body token.
831 fn streamed(body: &str) -> Vec<(u32, &str)> {
832 vec![
833 (OPEN_TAG_ID, OPEN_TAG),
834 (TEXT, body),
835 (CLOSE_TAG_ID, CLOSE_TAG),
836 ]
837 }
838
839 /// The property the streaming path exists to preserve: the same reply, one
840 /// token at a time, recovers the same call the buffered path recovers.
841 #[test]
842 fn a_streamed_block_recovers_what_the_buffered_path_recovers() {
843 let body = "\n{\"name\": \"glob\", \"arguments\": {\"pattern\": \"*/.rs\"}}\n";
844 let (content, calls) = drive(true, &streamed(body));
845 assert_eq!(content, "");
846 assert_eq!(calls.len(), 1);
847 assert_eq!(calls[0].0, 0);
848 assert_eq!(
849 calls[0].1,
850 extract(&format!("{OPEN_TAG}{body}{CLOSE_TAG}")).calls[0]
851 );
852 assert_eq!(calls[0].1.name, "glob");
853 assert_eq!(calls[0].1.arguments, r#"{"pattern": "*/.rs"}"#);
854 }
855
856 /// A body arrives as many tokens, because a body is many tokens. Only the
857 /// markers are single ids.
858 #[test]
859 fn a_body_split_across_tokens_is_reassembled_whole() {
860 let mut tokens = vec![(OPEN_TAG_ID, OPEN_TAG)];
861 for piece in [
862 "\n{\"na",
863 "me\": \"re",
864 "ad\", \"argu",
865 "ments\": {\"path\": \"a\"}}\n",
866 ] {
867 tokens.push((TEXT, piece));
868 }
869 tokens.push((CLOSE_TAG_ID, CLOSE_TAG));
870 let (content, calls) = drive(true, &tokens);
871 assert_eq!(content, "");
872 assert_eq!(calls[0].1.name, "read");
873 assert_eq!(calls[0].1.arguments, r#"{"path": "a"}"#);
874 }
875
876 #[test]
877 fn prose_before_between_and_after_streams_as_content() {
878 let mut tokens = vec![(TEXT, "I will "), (TEXT, "read both.\n")];
879 tokens.extend(streamed(r#"{"name": "read", "arguments": {"path": "a"}}"#));
880 tokens.push((TEXT, "\nand the other\n"));
881 tokens.extend(streamed(r#"{"name": "read", "arguments": {"path": "b"}}"#));
882 tokens.push((TEXT, "\nDone."));
883
884 let (content, calls) = drive(true, &tokens);
885 assert_eq!(content, "I will read both.\n\nand the other\n\nDone.");
886 assert_eq!(calls.len(), 2);
887 // The index is how a client tells two calls apart, so it counts.
888 assert_eq!(calls[0].0, 0);
889 assert_eq!(calls[1].0, 1);
890 assert_eq!(calls[0].1.arguments, r#"{"path": "a"}"#);
891 assert_eq!(calls[1].1.arguments, r#"{"path": "b"}"#);
892 }
893
894 /// What a reply that hit `max_tokens` mid-block looks like: no call, and
895 /// not one byte of what the model wrote is lost.
896 #[test]
897 fn an_unterminated_streamed_block_flushes_as_content() {
898 let tokens = [
899 (TEXT, "Reading it.\n"),
900 (OPEN_TAG_ID, OPEN_TAG),
901 (TEXT, "\n{\"name\": \"read\", \"argum"),
902 ];
903 let (content, calls) = drive(true, &tokens);
904 assert!(calls.is_empty());
905 assert_eq!(
906 content,
907 "Reading it.\n<tool_call>\n{\"name\": \"read\", \"argum"
908 );
909 }
910
911 /// Without tools the markers are ordinary text: the request never enabled
912 /// calling, so swallowing them would delete the user's answer.
913 #[test]
914 fn a_streamed_block_without_tools_is_ordinary_text() {
915 let body = r#"{"name": "read", "arguments": {}}"#;
916 let (content, calls) = drive(false, &streamed(body));
917 assert!(calls.is_empty());
918 assert_eq!(content, format!("{OPEN_TAG}{body}{CLOSE_TAG}"));
919 }
920
921 /// A closed block that names no function was never a call, and the whole
922 /// block — markers included — reaches the client as text.
923 #[test]
924 fn a_streamed_block_with_no_recoverable_name_is_content_markers_and_all() {
925 for body in ["{\"arguments\": {}}", "{\"name\": \"\"}", "words", ""] {
926 let (content, calls) = drive(true, &streamed(body));
927 assert!(calls.is_empty(), "{body:?} became a call");
928 assert_eq!(content, format!("{OPEN_TAG}{body}{CLOSE_TAG}"));
929 }
930 }
931
932 #[test]
933 fn a_stray_streamed_terminator_is_content() {
934 let tokens = [
935 (TEXT, "the tag is "),
936 (CLOSE_TAG_ID, CLOSE_TAG),
937 (TEXT, ", as shown"),
938 ];
939 let (content, calls) = drive(true, &tokens);
940 assert!(calls.is_empty());
941 assert_eq!(content, "the tag is </tool_call>, as shown");
942 }
943
944 /// A second opener inside an open block is body text, exactly as the
945 /// buffered scan treats it.
946 #[test]
947 fn a_nested_opener_is_body_text() {
948 let tokens = [
949 (OPEN_TAG_ID, OPEN_TAG),
950 (TEXT, "{\"name\": \"echo\", \"arguments\": {\"t\": \""),
951 (OPEN_TAG_ID, OPEN_TAG),
952 (TEXT, "\"}}"),
953 (CLOSE_TAG_ID, CLOSE_TAG),
954 ];
955 let (content, calls) = drive(true, &tokens);
956 assert_eq!(content, "");
957 assert_eq!(calls.len(), 1);
958 assert_eq!(calls[0].1.arguments, "{\"t\": \"<tool_call>\"}");
959 }
960
961 /// Models get this JSON wrong routinely. The call still goes out, with
962 /// whatever the `arguments` span held.
963 #[test]
964 fn malformed_json_in_a_streamed_block_still_yields_a_call() {
965 let (content, calls) = drive(
966 true,
967 &streamed(r#"{"name": "read", "arguments": {"p": 1,}}"#),
968 );
969 assert_eq!(content, "");
970 assert_eq!(calls[0].1.name, "read");
971 assert_eq!(calls[0].1.arguments, r#"{"p": 1,}"#);
972 }
973
974 /// The streaming and buffered paths mint the same ids for the same
975 /// completion, because there is one function that mints them.
976 #[test]
977 fn streamed_call_ids_match_the_buffered_paths() {
978 let body = r#"{"name": "read", "arguments": {}}"#;
979 let mut tokens = streamed(body);
980 tokens.extend(streamed(body));
981 let (_, calls) = drive(true, &tokens);
982
983 let parsed: Vec<ParsedCall> = calls.iter().map(|(_, call)| call.clone()).collect();
984 let buffered = wire_calls("chatcmpl-18f3a0001", &parsed);
985 let streamed: Vec<ToolCall> = calls
986 .iter()
987 .map(|(index, call)| wire_call("chatcmpl-18f3a0001", *index, call))
988 .collect();
989 assert_eq!(streamed, buffered);
990 assert_eq!(streamed[0].id, "call_18f3a0001_0");
991 assert_eq!(streamed[1].id, "call_18f3a0001_1");
992 }
993
994 /// The two constants are the mechanism. If they ever drift from the
995 /// tokenizer's `added_tokens`, streaming silently emits raw markup again.
996 #[test]
997 fn the_marker_ids_are_the_qwen3_added_tokens() {
998 assert_eq!(OPEN_TAG_ID, 151_657);
999 assert_eq!(CLOSE_TAG_ID, 151_658);
1000 assert_ne!(OPEN_TAG_ID, CLOSE_TAG_ID);
1001 }
1002
1003 /// The extracted call, re-rendered on the next turn, is the text the model
1004 /// wrote. That is the whole point of keeping the bytes: the renderer emits
1005 /// `<tool_call>\n{"name": "<name>", "arguments": <raw>}\n</tool_call>`,
1006 /// splicing a `ToolArguments::Raw` verbatim, so a preserved span rebuilds
1007 /// the block character for character.
1008 #[test]
1009 fn a_call_round_trips_back_into_the_block_the_model_emitted() {
1010 let reply = block(r#"{"path": "a.txt"}"#);
1011 let out = extract(&reply);
1012 let wire = wire_calls("chatcmpl-1", &out.calls);
1013 let rendered = format!(
1014 "<tool_call>\n{{\"name\": \"{}\", \"arguments\": {}}}\n</tool_call>",
1015 wire[0].function.name, wire[0].function.arguments
1016 );
1017 assert_eq!(rendered, reply);
1018 }
1019}