furnace/batching/
mod.rs

1//! Dynamic batching: concurrent requests share forward passes.
2//!
3//! A few workers run a model's forward passes, one batch each at a time, and requests wait for
4//! them in a shared queue. While some worker is idle, every request gets a forward pass of its
5//! own, started at once. Once all workers are busy, the queue fills, and the next worker to
6//! finish takes everything queued, up to `max_batch_size` items, as one batch. If the queue
7//! runs dry before the batch is full while another request is on its way in (its body is still
8//! being received or decoded), the worker waits for it, but never longer than `batch_timeout`
9//! after the first request was queued. A lone request therefore never waits.
10//!
11//! This order suits CPUs: one small pass leaves cores idle (between layers, and in layers
12//! too small to split across every thread), and a second pass running alongside uses them,
13//! which gains more than merging the two into one batch. Past a few passes at once,
14//! batching wins: more passes only compete for the same cores, and with no limit every
15//! request slows down together.
16//!
17//! The algorithm is the batching engine's. The server runs it on its models' deliberative pathways
18//! in the nervous system (`nerve::serve`); [`BatchConfig`] is how those pathways sum requests.
19
20pub(crate) mod engine;
21
22use std::time::Duration;
23
24/// How requests are grouped into batches.
25///
26/// Batching is off by default: every request runs its own forward pass at once, as in a
27/// hand-written server. Whether merging requests pays depends on the model. A convolutional
28/// network on a CPU (or most models on a GPU) runs a batch of 4 in about the time of 2 single
29/// passes, and serves more requests with batching on; a transformer on a CPU takes nearly 4
30/// times as long for 4, and merging only makes requests wait. Turn it on with
31/// `max_batch_size` above 1 for models of the first kind (`benches/burn` measures both).
32#[derive(Debug, Clone, Copy, PartialEq, Eq)]
33pub struct BatchConfig {
34    /// Most items in one forward pass; 1 (the default) turns batching off. A request with
35    /// more items than this still runs, alone.
36    pub max_batch_size: usize,
37    /// Longest wait for requests still being received, counted from when the first
38    /// request of the batch was queued
39    pub batch_timeout: Duration,
40    /// Forward passes that may run at the same time; requests are merged into batches
41    /// only while all of them are busy
42    pub workers: usize,
43}
44
45impl Default for BatchConfig {
46    fn default() -> Self {
47        Self {
48            max_batch_size: 1,
49            batch_timeout: Duration::from_millis(2),
50            workers: 4,
51        }
52    }
53}
54
55impl BatchConfig {
56    pub fn enabled(&self) -> bool {
57        self.max_batch_size > 1
58    }
59}
furnace/batching/mod.rs

furnace/batching/
mod.rs
