furnace/batching/
mod.rs

1//! Dynamic batching: concurrent requests share forward passes.
2//!
3//! A few workers run a model's forward passes, one batch each at a time, and requests wait for
4//! them in a shared queue. While some worker is idle, every request gets a forward pass of its
5//! own, started at once. Once all workers are busy, the queue fills, and the next worker to
6//! finish takes everything queued, up to `max_batch_size` items, as one batch. If the queue
7//! runs dry before the batch is full while another request is on its way in (its body is still
8//! being received or decoded), the worker waits for it, but never longer than `batch_timeout`
9//! after the first request was queued. A lone request therefore never waits.
10//!
11//! This order suits CPUs: one small pass leaves cores idle (between layers, and in layers
12//! too small to split across every thread), and a second pass running alongside uses them,
13//! which gains more than merging the two into one batch. Past a few passes at once,
14//! batching wins: more passes only compete for the same cores, and with no limit every
15//! request slows down together.
16//!
17//! The algorithm is the batching engine's, the crate amate. The server runs it on its
18//! models' deliberative pathways in the nervous system (`nerve::serve`); [`BatchConfig`] is how
19//! those pathways sum requests.
20
21use std::time::Duration;
22
23/// How requests are grouped into batches.
24///
25/// Batching is off by default: every request runs its own forward pass at once, as in a
26/// hand-written server. Whether merging requests pays depends on the model. A convolutional
27/// network on a CPU (or most models on a GPU) runs a batch of 4 in about the time of 2 single
28/// passes, and serves more requests with batching on; a transformer on a CPU takes nearly 4
29/// times as long for 4, and merging only makes requests wait. Turn it on with
30/// `max_batch_size` above 1 for models of the first kind (`benches/burn` measures both).
31#[derive(Debug, Clone, Copy, PartialEq, Eq)]
32pub struct BatchConfig {
33    /// Most items in one forward pass; 1 (the default) turns batching off. A request with
34    /// more items than this still runs, alone.
35    pub max_batch_size: usize,
36    /// Longest wait for requests still being received, counted from when the first
37    /// request of the batch was queued
38    pub batch_timeout: Duration,
39    /// Forward passes that may run at the same time; requests are merged into batches
40    /// only while all of them are busy
41    pub workers: usize,
42}
43
44impl Default for BatchConfig {
45    fn default() -> Self {
46        Self {
47            max_batch_size: 1,
48            batch_timeout: Duration::from_millis(2),
49            workers: 4,
50        }
51    }
52}
53
54impl BatchConfig {
55    pub fn enabled(&self) -> bool {
56        self.max_batch_size > 1
57    }
58}
furnace/batching/mod.rs

furnace/batching/
mod.rs
