Skip to main content

WalkPool

Struct WalkPool 

Source
pub struct WalkPool { /* private fields */ }
Expand description

The ceiling on what the whole server holds at once, in bytes of walk bookkeeping, shared by every in-flight transfer.

§Why a per-request ceiling was never a bound on the box

DEFAULT_WALK_LIMIT_BYTES is generous on purpose: the bar is that a kernel-sized repository with no .bitmap clones, and that costs 5.59 GB (MEASURED, see the constant). gunnar serve’s admission gate then admits 256 concurrent transfers by default — a number chosen against CPU, in gunnar_ssh::admission, with nothing in it about memory. Multiply them and the shipped server will cheerfully begin 256 clones that between them need two terabytes, on a box that has not got it. The failure is the OOM killer picking a victim, which is the one refusal an operator cannot read.

So the ceiling has to be spent from a shared purse, and that is the reason the unit had to become bytes: object counts from two differently shaped repositories do not add up to anything.

§It admits by size, so small transfers never queue behind a big one

A reservation is taken as the walk grows, not at the door, so a request is only ever charged for what it is actually holding. A clone of a small repository never reaches WalkPool::FLOOR_BYTES and so never touches the pool at all — the atomic is left for the transfers that can hurt the machine. Above it, reservations move in WalkPool::CHUNK_BYTES steps, so a fourteen-million-object clone costs about 700 atomic operations across sixty seconds rather than one per object.

§What happens at the ceiling

crate::Error::WalkPoolFull, which reaches the client through UploadPack::serve_session’s error seam as a sentence naming both numbers. A refusal a client can read and retry is the whole difference between this and the accidental ceiling it replaces.

§What it does, MEASURED

bench/examples/walkpool-probe.rs, 2026-08-05, thirty-two concurrent bitmap-less clones of a 352 396-object repository in one process, three runs:

VmHWMservedrefused
one clone alone219 MB1—
32 at once, no pool4.66–4.73 GB32—
32 at once, ~900 MB pool1.09–1.33 GB3–527–29

21.2–21.6× one clone against 4.97–6.09×, so the fan-out’s peak comes down 3.5–4.3× and every refusal was WalkPoolFull. The pooled peak is above the purse because the purse charges the walk: the two or three clones that got in are also running a pack writer, and the twenty-nine that did not each held up to WalkPool::FLOOR_BYTES before being refused.

§This alone does not guarantee progress, and the reason is not fixable here

A walk’s size is not known at the door, so reservations are taken as it grows. A purse that is tight against a single walk therefore lets every walk in a fan-out climb together and then refuses all of them with nobody finished. MEASURED the same day: four concurrent 14 042 000-object clones against a purse of two served zero, where thirty-two 352 396-object clones against the same ratio served two — thirty-two threads on thirty-two cores do not advance in lockstep and four do.

That is bin-packing without knowing the sizes. Three alternative reservation policies — coarser bites, doubling bites, and refusing a first reservation above a fraction of the purse — were each worked through on paper against the four-clone case above and each still has a fan-out that jams it; none was implemented and measured, which is why that is written as reasoning and not as a result. The answer is the architecture gunnar already has. This is the backstop and --max-concurrent-transfers is the primary gate: the slot cap keeps the fan-out narrow enough that the purse is not tight against it, and the purse is what stops N admitted slots from meaning N times the largest repository. WalkPool::for_this_machine sizing the purse to MemTotal rather than to a constant is the other half of keeping that true on a box nobody measured in advance.

Implementations§

Source§

impl WalkPool

Source

pub const FLOOR_BYTES: usize

Below this much held bookkeeping a request does not touch the pool.

8 MiB is about 21 800 objects at WALK_BYTES_PER_OBJECT. Every repository in gunnar’s own bench corpus except the two largest sits under it, which is the point: metering an ordinary clone would put an atomic in front of a transfer that could not exhaust anything.

Source

pub const CHUNK_BYTES: usize

How much a growing walk reserves at a time, above Self::FLOOR_BYTES.

Source

pub fn new(limit_bytes: usize) -> Arc<Self> ⓘ

A pool of limit_bytes.

Source

pub fn for_this_machine() -> Arc<Self> ⓘ

Half of this machine’s MemTotal, or DEFAULT_WALK_LIMIT_BYTES if /proc/meminfo cannot be read.

Half rather than all: the pool bounds one of the things a git server holds. The pack writer’s buffers, the receive path’s index, the page cache the mmap’d packs live in and whatever else shares the box are not in it, and a memory ceiling set to the whole machine is a ceiling that fires after the machine is already in trouble.

The fallback is deliberately the per-request number, so a machine whose size cannot be read gets a pool that admits exactly one large clone rather than an accidentally unbounded one.

Source

pub fn limit_bytes(&self) -> usize

The ceiling this pool was built with.

Source

pub fn in_flight_bytes(&self) -> usize

Bytes reserved by transfers in flight right now.

Source

pub fn peak_bytes(&self) -> usize

The most this pool has ever had reserved at once.

Source

pub fn refusals(&self) -> u64

How many walks this pool has refused. Monotonic.

Trait Implementations§

Source§

impl Debug for WalkPool

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more