Skip to main content

Module working

Module working 

Source
Expand description

Where the objects live between fetches, and who may read them.

A git store needs somewhere to put an object database. That directory holds the contents of a private repository, so it is created private at creation0700, by the call that creates it, never by a chmod afterwards. A chmod after the fact leaves a window in which the directory is world-readable, and a window is all an attacker on a shared host needs. It is the same rule the core crate’s on-disk cache follows.

§Two places it can be

A temporary directory, by default. Made when the first fetch runs, removed when the source is dropped. Nothing survives the process, which is the right default for a container: no growth to manage, no stale objects, no secret left on a volume after the pod dies. It costs one full fetch per process start.

A directory the caller names, with GitSource::cache_dir. It survives restarts, so a restart transfers almost nothing — worth it for a large repository or a fleet that restarts often.

§What a store may delete, and where

Either directory grows: a shallow fetch of a moving branch writes one pack per transfer, the old ones stop being reachable the moment the local ref moves, and nothing in git removes them until a gc. A long-lived watcher accumulates one pack per push for as long as it runs, and a caller-named directory carries that on across restarts.

So this crate compacts, and the rule it compacts under is narrow enough to state in one line: it deletes only what it wrote, only in a directory it created for itself, and only on a trigger the caller can see and turn off.

  • what it wrote — the working directory is a bare object database holding packs from this crate’s fetches and one ref this crate writes. There is nothing else in it to lose, and what is removed is re-obtained by the very fetch that removed it.
  • a directory it created — a marker file is written when this crate initialises the object database, and a directory without it is never touched. A caller who points cache_dir at a repository that already exists gets the old behaviour, unpruned, rather than losing a repository to a store that assumed the directory was its own.
  • a visible triggerBuilder::compact_after is the number of transfers a directory may accumulate, thirty-two by default, and 0 turns it off entirely for a caller who would rather run git gc --prune=now on their own cadence.

The compaction is not a gc: it empties the object database and lets the next fetch refill it, which is one full transfer every compact_after pushes and leaves the directory holding exactly the current commit again. Deleting a pack and repacking would be less to transfer and much more to get wrong — a store that rewrites an object database is a store that can corrupt one, and this one cannot: the only thing it can lose is a copy of something the remote still has.

It happens before a fetch that is going to read, never after one and never on a watch’s idle tick, so nothing the current call depends on is removed and nothing is emptied that is not about to be refilled.

§One directory, one source

Two sources fetching into one directory would interleave their ref updates and their packs. A caller-named directory is therefore claimed by the source that names it, and a second source in the same program naming the same directory is refused at construction — before anything is written — rather than corrupting it. The default, a temporary directory per source, cannot collide at all.

§Two processes are still not detected, and that is a decision

The claim above is in this process only. Two programs pointed at one directory are not detected, are not supported, and this crate will not grow a lock file to change that. The reasoning is worth writing down, because “add a lock file” is the obvious answer and it is the wrong one here:

  • A stale-lock heuristic is wrong exactly where sharing happens. The deployment that shares a working directory is not two programs on one host; it is one volume mounted into two containers. Pid liveness cannot see across a pid namespace — both containers have a live pid 1, so a lock left by a dead process reads as held and a lock held by a live one reads as held for a different reason. An age bound replaces that with a guess about how long a fetch may take, and two nodes writing one network volume do not agree on the clock the guess is measured against.
  • An advisory lock the kernel releases has no staleness — and no reach. flock would be the correct mechanism, and it is unreliable on precisely the network filesystems that make the sharing possible in the first place. It also costs a dependency in a crate whose small graph is one of the things it claims, to convert an unsupported configuration into a start-up failure on filesystems that do not implement locking.

What is done instead is to bound the damage. Concurrent fetches into one object database are what git itself is built for: objects are written to a temporary file and renamed, and a ref update takes a .lock. Compaction is the part this crate added, and it is why the marker above is required: a program that empties a directory another program is reading costs that program one failed fetch, which is a failure this crate already promises to survive — the previously fetched document stays installed and a watch waits out the interval. That is asserted rather than claimed: a_working_directory_emptied_by_another_program_costs_a_fetch_rather_than_the_source empties one underneath a live source and fetches again. Give each program its own directory anyway.