fork-map
A Rust library for running operations in a child process spawned by fork()
. Embrace fearful concurrency: fearful that your worker task will mess up your memory space.
Example
use fork_map;
Motivation
Some operations work best if run in their own process. Whether they impose single-threaded restrictions, they consume untold resources when left running, or you just want to abuse copy-on-write memory to eliminate startup time, sometimes you really just want to fork
and map
. My main uses for this crate have been trying to embed libClang, which unsafely uses static memory because it assumes it is running single-threaded, and running operations that leak memory.
Implementation and Support
fork_map
is written using libc::fork
and as such, will only work properly on *nix based systems that support fork
(sorry Windows users!). Since the child process inherits the parent's memory space (as copy-on-write), there are no constraints on the input value or the operation. The result value is serialized using serde_json
and sent over a libc
file handle via some incredibly C-inspired unsafe io code. The parent process reads the data from the file and waits for the child to exit before returning.
Use with rayon
It is generally expected that you will want to use this crate in conjunction with something like rayon
since the call to fork_map
blocks the thread of execution until the child process returns. In combination, you can have rayon
coordinate a pool of worker threads that each spawn and control child processes with minimal boilerplate. A lot of my use cases end up looking something like this:
use fork_map;
use *;
If you have a lot of small tasks that you can run on a child process, you can use rayon's chunks()
function and eliminate much of the overhead from calling fork()
a lot (which can be significant):
use fork_map;
use *;
Safety
Due to the nature of fork()
, this function is very unsound and likely violates most of Rust's guarantees about lifetimes, considering all of your memory gets duplicated into a second process, even though it calls exit(0)
after your closure is executed. Any threads other than the one calling fork_map
will not be present in the new process, so threaded lifetime guarantees are also violated. Don't even think about using async executors with this.