# Description
This library implements machinery for extremely fast multithreaded directory traversal.
Using multiple threads leverages the kernels abilities to schedule IO-Requests in an
optimal way.
# How it works
The user crates a 'Gatherer' object which spawns threads listening on an
PriorityQueue. Sending a 'directory' to this queue let one thread pick it up and traverse the
directory. Each element found is then send to a custom function/closure which may decides on
how to process it:
* Directories can be send again into the input PriorityQueue where other
threads may pick them up. This happens until the input queue is exhausted, eventually traversing
all sub-directories of the directory send initially.
* Files and Directories can be send to an output mcmp queue where they can be further
processed.
* Errors are send to the output queue as they happen.
* Once the input queue becomes empty a 'Done' message is send to the output to notify the
listener there.
## Queues
A priority queue is choosen for the input to ensure that directories are processed in a file
handled preserving order. This is depth first in ascending inode order.
## Memory Optimizations
Handling pathnames of millions of files would need considerably much memory. To conserve this
demands a ObjectPath implementation encodes any path by its filename and a reference to its
parent directory. Futher all names are interned thus same names would require only memory once
for their storage.
# Benchmarking Results
See the 'BENCH.md' file for some tests. As baseline was the 'gnu find' utility chosen. In the
most extreme case this code can perform directory traversal 20 times faster. With slow
spinning disks and moderate settings (16 threads) 1.6 times faster.