globazog
A high-performance, Windows-first globbing and directory-traversal library for Rust.
globazog answers a set of glob patterns over a set of roots in a single
traversal. It walks directories on a permit-bounded pool of background threads,
matches and filters entries inline using the filesystem's own inline metadata
(reparse bit, size, timestamps), and streams results back to you through a bounded,
no-drop completion ring. Native enumeration backends
(NtQueryDirectoryFile on Windows, getdents64 + statx on Linux) are used where
available, with a portable std::fs fallback everywhere else.
On a real machine, a full scan of a ~43,000-directory drive for **/*.h returns
~39,000 matches in ~1.3 s (release).
Features
- Multi-pattern, single traversal —
*.c+*.h+Cargo.tomlare answered in one walk; each match reports which patterns it satisfied. - Owned glob dialects —
posix(/separators,\escapes, case-sensitive) andwin(/and\separators,{{/}}brace escaping, case-insensitive), with*,?,**(whole-segment recursive), and{a,b}alternation. - Declarative predicates — per-pattern emit filters and a per-query descend filter over name, type, size, timestamps, reparse status, attributes, and depth — evaluated inline, no callbacks on the engine threads.
- Streaming completion ring — matches, directory enter/end markers, per-entry errors, and a terminal marker arrive as owned items you drain at your own pace; the drain rate is the backpressure throttle. Nothing is dropped.
- Cancellation — cancel a running walk and receive a terminal marker after the already-queued items; dropping the handle tears the walk down cleanly.
- Correct on messy filesystems — reversible name decoding (unpaired UTF-16 surrogates / non-UTF-8 bytes survive round-trip), reparse-cycle detection, and per-entry errors that never abort the walk.
Quick start
[]
= "0.1"
use ;
// Every `*.rs` under the current directory, recursively. Roots must be absolute
// (D-74), so resolve the CWD at your edge.
let handle = new
.root
.pattern
.submit
.expect;
let ring = handle.completions;
loop
Concepts
Query = roots × patterns
You build a query with QueryBuilder: add one or more roots (physical seed
directories), one or more patterns, then submit(). The pattern set is matched
against every root (the pattern-is-gospel model); an absolute pattern like
/etc/*.conf is lowered by peeling its literal prefix into a root. submit()
compiles the patterns (returning Err on a bad pattern) and starts the engine on
background threads, handing you a QueryHandle.
Dialects
Patterns are parsed in a named Dialect:
Dialect::Posix—/separators,\escapes, UTF-8, case-sensitive.Dialect::Win—/and\separators (so\cannot escape), UTF-8, case-insensitive; escape a literal brace by doubling it ({{→{,}}→}). Only usable on Windows.
Metacharacters: * (any run within a segment), ? (one code point), ** (a whole
segment: zero-or-more directory levels), and {a,b,c} (alternation).
Predicates: emit and descend
Beyond the glob, each entry can be filtered by declarative Leaf conditions
(ANDed together):
- Emit (per pattern) — the third argument to
.pattern(...). A match is reported only if the glob matches and every emit leaf holds. Empty = pass-all. - Descend (per query) —
.descend(...). The engine recurses into a directory only if some pattern still wants it and every descend leaf holds.
use ;
let handle = new
.root
// Only emit logs larger than 1 KiB.
.pattern
// Never descend into reparse points (junctions / symlinked dirs).
.descend
.submit
.expect;
Leaf constructors cover the common cases: Leaf::name_exact, Leaf::name_glob,
Leaf::name_contains, Leaf::name_extension, Leaf::name_in_set, plus
Size, Time, Depth, IsType, IsReparse, ReparseTag, AttrsAllSet, and
AttrsAllClear.
Portability note:
AttrsAllSet/AttrsAllClearandReparseTagcarry Win32 semantics — theFILE_ATTRIBUTE_*bitmask and reparse tag are0on non-Windows, so those leaves don't match there. UseIsReparse/IsTypefor cross-platform type and symlink/reparse checks.
The completion ring
handle.completions() is the CompletionRing you service. Pop items with
wait_pop() (blocking), pop() (non-blocking), or drain() (all queued). Each item
is a CqItem:
| Variant | Meaning |
|---|---|
Match(m) |
An entry matched. m.name, m.matched (a PatternMask — m.matched.iter() yields the pattern indices), and m.meta (size/timestamps/type). |
ContainerEnter(e) |
A directory scan started. e.id, e.parent, e.name. |
ContainerEnd(e) |
A directory subtree finished (bottom-up, 1:1 with its enter). |
Error(e) |
A per-entry/-subtree failure (e.g. permission denied); the walk continues past it. The one exception is a fatal error (see Failed below), which stops the walk. |
DecisionRequest(_) |
Reserved for defer-to-client (not yet emitted). |
Terminal(t) |
The walk ended. Always the last item. t.reason is Completed, Cancelled, or Failed — a fatal error (currently a root that could not be enumerated) stopped the walk; its causing Error item lands immediately before this terminal. |
Ordering guarantees you can rely on: a container's ContainerEnter precedes any
Match/child that references it, and ContainerEnds cascade bottom-up (a child ends
before its parent). Sibling directories interleave freely. Full paths are not shipped
per entry — reconstruct them client-side by keeping a container id → (parent, name)
map from the ContainerEnter stream and walking to the root, mirroring the engine's
own parent chain.
Options, backpressure, and cancellation
Options tunes permits (concurrent scans), ring_capacity, cycle_detection,
and a query-level case default. The ring is bounded and never drops: when it
fills, the engine parks its workers until you drain — so a slow consumer simply slows
the walk. Call handle.cancel() to stop early; you will still receive a terminal after
the already-queued items — Terminal { reason: Cancelled } when the request is honored
before the walk finishes, or Completed if the cancel raced with normal completion (a
Completed after cancel() is not a contract violation). Dropping the QueryHandle
cancels and joins the engine threads.
Building a tool
examples/glob.rs is a complete CLI built on this API —
multi-pattern, per-pattern emit filters, a per-pattern count summary, and a terminal
report. Run it with:
Platform support
- Windows — native
NtQueryDirectoryFile-based enumeration with inline attributes, reparse tag, 128-bit file id, and all timestamps. - Linux — native
openat+getdents64+statx(birth time included). - Other — a portable
std::fsbackend satisfying the same contract.
Status
The core library (matcher, dialects, predicates, native/portable backends, ring, and
the synchronous scheduler) is complete and tested on Windows and Linux. Planned
follow-ups: native async backends (IOCP / io_uring) with parent-relative open and
\\?\ long-path support, defer-to-client predicate escalation, and a zero-copy
name-blob fast path. See CHECKLIST.md and DESIGN-NOTES.md at the repository root.
License
MIT © Mike Grier