Crate fdu_core 

fdu — a fast, incremental file roll-up engine.

fdu answers, for any directory in a tree: how big is it, how many files does it hold, what changed most recently, and what kinds of files live in it — hierarchically, for every directory at once, from a single walk.

The shape: three artifacts, one contract

1. The index (Index) — the in-memory hierarchical structure: entry records plus per-directory roll-up state.
2. The snapshot (snapshot) — that index, serialized.
3. The change contract (Observation and Commit) — producers submit verified observations; the index commits clocked effective changes.

Everything else is a producer of observations or a consumer of exact commits. The walker establishes a baseline from upsert observations; the reconciler submits the conditional diff between indexed state and reality; the watch layer submits verified, coalesced observations. The index arbitrates them and re-rolls its reducers; a change feed consumes exact effective changes and state transitions from Commit.

A deliberate consequence: watching is not tied to the roll-up logic. The index knows apply(Observation) and nothing about filesystem events, so a batch scan, a test feeding synthetic observations, and a live watcher are indistinguishable to it.

Freshness is a ladder, not a set of alternatives

open is the conservative, blocking entry point: it loads a compatible snapshot, reconciles the configured filesystem scope, and only then returns. It does not serve the loaded baseline concurrently. Applications that want that model can own an IndexHandle, call the applying reconciliation APIs, and inspect Freshness while readers continue between short write batches. With the watch feature, watch::Watcher::apply_next verifies event hints and closes invalidations through subtree reconciliation; neither open nor the Python binding starts it implicitly. OpenedIndex is the additive long-lived owner: its clones share one live identity, cancellation domain, index, and joined shutdown. A cloned Index remains a detached image and never inherits that authority.

use fdu_core::{CachePolicy, open};
use fdu_core::query::{Basis, Delivery, Scope};
use std::path::Path;

let basis = Basis { root: Path::new(".").into(), scope: Scope::default(), content: Default::default() };
let (index, report) = open(&basis, &Delivery::new(CachePolicy::Auto, None))?;
let total = index.total();
println!("{} files, {} bytes ({:?})", total.files, total.bytes, report.path_taken);

Build features

• watch — the OS-native watch layer.

fdu-core has no default build features. The command and Python packages opt into watch, while embedding consumers can retain the smaller one-shot engine.

.gitignore handling is not a build feature: it has no dependency, so it is always compiled in, and whether a scan reads control files is decided at runtime by ScanConfig::read_controls.

Re-exports

pub use crate::watch_session as session;

pub use crate::admission::HiddenPolicy;

pub use crate::cache::CachePaths;

pub use crate::cache::CacheScope;

pub use crate::cache::CacheState;

pub use crate::cache::CacheStatus;

pub use crate::cache::ClearSummary;

pub use crate::cache::ContentInfo;

pub use crate::cache::ContentState;

pub use crate::cache::ContentStatus;

pub use crate::cache::LeftoverKind;

pub use crate::cache::SnapshotInfo;

pub use crate::cache::StaleReason;

pub use crate::cache::cache_status;

pub use crate::cache::clear_all_caches;

pub use crate::cache::clear_cache;

pub use crate::cache::list_caches;

pub use crate::control::CONTROL_FILE_NAME;

pub use crate::control::ControlAdmission;

pub use crate::control::ControlCoverage;

pub use crate::control::ControlIdentity;

pub use crate::control::ControlLimits;

pub use crate::control::ControlMatcher;

pub use crate::control::ControlObservation;

pub use crate::control::ControlRefusalReason;

pub use crate::control::ControlTable;

pub use crate::control::DEFAULT_CONTROL_BUDGET;

pub use crate::control::DEFAULT_CONTROL_LINE_LIMIT;

pub use crate::control::RefusedControl;

pub use crate::control::is_control_file;

pub use crate::scan::ReconcileReport;

pub use crate::scan::ScanConfig;

pub use crate::scan::ScanOrder;

pub use crate::scan::ScanReport;

pub use crate::watch_session::Batch;

pub use crate::watch_session::Change;

pub use crate::watch_session::ChangeKind;

pub use crate::watch_session::SaveOutcome;

pub use crate::watch_session::Session;

Modules

admission

Fixed rules that decide which filesystem facts belong in an index.

cache

Inspecting and clearing the snapshot cache.

classify

File-type recognition.

content

Optional, versioned file-content analysis.

control

Bounded, removal-aware control state used to classify retained filesystem facts.

counters

Low-distortion performance instrumentation for fdu.

query

Queries over a built index: what to select, which roll-ups to report, and the value grammars both are written in.

report_format

Serializing a Report to text, JSON, JSONL, and YAML.

scan

The scan layer: walking a tree, producing observations, and applying reconciliation.

snapshot

Persisting an index to disk and reading it back.

watch

Th OS-native watch layer: turning an unreliable event stream into verified observations.

watch_session

A live session: an index, a watcher, and the query they answer together.

Structs

AnalyzerProvenance

The analyzers a content tier’s records were produced by, and the options they ran with.

ApplyOutcome

Result of arbitrating and applying one producer observation.

ApplyStats

Summary of what one Index::apply call did.

Attrs

The stat fields an entry contributes to roll-ups, plus the ones that identify it.

ChangePoll

Result of one opened-root journal poll.

ChangeRequest

Input to one blocking opened-root journal poll.

ChildSnapshot

One direct child captured from a shared index at a single read boundary.

Clock

A monotonic logical clock, in the spirit of Watchman’s clockspec but process-local.

Commit

One atomic, exact index transition.

ContentAdmission

Proof that stored content can be projected to one requested identity.

ContentTierIdentity

Identity of a content tier: the per-file analysis records a sidecar holds.

ContinuationId

Opaque identifier for resumable work retained by one opened root.

DiscoveryBudget

Resource bounds applied to progressive discovery.

DiscoveryProgress

Committed, bounded discovery counters.

EngineVersion

Identity and exact sequence of one committed opened-root state.

EntryId

Identifier for an entry within an Index arena.

EntryScope

Which entries a scan retains: its depth, symlink, filesystem-boundary, hidden-entry, and special-object settings.

EntryTierIdentity

Identity of an entry tier: the entries a store holds and the roll-ups derived from them.

EntryValue

One immutable retained entry returned by an opened-root read.

ExtTally

Per-extension tally within a roll-up.

Fingerprint

The fingerprint used to decide whether an entry really changed.

FlatPage

One portable flat-entry page.

Impact

Bounded invalidation guidance derived from exact effective changes.

Index

The in-memory hierarchical index.

IndexHandle

Shareable owner for serving readers while reconciliation applies short writes.

IndexState

Coherent public state captured at an index commit boundary.

Issue

Bounded diagnostic evidence retained with an index state.

IssueSummary

Counts for the bounded issue details captured with a state.

Observation

A producer batch awaiting arbitration by the index.

ObservationOp

One observed operation together with its arbitration precondition.

OpenOptions

Configuration for a long-lived OpenedIndex.

OpenReport

What open did.

OpenedIndex

A long-lived, synchronously controlled filesystem index.

PageRequest

Output and work bounds for one resumable page.

PartitionRollUp

The two fixed aggregate partitions maintained for inventory reads.

PartitionRollUpSummary

Constant-size totals for the fixed all and unignored partitions.

PathExpectation

State and entry revisions captured at one observation boundary.

PendingSave

A snapshot write running alongside rendering.

PerformanceSummary

Operational work behind one one-shot report.

Plan

Validated policy shared by all engine execution routes.

PortablePath

A retained entry’s canonical POSIX-relative name, in the form ordered pages use.

Progress

A handle a route reports its progress through.

ProgressSnapshot

What a route has done so far, as read at one moment.

Provenance

Everything a consumer needs to decide how far to trust one value.

QueryLimit

Typed bounded-query result; no partial calculation is presented as exact.

ReadDiagnostics

Fixed-size diagnostics captured with a coherent read.

ReadRequest

Input to one coherent opened-root read.

ReadResponse

One coherent opened-root response.

RefreshResult

Result of one bounded, multi-path refresh.

RejectedRefreshPath

One refresh path that the engine declined, with its typed reason.

ReportRequest

The read half of a request at an opened root, plus one read’s work bound.

RollUp

Pre-computed aggregate state for one directory’s entire subtree.

RollUpSummary

Constant-size directory totals suitable for bounded interactive rows.

ScanScope

Semantic inputs that decide which entries and derived values belong in an index.

SemanticIdentity

Answer-semantics identity derived from validated classification and reducer rules.

SessionId

Opaque identity of one opened-root lifetime.

Since

Result of Index::since.

SnapshotIdentity

The identity of every tier a metadata snapshot holds.

TreePage

One structural page of a directory’s descendants, to the requested depth.

Work

Bounded work performed while committing producer input or serving engine reads.

Enums

CachePolicy

Whether a request may read and write the snapshot cache.

ChangeOutcome

Journal outcome at one coherent terminal version and state.

ControlTierIdentity

Identity of a .gitignore control tier.

CountResult

Product count whose exactness is explicit.

Coverage

Structural coverage of one opened root.

CoverageReason

Why an opened root cannot claim complete structural coverage.

EffectiveChange

One exact fact mutation performed by the index.

EntryKind

What kind of filesystem entry a record describes.

Error

Errors the engine can report.

Expectation

The condition under which an observation may be committed.

Freshness

Trust state for an index or queried subtree.

ImpactDomain

A stable fdu-native answer domain that one commit may have made stale.

InvalidateReason

Why a producer had to escalate to Op::InvalidateSubtree instead of describing a change precisely.

IssueKind

Stable category for one non-fatal condition or terminal provider failure.

Knowledge

Three-valued knowledge for a path lookup.

LifecyclePhase

Current activity of one opened root.

LimitedProjection

Projection whose deterministic work allowance was exhausted.

Load

Which persisted state execution may read.

Op

A single change to one path.

OpenPath

Which tier of the freshness ladder an open actually used.

OutcomeClass

Whether an answer fulfills the caller’s delivery contract.

PathState

The complete indexed state of one path at an observation boundary.

ProgressPhase

Which kind of work a route is doing.

ProjectionRefusal

Why one projection of a read refused, while every other projection still answered.

ProjectionResult

One projection result, in the same position as its request.

ReadProjection

One fdu-native projection requested under a coherent read boundary.

RefreshRejection

Why one requested refresh path was not verified.

Route

The engine lifecycle that will deliver an answer.

RowShape

Retained fields copied into portable page rows.

Serves

How a stored tier answers a request.

Source

Where a value came from, so a consumer can trade speed for certainty knowingly.

StateTransition

One observable transition that did not change a retained filesystem entry.

Status

Whether a value covers everything beneath its path.

Verify

Whether execution must verify the filesystem.

Constants

DEFAULT_COUNT_CAP

Default cap for a selection count not backed by an exact maintained aggregate.

DEFAULT_JOURNAL_CAPACITY_BYTES

Approximate bytes the exact commit history used by Index::since may retain.

MAX_CONTINUATION_RECORD_BYTES

Maximum retained payload for one handle-local continuation record.

MAX_COUNT_CAP

Maximum caller-selected cap for an on-demand aggregate.

MAX_DIRTY_PATHS

Maximum number of individual dirty paths retained in one commit.

MAX_ISSUE_MESSAGE_BYTES

Maximum UTF-8 bytes retained in one rendered issue message.

MAX_ISSUE_PATH_BYTES

Maximum native encoded bytes retained for one issue path.

MAX_PAGE_ROWS

Maximum rows returned by one page projection.

MAX_PAGE_WORK

Maximum deterministic work allowance accepted by one page projection.

MAX_PRIORITY_PATHS

Maximum paths accepted by one best-effort priority request.

MAX_READ_PROJECTIONS

Maximum native projections accepted by one coherent read.

MAX_REFRESH_PATHS

Maximum paths accepted by one refresh operation.

MAX_REPORT_VIEWS

Maximum report sections and reported omissions accepted in one opened read.

MAX_RETAINED_ISSUES

Maximum issue details retained by one index image.

MIN_JOURNAL_CAPACITY_BYTES

Smallest journal budget, in bytes, an opened root accepts.

Functions

default_cache_dir

The application cache directory, resolved independently of a scanned root.

default_cache_path

Conventional metadata snapshot path for callers without an explicit destination.

default_cache_path_in

The conventional metadata snapshot location for a root in the resolved directory.

open

Open a tree, using the snapshot cache when one is usable.

open_with_pending_save

Open a tree, returning the snapshot write for the caller to join.

plan

Validate a request and derive the least-retention plan for its delivery and route.

prepare_report

Execute a one-shot report, retaining the least state the request needs.

prepare_report_with_progress

Execute a one-shot report, reporting its progress through progress as it runs.

prepare_report_with_scan_diagnostics

Execute a one-shot report and retain scan diagnostics.

refresh

Reverify a retained index, refresh requested content, and persist according to delivery.

serves_snapshot

Whether a snapshot of the stored identity answers a request for wanted.

throughput_rates

Cumulative walk rates over one actual elapsed sample. The byte rate is binary GiB/s, rounded to three decimals; the grouped file rate counts complete files per second. Returns (files_per_second, gib_per_second) without unit labels, or None when elapsed time is zero. Neither rate estimates storage read bandwidth.

Type Aliases

Result

Result alias for engine operations.