content_scan
A small, extensible content scanning framework for Rust.
content_scan gives you a plug-in style pipeline for identifying, analyzing and extracting structured content out of arbitrary byte streams (files, buffers, embedded archives, etc.). You define your own content types and plug in identifiers, analyzers and extractors — the scanner takes care of dispatching, recursion and filtering.
Typical use cases:
- File-type / MIME-like detection based on magic bytes, extensions or file names.
- Static analysis pipelines (metrics, heuristics, feature extraction).
- Recursive scanning of container formats (archives, bundles, embedded blobs) with a configurable depth limit.
- Walking a directory tree and scanning every file in it (
FolderContent+FolderExtractor). - Building custom "scanners" (antivirus-like tools, indexers, linters, forensics tools, …) on top of a common core.
Status: early / experimental (
0.1.x).
Table of contents
- content_scan
- Scanning pipeline
Workspace layout
This repository is a Cargo workspace with three members:
| Crate | Path | Purpose |
|---|---|---|
content_scan |
content_scan/ |
The main library: scanner, traits, matchers, filters. |
content_scan_proc_macro |
content-scan-proc-macro/ |
Companion proc-macro crate exposing #[derive(ContentType)]. Re-exported from content_scan. |
examples |
examples/ |
Runnable examples (sum, vowals, image_size). |
You normally only depend on content_scan — the proc-macro is re-exported for you.
Core concepts
The framework is built around a few small traits:
ContentType— a#[repr(u16)]enum describing the kinds of content your scanner knows about. Derived with#[derive(ContentType)].Content<T>— an abstract, seekable, read-only byte source with aContentPathand a size. Ready-madeBufferContent<T>(in-memory),FileContent<T>(memory-mapped file) andFolderContent<T>(a directory, used as a container) implementations are provided.ContentPath— the path or synthetic address of a piece of content. Holds a UTF-8 printable view always, and keeps the original OS path when the name is not valid UTF-8 so the file can still be opened.ContentIdentifier<T>— decides what a piece of content is (by magic bytes, extension, or file name) and validates the guess.ContentAnalyzer<T>— reads content and produces information (stored in a sharedContext).ContentExtractor<T>— pulls sub-contents out of a container through anacquire/advance/extract/releasesession keyed by anExtractionHandle, then hands children back to the scanner for recursion.ExtractionPool<T>is the helper that mints those handles and stores the per-session state behind them.FolderExtractor<T>is a ready-made extractor that enumerates a directory.Filter— decides which paths / sizes should be processed at all.Scanner<T>— the orchestrator; built viaScannerBuilder<T>.ScanResult<T>/ScanContentHandle— after a scan, the framework exposes the full tree of visited objects (parent / child / sibling links), each with its interned path, resolved content type and its own localVarMap.
Analyzers and extractors are either specific to a ContentType or generic (run on every scanned object). Each is registered with a priority byte to control execution order.
Getting started
Add the crate to your Cargo.toml (from a local path or git checkout while the crate is not yet published):
[]
= { = "path/to/content_scan/content_scan" }
Bring the prelude into scope:
use *;
Define your content types, plug in identifiers / analyzers / extractors, build a Scanner, and call scan():
use *;
// ... implement ContentIdentifier / ContentAnalyzer / ContentExtractor ...
Examples
The examples/ directory contains runnable programs. From the workspace root:
Counting vowels
A single analyzer that counts vowels in a magic-tagged buffer. See examples/vowals/main.rs.
use *;
;
;
Summing numbers extracted from text
This example shows the full pipeline — a text container is identified by magic bytes, an extractor pulls numeric substrings out of it as independent Number contents, and a numeric analyzer both sums them into a shared global variable and stashes each value into a per-object local VarMap. After the scan, the example walks the resulting tree via ScanResult. The extractor keeps its cursor in an ExtractionPool, so it can be re-entered while a previous session is still open. See examples/sum/main.rs.
;
Reading PNG / BMP / JPEG dimensions
The image_size example registers one identifier + analyzer pair per image format (png.rs, bmp.rs, jpeg.rs) and each analyzer stores a Size { width, height } in the object's local VarMap. It accepts either a single file or a whole directory:
A file is wrapped in a FileContent and scanned directly. A directory is wrapped in a FolderContent tagged with the user's own ImageType::Folder variant, and a recursive FolderExtractor registered for that type turns each directory entry (including nested folders) into a child content object:
let mut scanner = new
.filter
.add_identifier
.add_analyzer
// ...bmp / jpeg...
.add_extractor
.build;
let res = if new.is_dir else ;
Note the false passed to scan for the directory case: the filter only allows image extensions, so applying it to the root folder would reject the scan before it starts. The example then prints the resulting tree, indented by depth, with {width} x {height} next to every recognized image.
API overview
ContentType
Every scanner is parameterized by a user-defined enum implementing ContentType. The #[derive(ContentType)] macro implements the trait automatically. Requirements:
- The enum must be annotated with
#[repr(u16)]. - Variants must be unit variants with no explicit discriminants.
- Up to
65536variants are supported.
bool also implements ContentType and is used internally by the matcher / filter machinery.
Content
A Content is any addressable byte source. It exposes a ContentPath (used for identification, filtering, and display), a size, and a read(offset, count) method that returns a borrowed slice.
Three implementations ship with the crate — an in-memory one, a file-backed one, and a directory marker:
new;
with_content_type;
from_parts;
new; // impl AsRef<Path>; false = shared read
with_content_type;
with_size; // size already known, skip the stat
with_content_type;
BufferContent constructors take a &str and treat it as a synthetic address (always UTF-8). FileContent and FolderContent take impl AsRef<Path> and go through ContentPath::from_os, so a real filesystem name that is not valid UTF-8 stays openable.
FileContent opens the file lazily on the first read(). Pass exclusive = true to memory-map with an exclusive lock (files already open elsewhere will fail). Pass false for shared access through an LRU page cache. with_size is useful when the size is already known (a directory walk has just stat'ed the entry, for instance) and you want to avoid a second filesystem call.
FolderContent represents a directory rather than bytes: its size() is 0 and read() always returns None. It always reports the content type you give it, so the scanner dispatches straight to the extractor registered for that type — see Walking the file system.
You can also implement Content<T> for memory-mapped regions, network streams, archive entries, etc.
ContentPath
ContentPath is the path type used everywhere a piece of content is named: Content::path(), Entry::path, filter callbacks, and (as interned bytes) the scan result tree. It covers both virtual addresses (archive.zip://inner/file.txt, "number") and real OS paths, including names that are not valid UTF-8.
from_str/set_from_str— use for archive members, in-memory buffers, and any label you already know is UTF-8. Do not stringify a real OS path and pass it here: a non-UTF-8 filesystem name would lose the bytes needed to reopen it.from_os/set_from_os— use forDirEntry,PathBuf, and anything that came from the filesystem. When the path is valid UTF-8 only the string is stored. Otherwise the originalOsStringis kept alongside a lossy printable view (U+FFFDfor invalid sequences), soas_path()still names the original file andis_lossless()isfalse.as_printable_string— always available, never fails. Safe to log or print. Exact only whenis_lossless()istrue.as_path— the&Pathto hand tofs::read_dir,File::open, and similar. For a non-UTF-8 path this is the preserved OS name, not the lossy string.as_bytes— what identification andFilterinspect (file name / extension). On Unix this is the faithful path bytes; on Windows it is the printable string's bytes.
IntoContentPath lets an API accept &str, &Path, or an owned ContentPath and route each to the cheapest correct constructor.
FolderExtractor fills each Entry with set_from_os, so a directory walk does not drop non-UTF-8 names the way a to_str().unwrap_or_default() conversion would.
ContentIdentifier
Identifiers tell the scanner what type a piece of content is when the content itself doesn't already report one.
Fast identification is performed with an internal matcher (single-pattern, packed magic table, or trie depending on the number and shape of patterns). After a fast match, validate() is called to confirm the guess. Overlapping magic prefixes resolve to the longest match in both the packed table and the trie.
If identify_method returns None, validate() is still called — after magic, file name, and extension have all been considered — so you can classify content with custom logic (heuristics, path shape, size, …). Those identifiers are tried in the order they were registered.
At most one identifier per ContentType may be registered; the builder will panic otherwise.
ContentAnalyzer
Analyzers inspect content and write results into the shared Context. The returned NextAction controls the pipeline:
NextAction::Continue— run the next analyzer / extractor.NextAction::Skip— stop processing the current object (do not run further analyzers/extractors on it), but keep scanning siblings.NextAction::Exit— stop the whole scan.
Register analyzers with:
add_analyzer(content_type, priority, analyzer)— runs only when the content matchescontent_type.add_generic_analyzer(priority, analyzer)— runs for every scanned object, regardless of type.
Within a bucket, analyzers execute in ascending priority order.
ContentExtractor
Extractors turn a container into a stream of children, driven as a short session keyed by an opaque ExtractionHandle:
acquire— called once per parent. Receives a scratchVarMap(context.extract()) valid until the next object is scanned. ReturnSome(handle)to start the session, orNoneto skip this extractor. Handles are minted by anExtractionPool;ExtractionHandleis opaque and cannot be constructed directly.advance— advances the session to the next child and returns a lightweightEntrydescribing its path/size. ReturningNoneends the stream.extract— materializes the current child as a boxedContent<T>. The scanner then recursively scans it (subject tomax_depth). ReturningNoneskips just this entry; enumeration continues with the nextadvance.release— called exactly once for every successfully acquired handle (including when the scan stops early viaNextAction::Skip/NextAction::Exit). Use it to free per-session resources.
Set Entry::skip_from_filtering to true to exempt an entry from the active Filter. This matters for container entries that would never pass the filter themselves: a FolderExtractor restricted to *.png files, for example, still has to descend into subdirectories, whose names carry no .png extension. Keep one Entry as a field on the extractor and overwrite entry.path in place with ContentPath::set_from_str (synthetic names) or ContentPath::set_from_os (real OS paths) so advance does not allocate a new path for every child.
The handle lets one extractor instance keep per-session state even when extractions nest or interleave. The Entry itself is owned by the extractor (not the pool), because advance has to return &Entry while the pool may be borrowed mutably for session data. Extractors are registered with add_extractor / add_generic_extractor, mirroring analyzers.
ExtractionPool
Because one extractor instance is shared by every object of its type, per-session state cannot live in plain fields — a nested or re-entered extraction would overwrite it. ExtractionPool<T> solves this: it stores one T per live session and hands back the ExtractionHandle that identifies it.
Slots are recycled through a free list, and every handle carries a monotonically increasing uid, so a stale handle whose slot has already been reused resolves to None instead of silently aliasing another session's data. The Entry announced by advance lives on the extractor itself — overwrite entry.path with set_from_str / set_from_os so the path is not reallocated for every child.
Walking the file system
FolderExtractor<T> is a built-in extractor that enumerates a directory and emits one child per entry. Pair it with a FolderContent root and a ContentType variant of your own that stands for "directory":
let mut scanner = new
.add_extractor // recursive, shared file opens
.add_identifier
.add_analyzer
.build;
let mut root = with_content_type;
let res = scanner.scan;
Behaviour worth knowing:
- The
recursiveflag passed tonewdecides whether subdirectories are emitted at all. Whenfalse, only files directly inside the folder are scanned. - Subdirectories are emitted as
FolderContentcarrying the same content type as their parent, so the same extractor picks them up again. Files becomeFileContentbuilt withwith_size, reusing the size from the directory entry's metadata. Each entry's path is filled withContentPath::set_from_os, so non-UTF-8 filesystem names stay openable. - Directory symlinks are skipped, which keeps cyclic link structures from looping forever.
- An unreadable entry (permission error, failed
file_type) is skipped; the rest of the directory is still enumerated. - Subdirectory entries are marked
skip_from_filtering, so an extension-basedFilternarrows down the files without preventing the walk from descending. - Recursion is still bounded by the scanner's
max_depth(default8), which here translates into directory nesting levels.
Filter / FilterBuilder
Filters decide whether a given (ContentPath, size) should be processed. They combine typed rules (extension / file-name allow- and deny-lists, powered by fast matchers) with arbitrary predicate callbacks, plus a default fallback:
use *;
let filter = new
.include_extensions
.exclude_file_names
.exclude // skip huge files
.deny_the_rest // or .allow_the_rest()
.build;
let scanner = new
.filter
// ...
.build;
Available builder methods:
| Method | Effect |
|---|---|
include_extensions(prec, &["ext", …]) |
Allow if the file's extension matches (ASCII case-insensitive). |
exclude_extensions(prec, &["ext", …]) |
Deny if the file's extension matches (ASCII case-insensitive). |
include_file_names(prec, &["name", …]) |
Allow if the basename matches (ASCII case-insensitive). |
exclude_file_names(prec, &["name", …]) |
Deny if the basename matches (ASCII case-insensitive). |
include(prec, fn(&ContentPath, u64) -> bool) |
Allow if the callback returns true. |
exclude(prec, fn(&ContentPath, u64) -> bool) |
Deny if the callback returns true. |
deny_the_rest() / allow_the_rest() |
Set the default and finalize the builder. |
Rules are grouped by Precedence and evaluated from Highest to Lowest. Within the same precedence, rules keep the order they were added. The first matching rule wins. Extension and file-name rules are ASCII case-insensitive (Photo.JPG matches jpg).
Filters are also applied to entries produced by extractors, so filtering works transparently for embedded content.
Scanner / ScannerBuilder
Assemble a scanner with ScannerBuilder:
let scanner = new
.filter // optional
.max_depth // default: 8
.add_identifier
.add_analyzer
.add_generic_analyzer
.add_extractor
.add_generic_extractor
.build;
Then scan:
let result: ScanResult = scanner.scan;
max_depth limits how deep the scanner is allowed to recurse into extracted children (default 8, minimum 1). The root is depth 1; a child of a depth-N object is depth N + 1. Extraction stops when that next child would exceed max_depth, so max_depth(8) visits at most eight objects on any path.
The second argument to scan decides whether the configured Filter is applied to the root object itself. Pass true for a normal file — the scan then returns an empty ScanResult if the filter rejects it. Pass false when the root is a container that the filter was never written to accept, such as a folder being walked with a filter that only allows png files. Extracted children are always filtered regardless of this flag (unless their Entry opts out via skip_from_filtering).
A scanner is reusable: scan clears its internal Context on entry, so one instance can process many inputs in sequence.
Context / ScanResult
The Context passed to analyzers exposes three VarMaps (from the varmap crate, re-exported by content_scan):
context.global()— persists for the entirescan()call. Use it to accumulate results across all analyzed objects.context.local()— per-object scratch storage. The first call from an analyzer on a given object lazily grabs aVarMapfrom an internal pool, clears it, and attaches it to that object; subsequent calls (from other analyzers running on the same object) return the same map. It is kept alive after the scan and can be looked up on the correspondingScanContentHandleviaScanResult::local(handle).context.extract()— a scratch map handed to a single extractor'sacquirecall.
context.objects_scanned() returns how many objects have been visited so far.
After scan() returns, you can read results from the ScanResult<T>. In addition to the classic aggregate view, it also exposes the full tree of scanned objects:
let res = scanner.scan;
let sum = res.global..unwrap_or;
println!;
var!("name") is a compile-time typed key macro provided by varmap. Custom types can be stored in a VarMap by deriving VarMapValue, which is re-exported from content_scan as well:
context.local.set;
Navigating the scan result tree
Every object visited by the scanner is recorded, along with its resolved content type, its path (interned from ContentPath::as_printable_string() into an internal arena) and its optional local VarMap. Objects are linked as a parent / first-child / next-sibling tree that mirrors the extraction hierarchy.
You navigate the tree with opaque ScanContentHandles returned by ScanResult<T>:
ScanResult::path is the interned printable view of the object's ContentPath (as_printable_string() at scan time). For a live content object, use content.path().as_printable_string() to display it or content.path().as_path() to open it.
Typical walk:
if let Some = res.root
Because paths and local VarMaps live in pools owned by the scanner's Context, they are reused across successive scan() calls with zero re-allocation once steady-state is reached — the ScanResult simply borrows them for the lifetime of the current scan.
Scanning pipeline
For every scanned object, the scanner performs the following steps (see content_scan/src/scanner.rs):
-
Top-level filter check. If a
Filteris configured andscanwas called withfilter_root = true, the root content is tested first; if rejected, the scan returns immediately with an empty result. -
Type resolution. If the content already reports a
content_type(), it is used as-is. Otherwise the scanner tries, in order, usingContentPath::as_bytes()for the name-based steps:- magic bytes (first 16 bytes),
- exact file name,
- file extension,
- identifiers that returned
Nonefromidentify_method(eachvalidate()is tried in registration order).
Each fast-matcher candidate is confirmed via the corresponding identifier's
validate()method. Custom identifiers have no pre-filter;validate()is the identification. -
Type-specific analyzers for the resolved type run in priority order.
-
Generic analyzers run for every object in priority order.
-
Type-specific extractors run (
acquire→advance/extractloop →release) and, for each entry they emit, the scanner recurses (subject tomax_depthandFilter). Entries markedskip_from_filteringbypass theFiltercheck. -
Generic extractors run the same session lifecycle and recurse in the same way.
While this is happening, the scanner also records the object into Context::objects — interned from ContentPath::as_printable_string() into an internal arena, tagged with the resolved content type, and linked into its parent's child list. After scan() returns, that tree is exposed to the caller through ScanResult.
Any analyzer or extractor may short-circuit the current object with NextAction::Skip or abort the entire scan with NextAction::Exit.
Building & testing
Standard Cargo workflow from the workspace root:
# build everything
# run the unit tests
# run an example
The workspace pins resolver = "2" and applies a couple of shared Clippy overrides (module_inception, new_without_default) — see the root Cargo.toml.
License
Licensed under the MIT License. © 2026 Gavrilut Dragos.