content_scan
A small, extensible content scanning framework for Rust.
content_scan gives you a plug-in style pipeline for identifying, analyzing and extracting structured content out of arbitrary byte streams (files, buffers, embedded archives, etc.). You define your own content types and plug in identifiers, analyzers and extractors — the scanner takes care of dispatching, recursion and filtering.
Typical use cases:
- File-type / MIME-like detection based on magic bytes, extensions or file names.
- Static analysis pipelines (metrics, heuristics, feature extraction).
- Recursive scanning of container formats (archives, bundles, embedded blobs) with a configurable depth limit.
- Building custom "scanners" (antivirus-like tools, indexers, linters, forensics tools, …) on top of a common core.
Status: early / experimental (
0.1.x).
Table of contents
- content_scan
- Scanning pipeline
Workspace layout
This repository is a Cargo workspace with three members:
| Crate | Path | Purpose |
|---|---|---|
content_scan |
content_scan/ |
The main library: scanner, traits, matchers, filters. |
content_scan_proc_macro |
content-scan-proc-macro/ |
Companion proc-macro crate exposing #[derive(ContentType)]. Re-exported from content_scan. |
examples |
examples/ |
Runnable examples (sum, vowals). |
You normally only depend on content_scan — the proc-macro is re-exported for you.
Core concepts
The framework is built around a few small traits:
ContentType— a#[repr(u16)]enum describing the kinds of content your scanner knows about. Derived with#[derive(ContentType)].Content<T>— an abstract, seekable, read-only byte source with a path and a size. A ready-madeBufferContent<T>is provided for in-memory buffers.ContentIdentifier<T>— decides what a piece of content is (by magic bytes, extension, or file name) and validates the guess.ContentAnalyzer<T>— reads content and produces information (stored in a sharedContext).ContentExtractor<T>— pulls sub-contents out of a container and hands them back to the scanner, which recurses into them.Filter— decides which paths / sizes should be processed at all.Scanner<T>— the orchestrator; built viaScannerBuilder<T>.ScanResult<T>/ScanContentHandle— after a scan, the framework exposes the full tree of visited objects (parent / child / sibling links), each with its interned path, resolved content type and its own localVarMap.
Analyzers and extractors are either specific to a ContentType or generic (run on every scanned object). Each is registered with a priority byte to control execution order.
Getting started
Add the crate to your Cargo.toml (from a local path or git checkout while the crate is not yet published):
[]
= { = "path/to/content_scan/content_scan" }
Bring the prelude into scope:
use *;
Define your content types, plug in identifiers / analyzers / extractors, build a Scanner, and call scan():
use *;
// ... implement ContentIdentifier / ContentAnalyzer / ContentExtractor ...
Examples
The examples/ directory contains runnable programs. From the workspace root:
Counting vowels
A single analyzer that counts vowels in a magic-tagged buffer. See examples/vowals/main.rs.
use *;
;
;
Summing numbers extracted from text
This example shows the full pipeline — a text container is identified by magic bytes, an extractor pulls numeric substrings out of it as independent Number contents, and a numeric analyzer both sums them into a shared global variable and stashes each value into a per-object local VarMap. After the scan, the example walks the resulting tree via ScanResult. See examples/sum/main.rs.
;
API overview
ContentType
Every scanner is parameterized by a user-defined enum implementing ContentType. The #[derive(ContentType)] macro implements the trait automatically. Requirements:
- The enum must be annotated with
#[repr(u16)]. - Variants must be unit variants with no explicit discriminants.
- Up to
65536variants are supported.
bool also implements ContentType and is used internally by the matcher / filter machinery.
Content
A Content is any addressable byte source. It exposes a logical path (used for identification and filtering), a size, and a read(offset, count) method that returns a borrowed slice.
An in-memory implementation is provided:
new;
with_content_type;
from_parts;
You can implement Content<T> for files, memory-mapped regions, network streams, archive entries, etc.
ContentIdentifier
Identifiers tell the scanner what type a piece of content is when the content itself doesn't already report one.
Fast identification is performed with an internal matcher (single-pattern, packed magic table, or trie depending on the number and shape of patterns). After a fast match, validate() is called to confirm the guess.
At most one identifier per ContentType may be registered; the builder will panic otherwise.
ContentAnalyzer
Analyzers inspect content and write results into the shared Context. The returned NextAction controls the pipeline:
NextAction::Continue— run the next analyzer / extractor.NextAction::Skip— stop processing the current object (do not run further analyzers/extractors on it), but keep scanning siblings.NextAction::Exit— stop the whole scan.
Register analyzers with:
add_analyzer(content_type, priority, analyzer)— runs only when the content matchescontent_type.add_generic_analyzer(priority, analyzer)— runs for every scanned object, regardless of type.
Within a bucket, analyzers execute in ascending priority order.
ContentExtractor
Extractors turn a container into a stream of children:
init— called once, receives a scratchVarMap(context.extract()) valid until the next object is scanned. Returnfalseto skip this extractor.advance— advances to the next child and returns a lightweightEntrydescribing its path/size. ReturningNoneends the stream.extract— materializes the current child as a boxedContent<T>. The scanner then recursively scans it (subject tomax_depth).
Extractors are registered with add_extractor / add_generic_extractor, mirroring analyzers.
Filter / FilterBuilder
Filters decide whether a given (path, size) should be processed. They combine typed rules (extension / file-name allow- and deny-lists, powered by fast matchers) with arbitrary predicate callbacks, plus a default fallback:
use *;
let filter = new
.include_extensions
.exclude_file_names
.exclude // skip huge files
.deny_the_rest // or .allow_the_rest()
.build;
let scanner = new
.filter
// ...
.build;
Available builder methods:
| Method | Effect |
|---|---|
include_extensions(prec, &["ext", …]) |
Allow if the file's extension matches. |
exclude_extensions(prec, &["ext", …]) |
Deny if the file's extension matches. |
include_file_names(prec, &["name", …]) |
Allow if the file name matches exactly. |
exclude_file_names(prec, &["name", …]) |
Deny if the file name matches exactly. |
include(prec, fn(&str, u64) -> bool) |
Allow if the callback returns true. |
exclude(prec, fn(&str, u64) -> bool) |
Deny if the callback returns true. |
deny_the_rest() / allow_the_rest() |
Set the default and finalize the builder. |
Rules are evaluated in the order they were added; the first matching rule wins. The Precedence enum is provided to make the intent explicit.
Filters are also applied to entries produced by extractors, so filtering works transparently for embedded content.
Scanner / ScannerBuilder
Assemble a scanner with ScannerBuilder:
let scanner = new
.filter // optional
.max_depth // default: 8
.add_identifier
.add_analyzer
.add_generic_analyzer
.add_extractor
.add_generic_extractor
.build;
Then scan:
let result: ScanResult = scanner.scan;
max_depth limits how deep the scanner is allowed to recurse into extracted children (default 8, minimum 1).
Context / ScanResult
The Context passed to analyzers exposes three VarMaps (from the varmap crate, re-exported by content_scan):
context.global()— persists for the entirescan()call. Use it to accumulate results across all analyzed objects.context.local()— per-object scratch storage. The first call from an analyzer on a given object lazily grabs aVarMapfrom an internal pool, clears it, and attaches it to that object; subsequent calls (from other analyzers running on the same object) return the same map. It is kept alive after the scan and can be looked up on the correspondingScanContentHandleviaScanResult::local(handle).context.extract()— a scratch map handed to a single extractor'sinitcall.
context.objects_scanned() returns how many objects have been visited so far.
After scan() returns, you can read results from the ScanResult<T>. In addition to the classic aggregate view, it also exposes the full tree of scanned objects:
let res = scanner.scan;
let sum = res.global..unwrap_or;
println!;
var!("name") is a compile-time typed key macro provided by varmap.
Navigating the scan result tree
Every object visited by the scanner is recorded, along with its resolved content type, its path (interned in an internal arena) and its optional local VarMap. Objects are linked as a parent / first-child / next-sibling tree that mirrors the extraction hierarchy.
You navigate the tree with opaque ScanContentHandles returned by ScanResult<T>:
Typical walk:
if let Some = res.root
Because paths and local VarMaps live in pools owned by the scanner's Context, they are reused across successive scan() calls with zero re-allocation once steady-state is reached — the ScanResult simply borrows them for the lifetime of the current scan.
Scanning pipeline
For every scanned object, the scanner performs the following steps (see content_scan/src/scanner.rs):
-
Top-level filter check. If a
Filteris configured, the root content is tested first; if rejected, the scan returns immediately. -
Type resolution. If the content already reports a
content_type(), it is used as-is. Otherwise the scanner tries, in order:- magic bytes (first 16 bytes),
- exact file name,
- file extension.
Each candidate is confirmed via the corresponding identifier's
validate()method. -
Type-specific analyzers for the resolved type run in priority order.
-
Generic analyzers run for every object in priority order.
-
Type-specific extractors run and, for each entry they emit, the scanner recurses (subject to
max_depthandFilter). -
Generic extractors run and recurse in the same way.
While this is happening, the scanner also records the object into Context::objects — allocating its path in an internal arena, tagging it with the resolved content type, and linking it into its parent's child list. After scan() returns, that tree is exposed to the caller through ScanResult.
Any analyzer or extractor may short-circuit the current object with NextAction::Skip or abort the entire scan with NextAction::Exit.
Building & testing
Standard Cargo workflow from the workspace root:
# build everything
# run the unit tests
# run an example
The workspace pins resolver = "2" and applies a couple of shared Clippy overrides (module_inception, new_without_default) — see the root Cargo.toml.
License
Licensed under the MIT License. © 2026 Gavrilut Dragos.