content_scan
A small, extensible content scanning framework for Rust.
content_scan gives you a plug-in style pipeline for identifying, analyzing and extracting structured content out of arbitrary byte streams (files, buffers, embedded archives, etc.). You define your own content types and plug in identifiers, analyzers and extractors — the scanner takes care of dispatching, recursion and filtering.
Typical use cases:
- File-type / MIME-like detection based on magic bytes, extensions or file names.
- Static analysis pipelines (metrics, heuristics, feature extraction).
- Recursive scanning of container formats (archives, bundles, embedded blobs) with a configurable depth limit.
- Walking a directory tree and scanning every file in it (
FolderContent+FolderExtractor). - Building custom "scanners" (antivirus-like tools, indexers, linters, forensics tools, …) on top of a common core.
Status: early / experimental (
0.1.x).
Table of contents
- content_scan
- Scanning pipeline
Workspace layout
This repository is a Cargo workspace with three members:
| Crate | Path | Purpose |
|---|---|---|
content_scan |
content_scan/ |
The main library: scanner, traits, matchers, filters. |
content_scan_proc_macro |
content-scan-proc-macro/ |
Companion proc-macro crate exposing #[derive(ContentType)] and #[derive(Dependencies)]. Re-exported from content_scan. |
examples |
examples/ |
Runnable examples (sum, vowals, image_size, base64_find, md5, entropy, find_zip, zip_png_size). |
You normally only depend on content_scan — the proc-macro is re-exported for you.
Core concepts
The framework is built around a few small traits:
ContentType— a#[repr(u16)]enum describing the kinds of content your scanner knows about. Derived with#[derive(ContentType)].Content<T>— an abstract, seekable, read-only byte source with aContentPathand a size. Ready-madeBufferContent<T>(in-memory),FileContent<T>(memory-mapped file) andFolderContent<T>(a directory, used as a container) implementations are provided.ContentPath— the path or synthetic address of a piece of content. Holds a UTF-8 printable view always, and keeps the original OS path when the name is not valid UTF-8 so the file can still be opened.ContentIdentifier<T>— decides what a piece of content is (by magic bytes, extension, or file name) and validates the guess.ContentAnalyzer<T, M>— reads content and produces information (stored in a sharedContext). Analyzers write per-object or scan-wideVarMaps, emitFindings withcontext.add_finding(...), and can queue extra extraction withcontext.request_extract(ty). The optionalMisFindingMetadataattached to each finding (defaultNoMetadata). Every analyzer implementsDependencies(typically via#[derive(Dependencies)]) so debug builds can check that required analyzers are registered with a lower priority.ContentExtractor<T>— pulls sub-contents out of a container.create_sessionreceives anOwnedContentPtrto the parent and anExtractionContextdescribing the region to look at (offset, optionallength, optionalparams), and returns anExtractionSession. The session then yields children viaadvance/extract. Per-session state (cursors, open archives, the currentEntry) lives on that session object, not on the extractor — one extractor instance is shared and sessions may nest. Ready-made extractors:FolderExtractor<T>(directory walk) andZipExtractor<T>(ZIP members). Extractors that need astd::io::Readstream wrap the parent in aContentReader.Filter— decides which paths / sizes should be processed at all.Scanner<T, M>— the orchestrator; built viaScannerBuilder<T>(new()forNoMetadata, orwith_metadata::<M>()for typed finding extras).ScanResult<T, M>/ScanContentHandle— after a scan, the framework exposes the full tree of visited objects (parent / child / sibling links), each with its interned path, resolved content type and its own localVarMap, plus a flat list ofFindings recorded by analyzers.
Analyzers are either specific to a ContentType or generic (run on every scanned object), and each is registered with a priority byte to control execution order. In debug builds, ScannerBuilder::build also checks each analyzer's Dependencies requires list: every named analyzer must be registered with a strictly smaller priority. Extractors are registered per type and run in registration order — both when the current object is that type, and when an analyzer requests that type.
Getting started
Add the crate to your Cargo.toml (from a local path or git checkout while the crate is not yet published):
[]
= { = "path/to/content_scan/content_scan" }
Bring the prelude into scope:
use *;
Define your content types, plug in identifiers / analyzers / extractors, build a Scanner, and call scan(). Analyzers need #[derive(Dependencies)] in addition to ContentAnalyzer.
use *;
// ... implement ContentIdentifier / ContentAnalyzer / ContentExtractor ...
Examples
The examples/ directory contains runnable programs. From the workspace root:
Counting vowels
A single analyzer that counts vowels in a magic-tagged buffer. See examples/vowals/main.rs.
use *;
;
;
Summing numbers extracted from text
This example shows the full pipeline — a text container is identified by magic bytes, an extractor pulls numeric substrings out of it as independent Number contents, and a numeric analyzer both sums them into a shared global variable and stashes each value into a per-object local VarMap. After the scan, the example walks the resulting tree via ScanResult. The extractor's cursor lives on an ExtractionSession that holds an OwnedContentPtr to the parent, so a nested extraction can open another session without overwriting the first. See examples/sum/main.rs.
;
Reading PNG / BMP / JPEG dimensions
The image_size example registers one identifier + analyzer pair per image format (png.rs, bmp.rs, jpeg.rs) and each analyzer stores a Size { width, height } in the object's local VarMap. It accepts either a single file or a whole directory:
A file is wrapped in a FileContent and scanned directly. A directory is wrapped in a FolderContent tagged with the user's own ImageType::Folder variant, and a recursive FolderExtractor registered for that type turns each directory entry (including nested folders) into a child content object:
let mut scanner = new
.filter
.add_identifier
.add_analyzer
// ...bmp / jpeg...
.add_extractor
.build;
let res = if new.is_dir else ;
Note the false passed to scan for the directory case: the filter only allows image extensions, so applying it to the root folder would reject the scan before it starts. The example then prints the resulting tree, indented by depth, with {width} x {height} next to every recognized image.
Finding and decoding Base64
base64_find shows an analyzer requesting extractors of another type. A Text analyzer locates contiguous Base64 runs, emits request_extract(MyTypes::Base64) for each hit, a Base64 extractor decodes the slice into a Vec<u8> tagged Base64Decoded, and a Base64Decoded analyzer prints that buffer. With no arguments it scans a built-in sample; pass a file or folder to scan real content:
;
;
The extractor is registered for Base64, not for Text. It only runs because the finder queued that type; the parent file itself is never identified as Base64.
Computing MD5 hashes
md5 hashes every file with a generic analyzer. ComputeHashAnalyzer is registered with add_generic_analyzer, so it runs on every scanned object regardless of type. Instead of printing from the analyzer, it records each digest as a Finding via context.add_finding. After the scan, the example iterates res.findings(). A recursive FolderExtractor walks the tree when the argument is a directory; folders themselves are skipped (they have no bytes to hash).
;
Measuring Shannon entropy
entropy is the same generic-analyzer + findings pattern as md5, but each finding carries typed FindingMetadata: a newtype Entropy(f64) with the Shannon entropy (bits per byte) of that file. The scanner is built with ScannerBuilder::with_metadata::<Entropy>(), and the analyzer implements ContentAnalyzer<MyTypes, Entropy>. The finding text is a coarse classification — packed if entropy is greater than 7.8, encrypted if it is greater than 7.0, otherwise normal — while the exact value lives in the metadata.
;
;
Listing ZIP files
find_zip walks a directory with FolderExtractor and uses the built-in ZipIdentifier (local-file magic PK\x03\x04, then an EOCD check in validate) to decide which files are ZIPs. A type-specific analyzer prints each match. Identification is by content, not by .zip extension.
let mut scanner = new
.add_identifier
.add_analyzer
.add_extractor
.build;
Reading PNG dimensions inside a ZIP
zip_png_size combines ZipIdentifier + ZipExtractor with a PNG identifier / analyzer. The ZIP is the root (scan(..., false) so the filter is not applied to the archive itself); the extractor emits each member as a child, the filter keeps only *.png, and the PNG analyzer stores {width, height} in that child's local VarMap.
let mut scanner = new
.filter
.add_identifier
.add_extractor
.add_identifier
.add_analyzer
.build;
let mut content = new;
let res = scanner.scan; // don't filter the ZIP itself
API overview
ContentType
Every scanner is parameterized by a user-defined enum implementing ContentType. The #[derive(ContentType)] macro implements the trait automatically. Requirements:
- The enum must be annotated with
#[repr(u16)]. - Variants must be unit variants with no explicit discriminants.
- Derive
OrdandPartialOrdas well — the trait requires them. - Up to
65536variants are supported.
bool also implements ContentType and is used internally by the matcher / filter machinery.
Content
A Content is any addressable byte source. It exposes a ContentPath (used for identification, filtering, and display), a size, and a read(offset, count) method that returns a borrowed slice. set_content_type is what the scanner calls when an analyzer returns ReprocessAsType; built-in contents store the new type so the next analysis pass skips identifiers.
Three implementations ship with the crate — an in-memory one, a file-backed one, and a directory marker:
new;
from_vec; // move, no copy
with_content_type;
from_parts;
new; // impl AsRef<Path>; false = shared read
with_content_type;
with_size; // size already known, skip the stat
with_content_type;
BufferContent constructors take a &str and treat it as a synthetic address (always UTF-8). FileContent and FolderContent take impl AsRef<Path> and go through ContentPath::from_os, so a real filesystem name that is not valid UTF-8 stays openable.
FileContent opens the file lazily on the first read(). Pass exclusive = true to memory-map with an exclusive lock (files already open elsewhere will fail). Pass false for shared access through an LRU page cache. with_size is useful when the size is already known (a directory walk has just stat'ed the entry, for instance) and you want to avoid a second filesystem call.
FolderContent represents a directory rather than bytes: its size() is 0 and read() always returns None. It always reports the content type you give it, so the scanner dispatches straight to the extractor registered for that type — see Walking the file system.
You can also implement Content<T> for memory-mapped regions, network streams, archive entries, etc.
ContentPath
ContentPath is the path type used everywhere a piece of content is named: Content::path(), Entry::path, filter callbacks, and (as interned bytes) the scan result tree. It covers both virtual addresses (archive.zip://inner/file.txt, "number") and real OS paths, including names that are not valid UTF-8.
from_str/set_from_str— use for archive members, in-memory buffers, and any label you already know is UTF-8. Do not stringify a real OS path and pass it here: a non-UTF-8 filesystem name would lose the bytes needed to reopen it.from_os/set_from_os— use forDirEntry,PathBuf, and anything that came from the filesystem. When the path is valid UTF-8 only the string is stored. Otherwise the originalOsStringis kept alongside a lossy printable view (U+FFFDfor invalid sequences), soas_path()still names the original file andis_lossless()isfalse.empty/clear— construct or reset an empty path while keeping the string allocation. Sessions typicallyclearthenset_from_os/set_from_stron a reusedEntry.as_printable_string— always available, never fails. Safe to log or print. Exact only whenis_lossless()istrue.as_path— the&Pathto hand tofs::read_dir,File::open, and similar. For a non-UTF-8 path this is the preserved OS name, not the lossy string.as_bytes— what identification andFilterinspect (file name / extension). On Unix this is the faithful path bytes; on Windows it is the printable string's bytes.
ContentPath implements From for &str, String, &String, &Path, PathBuf, and &PathBuf. UTF-8 strings go through from_str (or with_string when already owned); OS paths go through from_os. FileContent and FolderContent take impl AsRef<Path>, so String and PathBuf work at those constructors too.
FolderExtractor fills each Entry with set_from_os, so a directory walk does not drop non-UTF-8 names the way a to_str().unwrap_or_default() conversion would.
ContentIdentifier
Identifiers tell the scanner what type a piece of content is when the content itself doesn't already report one.
Fast identification is performed with an internal matcher (single-pattern, packed magic table, or trie depending on the number and shape of patterns). After a fast match, validate() is called to confirm the guess. Overlapping magic prefixes resolve to the longest match in both the packed table and the trie.
Extension and file-name methods are ASCII case-insensitive: both the registered pattern and the path basename / extension are compared in lowercase. Notes.TXT matches Extension("txt"); makefile matches Name("Makefile"). Magic methods remain exact byte matches. A magic sequence (Magic / MultipleMagic) must be at most 16 bytes — that is the window the scanner reads (content.read(0, 16)). ScannerBuilder::build panics if a registered magic is longer. To inspect bytes past that window, return None from identify_method and call content.read in validate.
If identify_method returns None, validate() is still called — after magic, file name, and extension have all been considered — so you can classify content with custom logic (payload bytes via content.read, path shape, size, …). Those identifiers are tried in the order they were registered.
At most one identifier per ContentType may be registered; the builder will panic otherwise. It also panics if a Magic / MultipleMagic pattern is longer than 16 bytes.
ZipIdentifier
ZipIdentifier<T> is a built-in identifier for ZIP archives. Fast identification is the local-file magic PK\x03\x04; validate then looks for an End of Central Directory record in the tail of the content, so a file that merely starts with those four bytes is rejected. Identification is by content, not by .zip extension.
.add_identifier
ZipExtractor
ZipExtractor<T> is a built-in extractor that unpacks ZIP members as child content. Pair it with ZipIdentifier. Small entries (< 1 MiB) become an in-memory BufferContent; larger ones are decompressed to a temp file and wrapped in FileContent. Directory entries inside the archive are skipped. The session reads the parent through a ContentReader, so the archive can be a file, a buffer, or any other Content.
.add_identifier
.add_extractor
See Reading PNG dimensions inside a ZIP for a complete pipeline.
Dependencies
Every ContentAnalyzer must implement Dependencies. The #[derive(Dependencies)] macro (re-exported from content_scan) does that from a helper attribute:
;
;
;
nameis required and must be a non-empty string. Other analyzers refer to this plugin by that name inrequires.requiresis optional. It may be a single string or an array of strings naming analyzers that must run first.
The derived name() / dependencies() methods exist only when debug_assertions are enabled. In debug builds, ScannerBuilder::build checks that:
- Every name listed in
requiresis a registered analyzer (typed or generic — names are global, not perContentType). - Each required analyzer has a strictly smaller
prioritythan the one that requires it (lower numbers run first).
Release builds skip the check. Duplicate names among registered analyzers are not rejected; the last registration wins for the debug map.
Generics are not supported by the derive. Unit structs, tuple structs, named-field structs, enums, and unions all work.
ContentAnalyzer
Analyzers inspect content and write results into the shared Context. Use context.local() for per-object VarMaps, context.global() for scan-wide aggregates, and context.add_finding(...) for a flat list of Findings. To pull nested content out of the current object using extractors registered for a different type — for example an analyzer that locates an embedded ZIP and wants the Zip extractor to open it — call context.request_extract(ty) and emit an extraction request. Only analyzers return AnalysisOutcome; that value controls the rest of this object:
AnalysisOutcome::Continue— run the next analyzer for this object; after the last analyzer, run own-type extractors then requested extractors.AnalysisOutcome::Skip— stop this object: do not run remaining analyzers or any extractors on it. Siblings and later objects still scan.AnalysisOutcome::Exit— abort the entire scan.AnalysisOutcome::ReprocessAsType(ty)— stop remaining analyzers for this pass, run only requested extractors, then (if this object has changed type fewer times thanmax_change_type, default 2) assigntyon the same tree node and re-run typed + generic analyzers. Identifiers are not re-run. A generic analyzer that returns this is currently treated asContinue.
Register analyzers with:
add_analyzer(content_type, priority, analyzer)— runs only when the content matchescontent_type.add_generic_analyzer(priority, analyzer)— runs for every scanned object, regardless of type.
Within a bucket, analyzers execute in ascending priority order. If an analyzer requires another (see Dependencies), register that dependency with a smaller priority; debug builds enforce this in build().
ContentExtractor
Extractors turn a container into a stream of children. The scanner calls create_session once per parent; the returned ExtractionSession then enumerates children. Methods return Option — they do not return AnalysisOutcome and cannot Skip or Exit on their own.
create_session is called after that object's analyzers have run (Continue: own-type then requested extractors; ReprocessAsType: requested extractors only). It receives:
- an
OwnedContentPtrto the parent — store it on the session ifadvance/extractneed to read the parent; - an
ExtractionContextdescribing the region to look at (offset, optionallength, optionalparams). Copy those fields into the session; the context is only valid for this call.
Return Some(session) to start enumerating, or None to skip this extractor (the scanner moves on to the next one registered for the same type). The session is dropped when enumeration ends, when a nested child's analyzer returns AnalysisOutcome::Exit, or when the scanner moves on. Implement Drop on the session if you need to close files or free buffers.
An extractor registered for type T runs in two situations:
- The current object was identified as
T. The context then covers the whole object (offset = 0,length = Some(content.size()),params = None). - An analyzer requested extraction of
Tfrom the current object viacontext.request_extract. The context then carries the requested offset, length, and params. The parent does not need to have been identified asT.
One extractor instance is shared by every object of its type, and create_session can be re-entered while a previous session is still live (nested containers). Keep configuration on the extractor and put cursors, open archives, and the current Entry on the session. Extractors are registered with add_extractor for a specific ContentType. Multiple extractors for the same type run in registration order.
ExtractionSession
A session is a live extraction of children from one parent. The scanner drives it as a short loop:
advance— move to the next child and return a lightweightEntrydescribing its path/size. ReturningNoneends the stream and the session is dropped.extract— materialize that child as a boxedContent<T>. The scanner then recursively scans it (subject tomax_depth). ReturningNoneskips just this entry; enumeration continues with the nextadvance.
A child's Skip does not end this session — the scanner continues with the next advance. A child's Exit drops the session immediately as the scan unwinds.
Set Entry::skip_from_filtering to true to exempt an entry from the active Filter. This matters for container entries that would never pass the filter themselves: a FolderExtractor restricted to *.png files, for example, still has to descend into subdirectories, whose names carry no .png extension. Keep one Entry as a field on the session and overwrite entry.path in place with ContentPath::set_from_str (synthetic names) or ContentPath::set_from_os (real OS paths) so advance does not allocate a new path for every child.
OwnedContentPtr
OwnedContentPtr<T> is the handle create_session receives instead of a &mut dyn Content<T>. It does not own the parent — it is a non-owning pointer that derefs to dyn Content<T>, so a session can call path(), size(), read(), and content_type() through it across advance / extract without fighting the borrow checker.
The scanner guarantees the parent outlives the session. Do not store the pointer after the session is dropped.
A typical extractor therefore looks like this (the sum example, simplified):
;
ContentReader
ContentReader<T> is a sequential std::io::Read + Seek adapter over an OwnedContentPtr. Content::read is random-access and returns a borrowed slice; libraries that expect a stream (the zip crate used by ZipExtractor, parsers, decompressors, …) need a cursor instead.
A short slice from the underlying Content is not treated as EOF: the adapter copies what it got, advances, and the next Read::read continues from there. A Content::read that returns None before the advertised size() becomes an UnexpectedEof error. Seeking past the end is allowed (same as std::io::Cursor); a later read then returns Ok(0).
ExtractionContext
ExtractionContext is what create_session receives. It names a window inside the parent Content plus an optional parameter map:
offset— first byte of the region. Type-specific extraction of the parent itself always starts at0.length—Some(n)when the caller knows the region isnbytes.Nonemeans the extractor determines the extent itself (parse until the format ends, scan to EOF, …). Type-specific extraction of the parent itself passesSome(content.size()).params— extras an analyzer attached with.param(...)(password, codec, flags, …).Nonewhen the request had no.param()calls, including type-specific extraction of the parent itself. The borrow lasts only forcreate_session— copy values into the session if you need them later.
Requesting extraction
Analyzers that find nested content of another type queue a pass with context.request_extract(ty). The call returns an ExtractRequestBuilder; chain setters and commit with .emit(). Dropping the builder without emit does nothing.
Builder methods:
| Method | Effect |
|---|---|
at(offset) |
Byte offset within the parent. Defaults to 0. |
len(n) |
Asserts the region is n bytes. Omit to leave length = None. |
param(key, value) |
Adds one extractor-specific parameter. The first call reserves a pooled VarMap; later calls write into the same map. A request with no .param() carries no map (params = None). |
emit() |
Commits the request. Required — the builder is #[must_use]. |
After this object's analyzers finish with Continue, the scanner:
- Runs extractors registered for the object's own identified type (whole object).
- Then, in emission order, runs extractors registered for each requested type on the same parent, with that request's
ExtractionContext.
Skip or Exit skips extractors on this object, including queued requests. ReprocessAsType runs only the requested extractors (own-type extractors for the old type are skipped), then may re-analyze the same object as the new type.
The current object does not need to have been identified as the requested type. Several requests (of the same or different types) may be emitted from one analyzer; each is independent. Nested child scans start with an empty request queue.
The Zip extractor then copies the region out of create_session:
Walking the file system
FolderExtractor<T> is a built-in extractor that enumerates a directory and emits one child per entry. Pair it with a FolderContent root and a ContentType variant of your own that stands for "directory":
let mut scanner = new
.add_extractor // recursive, shared file opens
.add_identifier
.add_analyzer
.build;
let mut root = with_content_type;
let res = scanner.scan;
Behaviour worth knowing:
- The
recursiveflag passed tonewdecides whether subdirectories are emitted at all. Whenfalse, only files directly inside the folder are scanned. - Subdirectories are emitted as
FolderContentcarrying the same content type as their parent, so the same extractor picks them up again. Files becomeFileContentbuilt withwith_size, reusing the size from the directory entry's metadata. Each entry's path is filled withContentPath::set_from_os, so non-UTF-8 filesystem names stay openable. - Directory symlinks (and, on Windows, directory junctions / reparse points) are skipped, which keeps cyclic link structures from looping forever. Symlinks to regular files are followed and scanned like any other file; dangling links are skipped.
- An unreadable entry (permission error, failed
file_type) is skipped; the rest of the directory is still enumerated. - Subdirectory entries are marked
skip_from_filtering, so an extension-basedFilternarrows down the files without preventing the walk from descending. - Recursion is still bounded by the scanner's
max_depth(default8), which here translates into directory nesting levels.
Filter / FilterBuilder
Filters decide whether a given (ContentPath, size) should be processed. They combine typed rules (extension / file-name allow- and deny-lists, powered by fast matchers) with arbitrary predicate callbacks, plus a default fallback:
use *;
let filter = new
.include_extensions
.exclude_file_names
.exclude // skip huge files
.deny_the_rest // or .allow_the_rest()
.build;
let scanner = new
.filter
// ...
.build;
Available builder methods:
| Method | Effect |
|---|---|
include_extensions(prec, &["ext", …]) |
Allow if the file's extension matches (ASCII case-insensitive). |
exclude_extensions(prec, &["ext", …]) |
Deny if the file's extension matches (ASCII case-insensitive). |
include_file_names(prec, &["name", …]) |
Allow if the basename matches (ASCII case-insensitive). |
exclude_file_names(prec, &["name", …]) |
Deny if the basename matches (ASCII case-insensitive). |
include(prec, fn(&ContentPath, u64) -> bool) |
Allow if the callback returns true. |
exclude(prec, fn(&ContentPath, u64) -> bool) |
Deny if the callback returns true. |
deny_the_rest() / allow_the_rest() |
Set the default and finalize the builder. |
Rules are grouped by Precedence and evaluated from Highest to Lowest. Within the same precedence, rules keep the order they were added. The first matching rule wins. Extension and file-name rules are ASCII case-insensitive (Photo.JPG matches jpg).
Filters are also applied to entries produced by extractors, so filtering works transparently for embedded content.
Scanner / ScannerBuilder
Assemble a scanner with ScannerBuilder:
let scanner = new
.filter // optional
.max_depth // default: 8
.max_change_type // default: 2
.add_identifier
.add_analyzer
.add_generic_analyzer
.add_extractor
.build;
Then scan:
let result: ScanResult = scanner.scan;
max_depth limits how deep the scanner is allowed to recurse into extracted children (default 8, minimum 1). The root is depth 1; a child of a depth-N object is depth N + 1. Extraction stops when that next child would exceed max_depth, so max_depth(8) visits at most eight objects on any path.
max_change_type limits how many times one object may ReprocessAsType (default 2, clamped to 1..=T::COUNT). Each accepted change assigns the new type on the same tree node and re-runs analyzers; identifiers are not re-run. Over the cap, reprocessing stops and the object keeps its current type.
The second argument to scan decides whether the configured Filter is applied to the root object itself. Pass true for a normal file — the scan then returns an empty ScanResult if the filter rejects it. Pass false when the root is a container that the filter was never written to accept, such as a folder being walked with a filter that only allows png files. Extracted children are always filtered regardless of this flag (unless their Entry opts out via skip_from_filtering).
A scanner is reusable: scan clears its internal Context on entry (including the findings list), so one instance can process many inputs in sequence.
ScannerBuilder::new() produces a scanner whose findings carry no extra data (NoMetadata). When analyzers attach typed extras (severity, offset, rule id, …), start from ScannerBuilder::<MyType>::with_metadata::<M>() instead — see Findings.
build() panics if two identifiers are registered for the same ContentType, or if a magic pattern is longer than 16 bytes. In debug builds it also panics if an analyzer requires a name that is not registered, or if that dependency does not have a strictly smaller priority — see Dependencies.
Context / ScanResult
The Context passed to analyzers (from the varmap crate, re-exported by content_scan) exposes two VarMaps, a findings list, plus a way to queue extra extraction:
context.global()— persists for the entirescan()call. Use it to accumulate results across all analyzed objects.context.local()— per-object scratch storage. The first call from an analyzer on a given object lazily grabs aVarMapfrom an internal pool, clears it, and attaches it to that object; subsequent calls (from other analyzers running on the same object) return the same map. It is kept alive after the scan and can be looked up on the correspondingScanContentHandleviaScanResult::local(handle).context.add_finding(text, source, metadata)— records aFindingon the current object. After the scan, iterate them withres.findings().context.request_extract(ty)— queues an extra extraction pass. OnContinue, those extractors run after this object's own extractors; onReprocessAsType, they run instead. See Requesting extraction. The request queue is cleared at the start of every object's scan, including nested children.
context.objects_scanned() returns how many objects have been visited so far.
After scan() returns, you can read results from the ScanResult<T>. In addition to the classic aggregate view, it also exposes the full tree of scanned objects:
let res = scanner.scan;
let sum = res.global..unwrap_or;
println!;
var!("name") is a compile-time typed key macro provided by varmap. Custom types can be stored in a VarMap by deriving VarMapValue, which is re-exported from content_scan as well:
context.local.set;
Findings
Analyzers can emit a flat list of findings in addition to (or instead of) writing VarMaps. A finding is a short interned string attached to the object that was current when it was recorded, plus an optional source label and optional typed metadata.
Record one during analyze:
context.add_finding;
| Argument | Meaning |
|---|---|
finding |
The text itself (hash digest, message, match string, …). Interned into the scanner's path arena. |
source |
Optional label for who produced it (rule name, plugin id). None if unused. |
metadata |
Optional typed extra of type M. Pass None when using the default NoMetadata. |
Calling add_finding when no object is current (outside an analyzer) is a no-op. After scan() returns, iterate findings in emission order:
for f in res.findings
ScanResult::findings() returns a FindigsIterator that yields Findings. The iterator (and each Finding) borrows the ScanResult; do not start another scan on the same Scanner while they are still in use. The finding text and source stay valid for the lifetime of that result because they are interned in the same arena as object paths.
The default metadata type is NoMetadata — ScannerBuilder::new() and ContentAnalyzer<T> use it, and analyzers pass None as the third argument. To attach typed extras (severity, offset, rule id, …), implement the FindingMetadata marker trait (Copy is required so findings stay cheap) and build the scanner with with_metadata:
;
let mut scanner =
.add_generic_analyzer
.build;
The same M flows through Scanner<T, M>, Context<T, M>, ScanResult<T, M>, and ContentAnalyzer<T, M>. See Computing MD5 hashes for findings without custom metadata, and Measuring Shannon entropy for a complete example that attaches a f64 via FindingMetadata.
Navigating the scan result tree
Every object visited by the scanner is recorded, along with its resolved content type, its path (interned from ContentPath::as_printable_string() into an internal arena) and its optional local VarMap. Objects are linked as a parent / first-child / next-sibling tree that mirrors the extraction hierarchy.
You navigate the tree with opaque ScanContentHandles returned by ScanResult<T>:
ScanResult::path is the interned printable view of the object's ContentPath (as_printable_string() at scan time). For a live content object, use content.path().as_printable_string() to display it or content.path().as_path() to open it.
Typical walk:
if let Some = res.root
Because paths and local VarMaps live in pools owned by the scanner's Context, they are reused across successive scan() calls with zero re-allocation once steady-state is reached — the ScanResult simply borrows them for the lifetime of the current scan.
Scanning pipeline
For every scanned object, the scanner performs the following steps (see content_scan/src/scanner.rs):
-
Top-level filter check. If a
Filteris configured andscanwas called withfilter_root = true, the root content is tested first; if rejected, the scan returns immediately with an empty result. -
Type resolution. If the content already reports a
content_type(), it is used as-is. Otherwise the scanner tries, in order, usingContentPath::as_bytes()for the name-based steps:- magic bytes (first 16 bytes;
buildpanics if a registered magic is longer), - exact file name,
- file extension,
- identifiers that returned
Nonefromidentify_method(eachvalidate()is tried in registration order).
Each fast-matcher candidate is confirmed via the corresponding identifier's
validate()method. Custom identifiers have no pre-filter;validate()is the identification. - magic bytes (first 16 bytes;
-
Record. The object is appended to
Context::objects— interned fromContentPath::as_printable_string(), tagged with the resolved type, and linked into its parent's child list — before analyzers run. -
Type-specific analyzers for the current type run in priority order. They may write
VarMaps, emitFindings viacontext.add_finding, queue extra extraction, or return a steeringAnalysisOutcome. -
Generic analyzers run for every object in priority order (same recording APIs), unless a typed analyzer already returned
Skip,Exit, orReprocessAsType. -
Type-specific extractors for the resolved type run in registration order only if analysis returned
Continue(create_session→advance/extractloop; the session is dropped when the loop ends). Eachcreate_sessionreceives anOwnedContentPtrto the parent and anExtractionContextcovering the whole object (offset = 0,length = Some(size),params = None). For each entry they emit, the scanner recurses (subject tomax_depthandFilter). Entries markedskip_from_filteringbypass theFiltercheck. -
Extraction requests queued by analyzers via
context.request_extract(ty)then run, in emission order — after own-type extractors onContinue, or alone onReprocessAsType. For each request the extractors registered fortyrun on the same parent, with that request's offset, length, and params. The parent does not need to have been identified asty. -
ReprocessAsType. After those requested extractors, if this object has changed type fewer times thanmax_change_type, the scanner callsset_content_type, updates the tree node'stype_id, and re-enters analysis on the same node (steps 4–7). Identifiers, the filter, and the stop condition are not re-run. Over the cap, the object keeps its current type.
Findings emitted during the analyzer steps are appended to the same Context and later exposed through ScanResult::findings. After scan() returns, the tree is available through ScanResult.
Only analyzers return AnalysisOutcome. Extractor and session methods return Option (create_session / advance / extract); they cannot short-circuit the scan themselves.
- Analyzer
Skipon an object stops remaining analyzers and extractors on that object. The scanner maps that Skip toContinuefor the parent, so the session that produced the object keeps enumerating siblings. - Analyzer
Exitaborts the whole scan. The extraction session that produced the current object is dropped as the call stack unwinds; remaining extractors on ancestors do not run. - Analyzer
ReprocessAsType(ty)stops remaining analyzers for this pass, runs requested extractors, then re-analyzes the same object asty(capped bymax_change_type). A child'sReprocessAsTypedoes not end the parent session.
Building & testing
Standard Cargo workflow from the workspace root:
# build everything
# run the unit tests
# run an example
The workspace pins resolver = "2" and applies a couple of shared Clippy overrides (module_inception, new_without_default) — see the root Cargo.toml.
License
Licensed under the MIT License. © 2026 Gavrilut Dragos.