content_scan 0.2.2

An extensible, plugin-based framework for identifying, analyzing, and recursively extracting structured content from arbitrary bytes.
Documentation

content_scan

A small, extensible content scanning framework for Rust.

content_scan gives you a plug-in style pipeline for identifying, analyzing and extracting structured content out of arbitrary byte streams (files, buffers, embedded archives, etc.). You define your own content types and plug in identifiers, analyzers and extractors — the scanner takes care of dispatching, recursion and filtering.

Typical use cases:

  • File-type / MIME-like detection based on magic bytes, extensions or file names.
  • Static analysis pipelines (metrics, heuristics, feature extraction).
  • Recursive scanning of container formats (archives, bundles, embedded blobs) with a configurable depth limit.
  • Walking a directory tree and scanning every file in it (FolderContent + FolderExtractor).
  • Building custom "scanners" (antivirus-like tools, indexers, linters, forensics tools, …) on top of a common core.

Status: early / experimental (0.1.x).


Table of contents


Workspace layout

This repository is a Cargo workspace with three members:

Crate Path Purpose
content_scan content_scan/ The main library: scanner, traits, matchers, filters.
content_scan_proc_macro content-scan-proc-macro/ Companion proc-macro crate exposing #[derive(ContentType)] and #[derive(Dependencies)]. Re-exported from content_scan.
examples examples/ Runnable examples (sum, vowals, image_size, base64_find, md5, entropy, find_zip, zip_png_size).

You normally only depend on content_scan — the proc-macro is re-exported for you.


Core concepts

The framework is built around a few small traits:

  • ContentType — a #[repr(u16)] enum describing the kinds of content your scanner knows about. Derived with #[derive(ContentType)].
  • Content<T> — an abstract, seekable, read-only byte source with a ContentPath and a size. Ready-made BufferContent<T> (in-memory), FileContent<T> (memory-mapped file) and FolderContent<T> (a directory, used as a container) implementations are provided.
  • ContentPath — the path or synthetic address of a piece of content. Holds a UTF-8 printable view always, and keeps the original OS path when the name is not valid UTF-8 so the file can still be opened.
  • ContentIdentifier<T> — decides what a piece of content is (by magic bytes, extension, or file name) and validates the guess.
  • ContentAnalyzer<T, M> — reads content and produces information (stored in a shared Context). Analyzers write per-object or scan-wide VarMaps, emit Findings with context.add_finding(...), and can queue extra extraction with context.request_extract(ty). The optional M is FindingMetadata attached to each finding (default NoMetadata). Every analyzer implements Dependencies (typically via #[derive(Dependencies)]) so debug builds can check that required analyzers are registered with a lower priority.
  • ContentExtractor<T> — pulls sub-contents out of a container. create_session receives an OwnedContentPtr to the parent and an ExtractionContext describing the region to look at (offset, optional length, optional params), and returns an ExtractionSession. The session then yields children via advance / extract. Per-session state (cursors, open archives, the current Entry) lives on that session object, not on the extractor — one extractor instance is shared and sessions may nest. Ready-made extractors: FolderExtractor<T> (directory walk) and ZipExtractor<T> (ZIP members). Extractors that need a std::io::Read stream wrap the parent in a ContentReader.
  • Filter — decides which paths / sizes should be processed at all.
  • Scanner<T, M> — the orchestrator; built via ScannerBuilder<T> (new() for NoMetadata, or with_metadata::<M>() for typed finding extras).
  • ScanResult<T, M> / ScanContentHandle — after a scan, the framework exposes the full tree of visited objects (parent / child / sibling links), each with its interned path, resolved content type and its own local VarMap, plus a flat list of Findings recorded by analyzers.

Analyzers are either specific to a ContentType or generic (run on every scanned object), and each is registered with a priority byte to control execution order. In debug builds, ScannerBuilder::build also checks each analyzer's Dependencies requires list: every named analyzer must be registered with a strictly smaller priority. Extractors are registered per type and run in registration order — both when the current object is that type, and when an analyzer requests that type.


Getting started

Add the crate to your Cargo.toml (from a local path or git checkout while the crate is not yet published):

[dependencies]
content_scan = { path = "path/to/content_scan/content_scan" }

Bring the prelude into scope:

use content_scan::*;

Define your content types, plug in identifiers / analyzers / extractors, build a Scanner, and call scan(). Analyzers need #[derive(Dependencies)] in addition to ContentAnalyzer.

use content_scan::*;

#[derive(Debug, Copy, Clone, Eq, PartialEq, Ord, PartialOrd, ContentType)]
#[repr(u16)]
enum MyType {
    TextBuffer,
}

// ... implement ContentIdentifier / ContentAnalyzer / ContentExtractor ...

fn main() {
    let mut scanner = ScannerBuilder::<MyType>::new()
        // .add_identifier(MyType::TextBuffer, MyIdentifier {})
        // .add_analyzer(MyType::TextBuffer, 0, MyAnalyzer {})
        // .add_extractor(MyType::TextBuffer, MyExtractor::default())
        .build();

    let mut content = BufferContent::<MyType>::new(b"...", "input.bin");
    // the second argument decides whether the Filter is applied to the root object itself
    let result = scanner.scan(&mut content, true);

    println!("scanned {} objects", result.objects_scanned());
}

Examples

The examples/ directory contains runnable programs. From the workspace root:

cargo run --example vowals
cargo run --example sum
cargo run --example image_size -- path/to/image.png
cargo run --example image_size -- path/to/folder
cargo run --example base64_find
cargo run --example base64_find -- path/to/file_or_folder
cargo run --example md5 -- path/to/file_or_folder
cargo run --example entropy -- path/to/file_or_folder
cargo run --example find_zip -- path/to/folder
cargo run --example zip_png_size -- path/to/archive.zip

Counting vowels

A single analyzer that counts vowels in a magic-tagged buffer. See examples/vowals/main.rs.

use content_scan::*;

#[derive(Debug, Copy, Clone, Eq, PartialEq, Ord, PartialOrd, ContentType)]
#[repr(u16)]
enum MyType {
    TextBuffer,
}

#[derive(Dependencies)]
#[Dependencies(name = "VowelAnalyzer")]
struct VowelAnalyzer;
impl ContentAnalyzer<MyType> for VowelAnalyzer {
    fn analyze(&mut self, content: &mut dyn Content<MyType>, context: &mut Context) -> AnalysisOutcome {
        let sz = content.size();
        let mut count = 0u32;
        for i in 4..sz {
            if let Some(b) = content.read(i, 1) {
                let b = b[0].to_ascii_lowercase();
                if matches!(b, b'a' | b'e' | b'i' | b'o' | b'u') {
                    count += 1;
                }
            }
        }
        context.global().set(var!("count_vowels"), count);
        AnalysisOutcome::Continue
    }
}

struct TextBufferIdentifier;
impl ContentIdentifier<MyType> for TextBufferIdentifier {
    fn identify_method(&self) -> Option<IdentifyMethod> {
        Some(IdentifyMethod::Magic(b"TXBF"))
    }
    fn validate(&self, _: &mut dyn Content<MyType>) -> bool { true }
}

fn main() {
    let mut scanner = ScannerBuilder::new()
        .add_analyzer(MyType::TextBuffer, 0, VowelAnalyzer {})
        .add_identifier(MyType::TextBuffer, TextBufferIdentifier {})
        .build();

    let mut b = BufferContent::<MyType>::with_content_type(
        b"TXBF   Hellow World !", "test.txt", MyType::TextBuffer,
    );
    let res = scanner.scan(&mut b, true);
    println!("count_vowels: {}", res.global().get::<u32>(var!("count_vowels")).unwrap());
}

Summing numbers extracted from text

This example shows the full pipeline — a text container is identified by magic bytes, an extractor pulls numeric substrings out of it as independent Number contents, and a numeric analyzer both sums them into a shared global variable and stashes each value into a per-object local VarMap. After the scan, the example walks the resulting tree via ScanResult. The extractor's cursor lives on an ExtractionSession that holds an OwnedContentPtr to the parent, so a nested extraction can open another session without overwriting the first. See examples/sum/main.rs.

#[derive(Dependencies)]
#[Dependencies(name = "NumericAnalyzer")]
struct NumericAnalyzer;
impl ContentAnalyzer<MyTypes> for NumericAnalyzer {
    fn analyze(&mut self, content: &mut dyn Content<MyTypes>, context: &mut Context) -> AnalysisOutcome {
        let value = u32::from_str_radix(
            std::str::from_utf8(content.read(0, content.size() as u32).unwrap()).unwrap(),
            10,
        ).unwrap();

        // aggregate into the global VarMap...
        if !context.global().update(var!("sum"), |v: &mut u32| *v += value) {
            context.global().set(var!("sum"), value);
        }
        // ...and remember this object's own value in its local VarMap
        context.local().set(var!("value"), value);
        AnalysisOutcome::Continue
    }
}

fn main() {
    let mut scanner = ScannerBuilder::new()
        .add_analyzer(MyTypes::Number, 0, NumericAnalyzer {})
        .add_extractor(MyTypes::Text, NumericExtractor::default())
        .add_identifier(MyTypes::Text, TextIdentifier {})
        .build();

    let mut b = BufferContent::<MyTypes>::new(b"TXT   1+2+3=", "test.txt");
    let res = scanner.scan(&mut b, true);
    println!("sum: {}", res.global().get::<u32>(var!("sum")).unwrap_or(0)); // -> 6

    // Walk the scan tree: root ("test.txt") -> children ("number", "number", "number")
    let root = res.root().unwrap();
    println!("root: {}", res.path(root).unwrap());
    let mut c = res.child(root).unwrap();
    loop {
        let v = res.local(c).unwrap().get::<u32>(var!("value")).unwrap_or(0);
        println!("- child: {} => {}", res.path(c).unwrap(), v);
        match res.next_sibling(c) {
            Some(next) => c = next,
            None => break,
        }
    }
}

Reading PNG / BMP / JPEG dimensions

The image_size example registers one identifier + analyzer pair per image format (png.rs, bmp.rs, jpeg.rs) and each analyzer stores a Size { width, height } in the object's local VarMap. It accepts either a single file or a whole directory:

cargo run --example image_size -- photo.jpg
cargo run --example image_size -- ./pictures

A file is wrapped in a FileContent and scanned directly. A directory is wrapped in a FolderContent tagged with the user's own ImageType::Folder variant, and a recursive FolderExtractor registered for that type turns each directory entry (including nested folders) into a child content object:

let mut scanner = ScannerBuilder::new()
    .filter(
        FilterBuilder::new()
            .include_extensions(Precedence::Medium, &["jpg", "jpeg", "bmp", "png"])
            .deny_the_rest()
            .build(),
    )
    .add_identifier(ImageType::Png, png::PngIdentifier {})
    .add_analyzer(ImageType::Png, 0, png::PngAnalyzer {})
    // ...bmp / jpeg...
    .add_extractor(ImageType::Folder, FolderExtractor::<ImageType>::new(true, false))
    .build();

let res = if Path::new(&path).is_dir() {
    let mut content = FolderContent::<ImageType>::with_content_type(&path, ImageType::Folder);
    scanner.scan(&mut content, false)  // don't filter the root: a folder has no image extension
} else {
    let mut content = FileContent::<ImageType>::new(&path, false);
    scanner.scan(&mut content, true)
};

Note the false passed to scan for the directory case: the filter only allows image extensions, so applying it to the root folder would reject the scan before it starts. The example then prints the resulting tree, indented by depth, with {width} x {height} next to every recognized image.

Finding and decoding Base64

base64_find shows an analyzer requesting extractors of another type. A Text analyzer locates contiguous Base64 runs, emits request_extract(MyTypes::Base64) for each hit, a Base64 extractor decodes the slice into a Vec<u8> tagged Base64Decoded, and a Base64Decoded analyzer prints that buffer. With no arguments it scans a built-in sample; pass a file or folder to scan real content:

cargo run --example base64_find
cargo run --example base64_find -- notes.txt
cargo run --example base64_find -- ./inbox
#[derive(Dependencies)]
#[Dependencies(name = "Base64Finder")]
struct Base64Finder;
impl ContentAnalyzer<MyTypes> for Base64Finder {
    fn analyze(&mut self, content: &mut dyn Content<MyTypes>, context: &mut Context<MyTypes>) -> AnalysisOutcome {
        // ...locate a run at `start` of length `len`...
        context.request_extract(MyTypes::Base64).at(start).len(len).emit();
        AnalysisOutcome::Continue
    }
}

impl ContentExtractor<MyTypes> for Base64Extractor {
    fn create_session(
        &mut self,
        content: OwnedContentPtr<MyTypes>,
        extract_context: &ExtractionContext,
    ) -> Option<Box<dyn ExtractionSession<MyTypes>>> {
        let length = extract_context.length.unwrap_or(content.size().saturating_sub(extract_context.offset));
        if length == 0 {
            return None;
        }
        Some(Box::new(Base64Session {
            content,
            offset: extract_context.offset,
            length,
            done: false,
            entry: Entry::default(),
        }))
    }
}

impl ExtractionSession<MyTypes> for Base64Session {
    fn extract(&mut self) -> Option<Box<dyn Content<MyTypes>>> {
        let encoded = read_range(&mut *self.content, self.offset, self.length)?;
        let decoded = decode_base64(&encoded)?;
        Some(Box::new(BufferContent::<MyTypes>::from_parts(
            decoded,
            format!("base64@{}", self.offset),
            Some(MyTypes::Base64Decoded),
        )))
    }
    // ...
}

#[derive(Dependencies)]
#[Dependencies(name = "Base64DecodedAnalyzer")]
struct Base64DecodedAnalyzer;
impl ContentAnalyzer<MyTypes> for Base64DecodedAnalyzer {
    fn analyze(&mut self, content: &mut dyn Content<MyTypes>, _: &mut Context<MyTypes>) -> AnalysisOutcome {
        let buf = content.read(0, content.size() as u32).unwrap_or(&[]);
        println!("{}: {}", content.path().as_printable_string(), String::from_utf8_lossy(buf));
        AnalysisOutcome::Continue
    }
}

The extractor is registered for Base64, not for Text. It only runs because the finder queued that type; the parent file itself is never identified as Base64.

Computing MD5 hashes

md5 hashes every file with a generic analyzer. ComputeHashAnalyzer is registered with add_generic_analyzer, so it runs on every scanned object regardless of type. Instead of printing from the analyzer, it records each digest as a Finding via context.add_finding. After the scan, the example iterates res.findings(). A recursive FolderExtractor walks the tree when the argument is a directory; folders themselves are skipped (they have no bytes to hash).

cargo run --example md5 -- README.md
cargo run --example md5 -- ./content_scan/src
#[derive(Dependencies)]
#[Dependencies(name = "ComputeHashAnalyzer")]
struct ComputeHashAnalyzer;
impl ContentAnalyzer<MyTypes> for ComputeHashAnalyzer {
    fn analyze(&mut self, content: &mut dyn Content<MyTypes>, context: &mut Context<MyTypes>) -> AnalysisOutcome {
        if content.content_type() == Some(MyTypes::Folder) {
            return AnalysisOutcome::Continue;
        }
        // read the file in chunks, feed Md5...
        context.add_finding(format!("{:x}", hasher.finalize()).as_str(), None, None);
        AnalysisOutcome::Continue
    }
}

fn main() {
    let mut scanner = ScannerBuilder::new()
        .add_generic_analyzer(0, ComputeHashAnalyzer {})
        .add_extractor(MyTypes::Folder, FolderExtractor::<MyTypes>::new(true, false))
        .build();
    // FileContent for a file, FolderContent + Folder extractor for a directory
    let res = scanner.scan(&mut content, true);
    for f in res.findings() {
        println!("{}  {}", f.finding(), f.path().unwrap_or_default());
    }
}

Measuring Shannon entropy

entropy is the same generic-analyzer + findings pattern as md5, but each finding carries typed FindingMetadata: a newtype Entropy(f64) with the Shannon entropy (bits per byte) of that file. The scanner is built with ScannerBuilder::with_metadata::<Entropy>(), and the analyzer implements ContentAnalyzer<MyTypes, Entropy>. The finding text is a coarse classification — packed if entropy is greater than 7.8, encrypted if it is greater than 7.0, otherwise normal — while the exact value lives in the metadata.

cargo run --example entropy -- README.md
cargo run --example entropy -- ./content_scan/src
#[derive(Copy, Clone, Debug)]
struct Entropy(f64);
impl FindingMetadata for Entropy {}

#[derive(Dependencies)]
#[Dependencies(name = "EntropyAnalyzer")]
struct EntropyAnalyzer;
impl ContentAnalyzer<MyTypes, Entropy> for EntropyAnalyzer {
    fn analyze(&mut self, content: &mut dyn Content<MyTypes>, context: &mut Context<MyTypes, Entropy>) -> AnalysisOutcome {
        if content.content_type() == Some(MyTypes::Folder) {
            return AnalysisOutcome::Continue;
        }
        // count byte frequencies, compute Shannon entropy...
        let label = if entropy > 7.8 { "packed" } else if entropy > 7.0 { "encrypted" } else { "normal" };
        context.add_finding(label, None, Some(Entropy(entropy)));
        AnalysisOutcome::Continue
    }
}

fn main() {
    let mut scanner = ScannerBuilder::<MyTypes>::with_metadata::<Entropy>()
        .add_generic_analyzer(0, EntropyAnalyzer {})
        .add_extractor(MyTypes::Folder, FolderExtractor::<MyTypes>::new(true, false))
        .build();
    // FileContent for a file, FolderContent + Folder extractor for a directory
    let res = scanner.scan(&mut content, true);
    for f in res.findings() {
        let entropy = f.metadata().map(|m| m.0).unwrap_or(0.0);
        println!("{}  {:.4}  {}", f.finding(), entropy, f.path().unwrap_or_default());
    }
}

Listing ZIP files

find_zip walks a directory with FolderExtractor and uses the built-in ZipIdentifier (local-file magic PK\x03\x04, then an EOCD check in validate) to decide which files are ZIPs. A type-specific analyzer prints each match. Identification is by content, not by .zip extension.

cargo run --example find_zip -- ./archives
let mut scanner = ScannerBuilder::new()
    .add_identifier(MyTypes::Zip, ZipIdentifier::new())
    .add_analyzer(MyTypes::Zip, 0, ZipPrinter {})
    .add_extractor(MyTypes::Folder, FolderExtractor::<MyTypes>::new(true, false))
    .build();

Reading PNG dimensions inside a ZIP

zip_png_size combines ZipIdentifier + ZipExtractor with a PNG identifier / analyzer. The ZIP is the root (scan(..., false) so the filter is not applied to the archive itself); the extractor emits each member as a child, the filter keeps only *.png, and the PNG analyzer stores {width, height} in that child's local VarMap.

cargo run --example zip_png_size -- photos.zip
let mut scanner = ScannerBuilder::new()
    .filter(
        FilterBuilder::new()
            .include_extensions(Precedence::Medium, &["png"])
            .deny_the_rest()
            .build(),
    )
    .add_identifier(MyTypes::Zip, ZipIdentifier::new())
    .add_extractor(MyTypes::Zip, ZipExtractor::new())
    .add_identifier(MyTypes::Png, PngIdentifier)
    .add_analyzer(MyTypes::Png, 0, PngAnalyzer)
    .build();

let mut content = FileContent::<MyTypes>::new(&path, false);
let res = scanner.scan(&mut content, false); // don't filter the ZIP itself

API overview

ContentType

Every scanner is parameterized by a user-defined enum implementing ContentType. The #[derive(ContentType)] macro implements the trait automatically. Requirements:

  • The enum must be annotated with #[repr(u16)].
  • Variants must be unit variants with no explicit discriminants.
  • Derive Ord and PartialOrd as well — the trait requires them.
  • Up to 65536 variants are supported.
#[derive(Debug, Copy, Clone, Eq, PartialEq, Ord, PartialOrd, ContentType)]
#[repr(u16)]
enum MyTypes {
    Number,
    Text,
}

bool also implements ContentType and is used internally by the matcher / filter machinery.

Content

pub trait Content<T: ContentType> {
    fn content_type(&self) -> Option<T> { None }
    fn set_content_type(&mut self, ty: T);
    fn path(&self) -> &ContentPath;
    fn size(&self) -> u64;
    fn read(&mut self, offset: u64, count: u32) -> Option<&[u8]>;
}

A Content is any addressable byte source. It exposes a ContentPath (used for identification, filtering, and display), a size, and a read(offset, count) method that returns a borrowed slice. set_content_type is what the scanner calls when an analyzer returns ReprocessAsType; built-in contents store the new type so the next analysis pass skips identifiers.

Three implementations ship with the crate — an in-memory one, a file-backed one, and a directory marker:

BufferContent::<MyType>::new(buffer, "path.ext");
BufferContent::<MyType>::from_vec(owned_vec, "path.ext");          // move, no copy
BufferContent::<MyType>::with_content_type(buffer, "path.ext", MyType::Text);
BufferContent::<MyType>::from_parts(vec, "path".into(), Some(MyType::Text));

FileContent::<MyType>::new("path/to/file.bin", false);           // impl AsRef<Path>; false = shared read
FileContent::<MyType>::with_content_type("path/to/file.bin", MyType::Text, false);
FileContent::<MyType>::with_size("path/to/file.bin", 4096, false);   // size already known, skip the stat

FolderContent::<MyType>::with_content_type("path/to/dir", MyType::Folder);

BufferContent constructors take a &str and treat it as a synthetic address (always UTF-8). FileContent and FolderContent take impl AsRef<Path> and go through ContentPath::from_os, so a real filesystem name that is not valid UTF-8 stays openable.

FileContent opens the file lazily on the first read(). Pass exclusive = true to memory-map with an exclusive lock (files already open elsewhere will fail). Pass false for shared access through an LRU page cache. with_size is useful when the size is already known (a directory walk has just stat'ed the entry, for instance) and you want to avoid a second filesystem call.

FolderContent represents a directory rather than bytes: its size() is 0 and read() always returns None. It always reports the content type you give it, so the scanner dispatches straight to the extractor registered for that type — see Walking the file system.

You can also implement Content<T> for memory-mapped regions, network streams, archive entries, etc.

ContentPath

ContentPath is the path type used everywhere a piece of content is named: Content::path(), Entry::path, filter callbacks, and (as interned bytes) the scan result tree. It covers both virtual addresses (archive.zip://inner/file.txt, "number") and real OS paths, including names that are not valid UTF-8.

pub struct ContentPath { /* ... */ }

impl ContentPath {
    pub fn from_str(s: &str) -> Self;            // synthetic / known UTF-8
    pub fn from_os(p: &Path) -> Self;            // real filesystem path
    pub fn empty() -> Self;                      // reusable empty buffer
    pub fn clear(&mut self);                     // reset, keep capacity
    pub fn set_from_str(&mut self, s: &str);     // reuse allocation
    pub fn set_from_os(&mut self, p: &Path);     // reuse allocation
    pub fn as_printable_string(&self) -> &str;   // always valid UTF-8 (lossy if needed)
    pub fn as_path(&self) -> &Path;              // openable OS path
    pub fn as_bytes(&self) -> &[u8];             // filtering / identification
    pub fn is_lossless(&self) -> bool;
}
  • from_str / set_from_str — use for archive members, in-memory buffers, and any label you already know is UTF-8. Do not stringify a real OS path and pass it here: a non-UTF-8 filesystem name would lose the bytes needed to reopen it.
  • from_os / set_from_os — use for DirEntry, PathBuf, and anything that came from the filesystem. When the path is valid UTF-8 only the string is stored. Otherwise the original OsString is kept alongside a lossy printable view (U+FFFD for invalid sequences), so as_path() still names the original file and is_lossless() is false.
  • empty / clear — construct or reset an empty path while keeping the string allocation. Sessions typically clear then set_from_os / set_from_str on a reused Entry.
  • as_printable_string — always available, never fails. Safe to log or print. Exact only when is_lossless() is true.
  • as_path — the &Path to hand to fs::read_dir, File::open, and similar. For a non-UTF-8 path this is the preserved OS name, not the lossy string.
  • as_bytes — what identification and Filter inspect (file name / extension). On Unix this is the faithful path bytes; on Windows it is the printable string's bytes.

ContentPath implements From for &str, String, &String, &Path, PathBuf, and &PathBuf. UTF-8 strings go through from_str (or with_string when already owned); OS paths go through from_os. FileContent and FolderContent take impl AsRef<Path>, so String and PathBuf work at those constructors too.

FolderExtractor fills each Entry with set_from_os, so a directory walk does not drop non-UTF-8 names the way a to_str().unwrap_or_default() conversion would.

ContentIdentifier

Identifiers tell the scanner what type a piece of content is when the content itself doesn't already report one.

pub trait ContentIdentifier<T: ContentType> {
    fn identify_method(&self) -> Option<IdentifyMethod>;
    fn validate(&self, content: &mut dyn Content<T>) -> bool;
}

pub enum IdentifyMethod {
    Magic(&'static [u8]),
    MultipleMagic(&'static [&'static [u8]]),
    Extension(&'static str),
    Extensions(&'static [&'static str]),
    Name(&'static str),
    Names(&'static [&'static str]),
}

Fast identification is performed with an internal matcher (single-pattern, packed magic table, or trie depending on the number and shape of patterns). After a fast match, validate() is called to confirm the guess. Overlapping magic prefixes resolve to the longest match in both the packed table and the trie.

Extension and file-name methods are ASCII case-insensitive: both the registered pattern and the path basename / extension are compared in lowercase. Notes.TXT matches Extension("txt"); makefile matches Name("Makefile"). Magic methods remain exact byte matches. A magic sequence (Magic / MultipleMagic) must be at most 16 bytes — that is the window the scanner reads (content.read(0, 16)). ScannerBuilder::build panics if a registered magic is longer. To inspect bytes past that window, return None from identify_method and call content.read in validate.

If identify_method returns None, validate() is still called — after magic, file name, and extension have all been considered — so you can classify content with custom logic (payload bytes via content.read, path shape, size, …). Those identifiers are tried in the order they were registered.

At most one identifier per ContentType may be registered; the builder will panic otherwise. It also panics if a Magic / MultipleMagic pattern is longer than 16 bytes.

ZipIdentifier

ZipIdentifier<T> is a built-in identifier for ZIP archives. Fast identification is the local-file magic PK\x03\x04; validate then looks for an End of Central Directory record in the tail of the content, so a file that merely starts with those four bytes is rejected. Identification is by content, not by .zip extension.

.add_identifier(MyTypes::Zip, ZipIdentifier::new())

ZipExtractor

ZipExtractor<T> is a built-in extractor that unpacks ZIP members as child content. Pair it with ZipIdentifier. Small entries (< 1 MiB) become an in-memory BufferContent; larger ones are decompressed to a temp file and wrapped in FileContent. Directory entries inside the archive are skipped. The session reads the parent through a ContentReader, so the archive can be a file, a buffer, or any other Content.

.add_identifier(MyTypes::Zip, ZipIdentifier::new())
.add_extractor(MyTypes::Zip, ZipExtractor::new())

See Reading PNG dimensions inside a ZIP for a complete pipeline.

Dependencies

Every ContentAnalyzer must implement Dependencies. The #[derive(Dependencies)] macro (re-exported from content_scan) does that from a helper attribute:

#[derive(Dependencies)]
#[Dependencies(name = "NeedsHash", requires = "ComputeHash")]
struct NeedsHash;

#[derive(Dependencies)]
#[Dependencies(name = "ComputeHash", requires = ["OpenFile", "MapPages"])]
struct ComputeHash;

#[derive(Dependencies)]
#[Dependencies(name = "OpenFile")]
struct OpenFile;
  • name is required and must be a non-empty string. Other analyzers refer to this plugin by that name in requires.
  • requires is optional. It may be a single string or an array of strings naming analyzers that must run first.

The derived name() / dependencies() methods exist only when debug_assertions are enabled. In debug builds, ScannerBuilder::build checks that:

  1. Every name listed in requires is a registered analyzer (typed or generic — names are global, not per ContentType).
  2. Each required analyzer has a strictly smaller priority than the one that requires it (lower numbers run first).

Release builds skip the check. Duplicate names among registered analyzers are not rejected; the last registration wins for the debug map.

Generics are not supported by the derive. Unit structs, tuple structs, named-field structs, enums, and unions all work.

ContentAnalyzer

pub trait ContentAnalyzer<T: ContentType, M: FindingMetadata = NoMetadata>: Dependencies {
    fn analyze(&mut self, content: &mut dyn Content<T>, context: &mut Context<T, M>) -> AnalysisOutcome;
}

Analyzers inspect content and write results into the shared Context. Use context.local() for per-object VarMaps, context.global() for scan-wide aggregates, and context.add_finding(...) for a flat list of Findings. To pull nested content out of the current object using extractors registered for a different type — for example an analyzer that locates an embedded ZIP and wants the Zip extractor to open it — call context.request_extract(ty) and emit an extraction request. Only analyzers return AnalysisOutcome; that value controls the rest of this object:

  • AnalysisOutcome::Continue — run the next analyzer for this object; after the last analyzer, run own-type extractors then requested extractors.
  • AnalysisOutcome::Skip — stop this object: do not run remaining analyzers or any extractors on it. Siblings and later objects still scan.
  • AnalysisOutcome::Exit — abort the entire scan.
  • AnalysisOutcome::ReprocessAsType(ty) — stop remaining analyzers for this pass, run only requested extractors, then (if this object has changed type fewer times than max_change_type, default 2) assign ty on the same tree node and re-run typed + generic analyzers. Identifiers are not re-run. A generic analyzer that returns this is currently treated as Continue.

Register analyzers with:

  • add_analyzer(content_type, priority, analyzer) — runs only when the content matches content_type.
  • add_generic_analyzer(priority, analyzer) — runs for every scanned object, regardless of type.

Within a bucket, analyzers execute in ascending priority order. If an analyzer requires another (see Dependencies), register that dependency with a smaller priority; debug builds enforce this in build().

ContentExtractor

pub trait ContentExtractor<T: ContentType> {
    fn create_session(
        &mut self,
        content: OwnedContentPtr<T>,
        extract_context: &ExtractionContext,
    ) -> Option<Box<dyn ExtractionSession<T>>>;
}

pub struct Entry {
    pub path: ContentPath,
    pub size: u64,
    pub skip_from_filtering: bool,
}

Extractors turn a container into a stream of children. The scanner calls create_session once per parent; the returned ExtractionSession then enumerates children. Methods return Option — they do not return AnalysisOutcome and cannot Skip or Exit on their own.

create_session is called after that object's analyzers have run (Continue: own-type then requested extractors; ReprocessAsType: requested extractors only). It receives:

  • an OwnedContentPtr to the parent — store it on the session if advance / extract need to read the parent;
  • an ExtractionContext describing the region to look at (offset, optional length, optional params). Copy those fields into the session; the context is only valid for this call.

Return Some(session) to start enumerating, or None to skip this extractor (the scanner moves on to the next one registered for the same type). The session is dropped when enumeration ends, when a nested child's analyzer returns AnalysisOutcome::Exit, or when the scanner moves on. Implement Drop on the session if you need to close files or free buffers.

An extractor registered for type T runs in two situations:

  • The current object was identified as T. The context then covers the whole object (offset = 0, length = Some(content.size()), params = None).
  • An analyzer requested extraction of T from the current object via context.request_extract. The context then carries the requested offset, length, and params. The parent does not need to have been identified as T.

One extractor instance is shared by every object of its type, and create_session can be re-entered while a previous session is still live (nested containers). Keep configuration on the extractor and put cursors, open archives, and the current Entry on the session. Extractors are registered with add_extractor for a specific ContentType. Multiple extractors for the same type run in registration order.

ExtractionSession

pub trait ExtractionSession<T: ContentType> {
    fn advance(&mut self) -> Option<&Entry>;
    fn extract(&mut self) -> Option<Box<dyn Content<T>>>;
}

A session is a live extraction of children from one parent. The scanner drives it as a short loop:

  1. advance — move to the next child and return a lightweight Entry describing its path/size. Returning None ends the stream and the session is dropped.
  2. extract — materialize that child as a boxed Content<T>. The scanner then recursively scans it (subject to max_depth). Returning None skips just this entry; enumeration continues with the next advance.

A child's Skip does not end this session — the scanner continues with the next advance. A child's Exit drops the session immediately as the scan unwinds.

Set Entry::skip_from_filtering to true to exempt an entry from the active Filter. This matters for container entries that would never pass the filter themselves: a FolderExtractor restricted to *.png files, for example, still has to descend into subdirectories, whose names carry no .png extension. Keep one Entry as a field on the session and overwrite entry.path in place with ContentPath::set_from_str (synthetic names) or ContentPath::set_from_os (real OS paths) so advance does not allocate a new path for every child.

OwnedContentPtr

OwnedContentPtr<T> is the handle create_session receives instead of a &mut dyn Content<T>. It does not own the parent — it is a non-owning pointer that derefs to dyn Content<T>, so a session can call path(), size(), read(), and content_type() through it across advance / extract without fighting the borrow checker.

The scanner guarantees the parent outlives the session. Do not store the pointer after the session is dropped.

A typical extractor therefore looks like this (the sum example, simplified):

struct NumericExtractor;

struct NumericSession {
    content: OwnedContentPtr<MyTypes>,
    pos: u64,
    start: u64,
    len: u64,
    entry: Entry,
}

impl ContentExtractor<MyTypes> for NumericExtractor {
    fn create_session(
        &mut self,
        content: OwnedContentPtr<MyTypes>,
        _: &ExtractionContext,
    ) -> Option<Box<dyn ExtractionSession<MyTypes>>> {
        Some(Box::new(NumericSession {
            content,
            pos: 0,
            start: u64::MAX,
            len: 0,
            entry: Entry::default(),
        }))
    }
}

impl ExtractionSession<MyTypes> for NumericSession {
    fn advance(&mut self) -> Option<&Entry> {
        // ...scan forward through `self.content`, updating `self.pos` / `self.start` / `self.len`...
        self.entry.path.set_from_str("number");
        self.entry.size = self.len;
        self.entry.skip_from_filtering = false;
        Some(&self.entry)
    }
    fn extract(&mut self) -> Option<Box<dyn Content<MyTypes>>> {
        let buf = self.content.read(self.start, self.len as u32)?;
        Some(Box::new(BufferContent::<MyTypes>::with_content_type(buf, "number", MyTypes::Number)))
    }
}

ContentReader

ContentReader<T> is a sequential std::io::Read + Seek adapter over an OwnedContentPtr. Content::read is random-access and returns a borrowed slice; libraries that expect a stream (the zip crate used by ZipExtractor, parsers, decompressors, …) need a cursor instead.

pub struct ContentReader<T: ContentType> { /* ... */ }

impl<T: ContentType> ContentReader<T> {
    pub fn new(content: OwnedContentPtr<T>) -> Self;  // cursor starts at offset 0
}

impl<T: ContentType> Read for ContentReader<T> { /* ... */ }
impl<T: ContentType> Seek for ContentReader<T> { /* ... */ }

A short slice from the underlying Content is not treated as EOF: the adapter copies what it got, advances, and the next Read::read continues from there. A Content::read that returns None before the advertised size() becomes an UnexpectedEof error. Seeking past the end is allowed (same as std::io::Cursor); a later read then returns Ok(0).

fn create_session(
    &mut self,
    content: OwnedContentPtr<MyTypes>,
    _: &ExtractionContext,
) -> Option<Box<dyn ExtractionSession<MyTypes>>> {
    let archive = zip::ZipArchive::new(ContentReader::new(content)).ok()?;
    Some(Box::new(MyZipSession { archive, ... }))
}

ExtractionContext

ExtractionContext is what create_session receives. It names a window inside the parent Content plus an optional parameter map:

pub struct ExtractionContext<'a> {
    pub offset: u64,               // start byte within the parent
    pub length: Option<u64>,       // Some(n) = known size; None = extractor decides
    pub params: Option<&'a VarMap>, // analyzer-supplied extras, or None
}
  • offset — first byte of the region. Type-specific extraction of the parent itself always starts at 0.
  • length — Some(n) when the caller knows the region is n bytes. None means the extractor determines the extent itself (parse until the format ends, scan to EOF, …). Type-specific extraction of the parent itself passes Some(content.size()).
  • params — extras an analyzer attached with .param(...) (password, codec, flags, …). None when the request had no .param() calls, including type-specific extraction of the parent itself. The borrow lasts only for create_session — copy values into the session if you need them later.

Requesting extraction

Analyzers that find nested content of another type queue a pass with context.request_extract(ty). The call returns an ExtractRequestBuilder; chain setters and commit with .emit(). Dropping the builder without emit does nothing.

impl ContentAnalyzer<MyTypes> for PeAnalyzer {
    fn analyze(&mut self, _: &mut dyn Content<MyTypes>, context: &mut Context<MyTypes>) -> AnalysisOutcome {
        // Found an embedded ZIP inside this PE; run Zip extractors on that slice.
        context.request_extract(MyTypes::Zip)
            .at(0x1000)                          // offset within the parent
            .len(4096)                           // optional; omit if unknown
            .param(var!("password"), "secret")   // optional; repeatable
            .emit();
        AnalysisOutcome::Continue
    }
}

Builder methods:

Method Effect
at(offset) Byte offset within the parent. Defaults to 0.
len(n) Asserts the region is n bytes. Omit to leave length = None.
param(key, value) Adds one extractor-specific parameter. The first call reserves a pooled VarMap; later calls write into the same map. A request with no .param() carries no map (params = None).
emit() Commits the request. Required — the builder is #[must_use].

After this object's analyzers finish with Continue, the scanner:

  1. Runs extractors registered for the object's own identified type (whole object).
  2. Then, in emission order, runs extractors registered for each requested type on the same parent, with that request's ExtractionContext.

Skip or Exit skips extractors on this object, including queued requests. ReprocessAsType runs only the requested extractors (own-type extractors for the old type are skipped), then may re-analyze the same object as the new type.

The current object does not need to have been identified as the requested type. Several requests (of the same or different types) may be emitted from one analyzer; each is independent. Nested child scans start with an empty request queue.

The Zip extractor then copies the region out of create_session:

fn create_session(
    &mut self,
    content: OwnedContentPtr<MyTypes>,
    ctx: &ExtractionContext,
) -> Option<Box<dyn ExtractionSession<MyTypes>>> {
    let start = ctx.offset;
    let len = ctx.length; // None = parse until the ZIP ends
    let password = ctx.params.and_then(|p| p.get::<&str>(var!("password"))).map(str::to_string);
    Some(Box::new(ZipSession { content, start, len, password, ... }))
}

Walking the file system

FolderExtractor<T> is a built-in extractor that enumerates a directory and emits one child per entry. Pair it with a FolderContent root and a ContentType variant of your own that stands for "directory":

let mut scanner = ScannerBuilder::<MyType>::new()
    .add_extractor(MyType::Folder, FolderExtractor::<MyType>::new(true, false)) // recursive, shared file opens
    .add_identifier(MyType::Png, PngIdentifier {})
    .add_analyzer(MyType::Png, 0, PngAnalyzer {})
    .build();

let mut root = FolderContent::<MyType>::with_content_type("C:/pictures", MyType::Folder);
let res = scanner.scan(&mut root, false);

Behaviour worth knowing:

  • The recursive flag passed to new decides whether subdirectories are emitted at all. When false, only files directly inside the folder are scanned.
  • Subdirectories are emitted as FolderContent carrying the same content type as their parent, so the same extractor picks them up again. Files become FileContent built with with_size, reusing the size from the directory entry's metadata. Each entry's path is filled with ContentPath::set_from_os, so non-UTF-8 filesystem names stay openable.
  • Directory symlinks (and, on Windows, directory junctions / reparse points) are skipped, which keeps cyclic link structures from looping forever. Symlinks to regular files are followed and scanned like any other file; dangling links are skipped.
  • An unreadable entry (permission error, failed file_type) is skipped; the rest of the directory is still enumerated.
  • Subdirectory entries are marked skip_from_filtering, so an extension-based Filter narrows down the files without preventing the walk from descending.
  • Recursion is still bounded by the scanner's max_depth (default 8), which here translates into directory nesting levels.

Filter / FilterBuilder

Filters decide whether a given (ContentPath, size) should be processed. They combine typed rules (extension / file-name allow- and deny-lists, powered by fast matchers) with arbitrary predicate callbacks, plus a default fallback:

use content_scan::*;

let filter = FilterBuilder::new()
    .include_extensions(Precedence::High,   &["rs", "toml"])
    .exclude_file_names(Precedence::Medium, &["Cargo.lock"])
    .exclude(Precedence::Low, |_path, size| size > 10 * 1024 * 1024) // skip huge files
    .deny_the_rest()   // or .allow_the_rest()
    .build();

let scanner = ScannerBuilder::<MyType>::new()
    .filter(filter)
    // ...
    .build();

Available builder methods:

Method Effect
include_extensions(prec, &["ext", …]) Allow if the file's extension matches (ASCII case-insensitive).
exclude_extensions(prec, &["ext", …]) Deny if the file's extension matches (ASCII case-insensitive).
include_file_names(prec, &["name", …]) Allow if the basename matches (ASCII case-insensitive).
exclude_file_names(prec, &["name", …]) Deny if the basename matches (ASCII case-insensitive).
include(prec, fn(&ContentPath, u64) -> bool) Allow if the callback returns true.
exclude(prec, fn(&ContentPath, u64) -> bool) Deny if the callback returns true.
deny_the_rest() / allow_the_rest() Set the default and finalize the builder.

Rules are grouped by Precedence and evaluated from Highest to Lowest. Within the same precedence, rules keep the order they were added. The first matching rule wins. Extension and file-name rules are ASCII case-insensitive (Photo.JPG matches jpg).

Filters are also applied to entries produced by extractors, so filtering works transparently for embedded content.

Scanner / ScannerBuilder

Assemble a scanner with ScannerBuilder:

let scanner = ScannerBuilder::<MyType>::new()
    .filter(filter)                                   // optional
    .max_depth(8)                                     // default: 8
    .max_change_type(2)                               // default: 2
    .add_identifier(MyType::Text, TextIdentifier {})
    .add_analyzer(MyType::Text, 0, MyAnalyzer {})
    .add_generic_analyzer(10, LoggingAnalyzer {})
    .add_extractor(MyType::Zip, ZipExtractor::default())
    .build();

Then scan:

let result: ScanResult = scanner.scan(&mut content, /* filter_root */ true);

max_depth limits how deep the scanner is allowed to recurse into extracted children (default 8, minimum 1). The root is depth 1; a child of a depth-N object is depth N + 1. Extraction stops when that next child would exceed max_depth, so max_depth(8) visits at most eight objects on any path.

max_change_type limits how many times one object may ReprocessAsType (default 2, clamped to 1..=T::COUNT). Each accepted change assigns the new type on the same tree node and re-runs analyzers; identifiers are not re-run. Over the cap, reprocessing stops and the object keeps its current type.

The second argument to scan decides whether the configured Filter is applied to the root object itself. Pass true for a normal file — the scan then returns an empty ScanResult if the filter rejects it. Pass false when the root is a container that the filter was never written to accept, such as a folder being walked with a filter that only allows png files. Extracted children are always filtered regardless of this flag (unless their Entry opts out via skip_from_filtering).

A scanner is reusable: scan clears its internal Context on entry (including the findings list), so one instance can process many inputs in sequence.

ScannerBuilder::new() produces a scanner whose findings carry no extra data (NoMetadata). When analyzers attach typed extras (severity, offset, rule id, …), start from ScannerBuilder::<MyType>::with_metadata::<M>() instead — see Findings.

build() panics if two identifiers are registered for the same ContentType, or if a magic pattern is longer than 16 bytes. In debug builds it also panics if an analyzer requires a name that is not registered, or if that dependency does not have a strictly smaller priority — see Dependencies.

Context / ScanResult

The Context passed to analyzers (from the varmap crate, re-exported by content_scan) exposes two VarMaps, a findings list, plus a way to queue extra extraction:

  • context.global() — persists for the entire scan() call. Use it to accumulate results across all analyzed objects.
  • context.local() — per-object scratch storage. The first call from an analyzer on a given object lazily grabs a VarMap from an internal pool, clears it, and attaches it to that object; subsequent calls (from other analyzers running on the same object) return the same map. It is kept alive after the scan and can be looked up on the corresponding ScanContentHandle via ScanResult::local(handle).
  • context.add_finding(text, source, metadata) — records a Finding on the current object. After the scan, iterate them with res.findings().
  • context.request_extract(ty) — queues an extra extraction pass. On Continue, those extractors run after this object's own extractors; on ReprocessAsType, they run instead. See Requesting extraction. The request queue is cleared at the start of every object's scan, including nested children.

context.objects_scanned() returns how many objects have been visited so far.

After scan() returns, you can read results from the ScanResult<T>. In addition to the classic aggregate view, it also exposes the full tree of scanned objects:

let res = scanner.scan(&mut content, true);
let sum = res.global().get::<u32>(var!("sum")).unwrap_or(0);
println!("scanned {} objects, sum = {}", res.objects_scanned(), sum);

var!("name") is a compile-time typed key macro provided by varmap. Custom types can be stored in a VarMap by deriving VarMapValue, which is re-exported from content_scan as well:

#[derive(Debug, Copy, Clone, Eq, PartialEq, VarMapValue)]
pub struct Size { pub width: u32, pub height: u32 }

context.local().set(var!("size"), Size { width, height });

Findings

Analyzers can emit a flat list of findings in addition to (or instead of) writing VarMaps. A finding is a short interned string attached to the object that was current when it was recorded, plus an optional source label and optional typed metadata.

Record one during analyze:

context.add_finding("deadbeef", Some("ComputeHash"), None);
Argument Meaning
finding The text itself (hash digest, message, match string, …). Interned into the scanner's path arena.
source Optional label for who produced it (rule name, plugin id). None if unused.
metadata Optional typed extra of type M. Pass None when using the default NoMetadata.

Calling add_finding when no object is current (outside an analyzer) is a no-op. After scan() returns, iterate findings in emission order:

for f in res.findings() {
    println!("{}  {}", f.finding(), f.path().unwrap_or("?"));
    if let Some(src) = f.source() {
        println!("  from {src}");
    }
}
pub struct Finding<'a, T: ContentType, M: FindingMetadata> { /* ... */ }

impl<'a, T: ContentType, M: FindingMetadata> Finding<'a, T, M> {
    pub fn finding(&self) -> &'a str;           // the recorded text
    pub fn source(&self) -> Option<&'a str>;    // optional producer label
    pub fn metadata(&self) -> Option<&'a M>;    // typed extras, if any
    pub fn path(&self) -> Option<&'a str>;      // printable path of the object
    pub fn content_type(&self) -> Option<T>;    // identified type of the object
}

ScanResult::findings() returns a FindigsIterator that yields Findings. The iterator (and each Finding) borrows the ScanResult; do not start another scan on the same Scanner while they are still in use. The finding text and source stay valid for the lifetime of that result because they are interned in the same arena as object paths.

The default metadata type is NoMetadata — ScannerBuilder::new() and ContentAnalyzer<T> use it, and analyzers pass None as the third argument. To attach typed extras (severity, offset, rule id, …), implement the FindingMetadata marker trait (Copy is required so findings stay cheap) and build the scanner with with_metadata:

#[derive(Copy, Clone, Debug)]
enum Severity { Info, Warn, Error }
impl FindingMetadata for Severity {}

#[derive(Dependencies)]
#[Dependencies(name = "MyAnalyzer")]
struct MyAnalyzer;
impl ContentAnalyzer<MyTypes, Severity> for MyAnalyzer {
    fn analyze(&mut self, _: &mut dyn Content<MyTypes>, context: &mut Context<MyTypes, Severity>) -> AnalysisOutcome {
        context.add_finding("suspicious overlay", Some("MyAnalyzer"), Some(Severity::Warn));
        AnalysisOutcome::Continue
    }
}

let mut scanner = ScannerBuilder::<MyTypes>::with_metadata::<Severity>()
    .add_generic_analyzer(0, MyAnalyzer {})
    .build();

The same M flows through Scanner<T, M>, Context<T, M>, ScanResult<T, M>, and ContentAnalyzer<T, M>. See Computing MD5 hashes for findings without custom metadata, and Measuring Shannon entropy for a complete example that attaches a f64 via FindingMetadata.

Navigating the scan result tree

Every object visited by the scanner is recorded, along with its resolved content type, its path (interned from ContentPath::as_printable_string() into an internal arena) and its optional local VarMap. Objects are linked as a parent / first-child / next-sibling tree that mirrors the extraction hierarchy.

You navigate the tree with opaque ScanContentHandles returned by ScanResult<T>:

pub struct ScanContentHandle { /* opaque */ }

impl<'a, T: ContentType, M: FindingMetadata> ScanResult<'a, T, M> {
    pub fn global(&self) -> &VarMap;
    pub fn objects_scanned(&self) -> u32;
    pub fn findings(&self) -> FindigsIterator<'a, T, M>;

    // Tree navigation
    pub fn root(&self) -> Option<ScanContentHandle>;
    pub fn parent(&self, handle: ScanContentHandle) -> Option<ScanContentHandle>;
    pub fn child(&self, handle: ScanContentHandle) -> Option<ScanContentHandle>;
    pub fn next_sibling(&self, handle: ScanContentHandle) -> Option<ScanContentHandle>;

    // Per-object data
    pub fn path(&self, handle: ScanContentHandle) -> Option<&str>;
    pub fn content_type(&self, handle: ScanContentHandle) -> Option<T>;
    pub fn local(&self, handle: ScanContentHandle) -> Option<&VarMap>;
}

ScanResult::path is the interned printable view of the object's ContentPath (as_printable_string() at scan time). For a live content object, use content.path().as_printable_string() to display it or content.path().as_path() to open it.

Typical walk:

fn dump<T: ContentType>(res: &ScanResult<T>, h: ScanContentHandle, depth: usize) {
    let pad = "  ".repeat(depth);
    let path = res.path(h).unwrap_or("?");
    let ty   = res.content_type(h);
    println!("{pad}- {path} ({ty:?})");

    // walk children left-to-right via first-child / next-sibling
    let mut c = res.child(h);
    while let Some(cur) = c {
        dump(res, cur, depth + 1);
        c = res.next_sibling(cur);
    }
}

if let Some(root) = res.root() {
    dump(&res, root, 0);
}

Because paths and local VarMaps live in pools owned by the scanner's Context, they are reused across successive scan() calls with zero re-allocation once steady-state is reached — the ScanResult simply borrows them for the lifetime of the current scan.


Scanning pipeline

For every scanned object, the scanner performs the following steps (see content_scan/src/scanner.rs):

  1. Top-level filter check. If a Filter is configured and scan was called with filter_root = true, the root content is tested first; if rejected, the scan returns immediately with an empty result.

  2. Type resolution. If the content already reports a content_type(), it is used as-is. Otherwise the scanner tries, in order, using ContentPath::as_bytes() for the name-based steps:

    1. magic bytes (first 16 bytes; build panics if a registered magic is longer),
    2. exact file name,
    3. file extension,
    4. identifiers that returned None from identify_method (each validate() is tried in registration order).

    Each fast-matcher candidate is confirmed via the corresponding identifier's validate() method. Custom identifiers have no pre-filter; validate() is the identification.

  3. Record. The object is appended to Context::objects — interned from ContentPath::as_printable_string(), tagged with the resolved type, and linked into its parent's child list — before analyzers run.

  4. Type-specific analyzers for the current type run in priority order. They may write VarMaps, emit Findings via context.add_finding, queue extra extraction, or return a steering AnalysisOutcome.

  5. Generic analyzers run for every object in priority order (same recording APIs), unless a typed analyzer already returned Skip, Exit, or ReprocessAsType.

  6. Type-specific extractors for the resolved type run in registration order only if analysis returned Continue (create_session → advance/extract loop; the session is dropped when the loop ends). Each create_session receives an OwnedContentPtr to the parent and an ExtractionContext covering the whole object (offset = 0, length = Some(size), params = None). For each entry they emit, the scanner recurses (subject to max_depth and Filter). Entries marked skip_from_filtering bypass the Filter check.

  7. Extraction requests queued by analyzers via context.request_extract(ty) then run, in emission order — after own-type extractors on Continue, or alone on ReprocessAsType. For each request the extractors registered for ty run on the same parent, with that request's offset, length, and params. The parent does not need to have been identified as ty.

  8. ReprocessAsType. After those requested extractors, if this object has changed type fewer times than max_change_type, the scanner calls set_content_type, updates the tree node's type_id, and re-enters analysis on the same node (steps 4–7). Identifiers, the filter, and the stop condition are not re-run. Over the cap, the object keeps its current type.

Findings emitted during the analyzer steps are appended to the same Context and later exposed through ScanResult::findings. After scan() returns, the tree is available through ScanResult.

Only analyzers return AnalysisOutcome. Extractor and session methods return Option (create_session / advance / extract); they cannot short-circuit the scan themselves.

  • Analyzer Skip on an object stops remaining analyzers and extractors on that object. The scanner maps that Skip to Continue for the parent, so the session that produced the object keeps enumerating siblings.
  • Analyzer Exit aborts the whole scan. The extraction session that produced the current object is dropped as the call stack unwinds; remaining extractors on ancestors do not run.
  • Analyzer ReprocessAsType(ty) stops remaining analyzers for this pass, runs requested extractors, then re-analyzes the same object as ty (capped by max_change_type). A child's ReprocessAsType does not end the parent session.

Building & testing

Standard Cargo workflow from the workspace root:

# build everything
cargo build

# run the unit tests
cargo test

# run an example
cargo run --example vowals
cargo run --example sum
cargo run --example image_size -- path/to/image.jpg
cargo run --example md5 -- path/to/file_or_folder
cargo run --example zip_png_size -- path/to/archive.zip

The workspace pins resolver = "2" and applies a couple of shared Clippy overrides (module_inception, new_without_default) — see the root Cargo.toml.


License

Licensed under the MIT License. © 2026 Gavrilut Dragos.