text-mirror

Converts binary and rich files into plain text and builds a replica text-mirror tree with a manifest that records how every file was handled.

Point it at a directory tree. It walks the tree, detects each format by content, runs the matching converter, and writes a parallel tree of text artifacts. Q3 Budget.xlsx becomes Q3 Budget.xlsx.txt at the same relative path under its division's mirror root. Shipped converters cover plain text, Markdown, CSV, and text-native developer formats, Word documents and presentations in both current and legacy formats, workbooks with hidden sheets, rows, and columns marked, text-layer PDF, HTML, and email with attachments expanded as child records. Zip archives expand into their members, each routed back through detection. Parquet, avro, and sqlite convert behind the opt-in records-worker feature described below. Raster images convert behind the opt-in image-ocr and image-metadata features described below. Audio transcribes behind the opt-in audio-asr feature described below. Video is detected and recorded as unsupported with reason engine-unpinned until an engine is pinned in the rules data. The append-only JSONL manifest covers every file, including the ones that did not convert, so the mirror always has a known denominator.

The mirror feeds downstream text tooling that cannot read binary formats. The companion crate data-classification is one such consumer. The text artifacts carry no in-band metadata, so the tree also serves directly as a corpus.

Design points

• Fail closed. Every source file produces either a text artifact or a manifest record explaining why not. There is no silent skip.
• Deterministic. Converter versions, model hashes, and decode options are pinned in versioned rules data. Same inputs plus same rules re-run byte-identical.
• Incremental. The manifest is the checkpoint. A killed run resumes by re-walking and skipping completed records.
• Sandboxed. Native tools and models run as subprocess adapters under timeouts, output caps, and a no-network temp jail.
• Verifiable handoff. text-mirror bundle packages a mirror for transfer, and text-mirror verify recomputes every hash on the receiving side before any consumer touches it.

Early development. The manifest schema text-mirror/manifest@1 and the architecture are documented in docs/DESIGN.md.

The records-worker feature

Parquet, avro, and sqlite conversion is an opt-in feature, records-worker, off by default. These parsers pull large native trees with CVE history, so they run only inside the subprocess jail, and the default build links none of them.

Build with the feature to convert these formats:

cargo build --release --features records-worker

A default build records a parquet, avro, or sqlite source as unsupported with reason records-worker-not-built and handles every other format exactly as a featured build does.

The image features

Pixel OCR for png, jpeg, and webp is gated behind the image-ocr feature. The reader that lifts exif, png text-chunk, xmp, and iptc strings out of an image and records them as a hidden derived child at <source>.d/#image-metadata is gated behind image-metadata. Both are off by default and both run inside the subprocess jail.

cargo build --release --features image-ocr,image-metadata

A default build records a png, jpeg, or webp source as unsupported with reason image-ocr-not-built and writes the warning image-metadata-not-built on each image record, so a later build that carries the converters re-runs those files.

The svg provider

Pixel OCR for a vector source needs something that can render it, and nothing in this crate can. The svg-provider feature, off by default and building on image-ocr, adds the leg that hands the source to two executables a deployment supplies: one that renders it to a raster and one that flattens that raster onto white. Both are named by generic role label, bound by BLAKE3 and exact version, and run inside a jail measured for them. The jailed worker repeats every assertion before each execution.

cargo build --release --features svg-provider
text-mirror run <root> --mirror <mirror> --manifest <manifest> --division <name> \
  --provider-config provider.toml

schema = "text-mirror/provider@1"

[providers.svg."raster-browser"]
path = "/deployment/path/to/renderer"
closure_roots = ["/deployment/path/to/renderer-closure"]
jail_service_prefix = "^com\\.example\\.Product\\."
jail_temp_env = "EXAMPLE_TMPDIR"

[providers.svg."raster-magick"]
path = "/deployment/path/to/encoder"
closure_roots = [
  "/deployment/path/to/encoder-installation",
  "/deployment/path/to/a-dependency.dylib",
]

The file carries paths and jail parameters only. What the worker asserts about each executable, its BLAKE3, its exact version, and the aggregate digest over its closure, is versioned rules data in the [image_ocr.svg_provider] section, pinned per role label like every other engine expectation this crate carries.

Enumerate a closure, and enumerate it narrowly. Everything under each entry is readable and executable inside the jail, so name the measured dependencies: the installation directory and the specific libraries it loads. A package store, an application directory, or any other directory many programs share is refused, because granting one would put every unrelated program, and every other version of the pinned component, inside the boundary. A role that enumerates a closure must have that closure pinned in the rules, or the run is refused before it starts.

The configuration also carries the two jail parameters the components need: the service namespace their runtime registers under, written as an anchored dotted prefix, and the name of an extra temp-directory variable. Both have closed shapes and a value outside them refuses the run before the walk starts. Both are part of the jail's identity: the profile was measured with one set of values, and supplying others gives a jail nobody has measured with them.

The provider is opt-in twice: without the feature, and without a complete configuration, a vector source converts exactly as it does today and no child is written. A configured provider that cannot run records a failed child naming the role and the reason, one of rasterizer-missing, rasterizer-hash-drift, rasterizer-version-drift, raster-exec-failed, raster-geometry-mismatch, or encoder-area-cap, and leaves the source's own artifact untouched. Configuring a provider changes the effective rules version to 10+svg.<digest> over what the rules pin and the jail parameters the configuration supplies, so enabling, repinning, or re-parameterizing one re-runs the affected sources and moving an identical installation to another path re-runs nothing.

Recognition itself is the image-OCR path, so it needs that path's engine as well: with the provider wired and no engine configured, the child fails with image-ocr-runtime-missing after the rendering succeeds. heic is unaffected and stays deferred: this release carries no heic provider and the schema has no place to configure one.

The audio feature

Speech transcription for wav, mp3, flac, and m4a is gated behind the audio-asr feature, off by default. The audio bytes are decoded inside the subprocess jail by the pure-Rust symphonia crate into a wav at the decoder-reported sample rate and channel count, and the pinned speech engine reads that wav. The engine, its weights, and its backend probe are deployment artifacts named by generic role labels and bound by BLAKE3 in the rules. A run supplies the role paths with --runtime-inventory, and the jailed worker re-hashes every file before use. Until a deployment wires a complete pinned runtime, a featured build decodes the audio and fails closed with asr-runtime-missing.

cargo build --release --features audio-asr
text-mirror run <root> --mirror <mirror> --manifest <manifest> --division <name> \
  --runtime-inventory inventory.toml

A default build records a wav, mp3, flac, or m4a source as unsupported with reason audio-asr-not-built. The pinned engine identity exists for one machine class, so on any other platform a featured build records the same formats as unsupported with reason engine-unpinned until that platform gains its own pin.

Ceilings worth knowing: decoded duration is capped at one hour, and the crate-wide 128 MiB source ceiling applies before conversion, so an hour-long uncompressed wav at 44.1 or 48 kHz fails with resource_limit before it is decoded. Hour-long mp3, m4a, and flac sources fit. A transcript record carries a media block naming the decoded duration, the language, and the pinned engine identity by role label and hash. m4a is admitted for the low-complexity AAC profile only, and other codec shapes fail closed with asr-codec-unsupported.

License

MIT OR Apache-2.0