# text-mirror
Converts binary and rich files into plain text and builds a replica text-mirror tree with a manifest that records how every file was handled.
Point it at a directory tree. It walks the tree, detects each format by content, runs the matching converter, and writes a parallel tree of text artifacts. `Q3 Budget.xlsx` becomes `Q3 Budget.xlsx.txt` at the same relative path under its division's mirror root. Shipped converters cover plain text, Markdown, CSV, and text-native developer formats, Word documents and presentations in both current and legacy formats, workbooks with hidden sheets, rows, and columns marked, text-layer PDF, HTML, and email with attachments expanded as child records. Zip archives expand into their members, each routed back through detection. Parquet, avro, and sqlite convert behind the opt-in `records-worker` feature described below. Raster images convert behind the opt-in `image-ocr` and `image-metadata` features described below. Audio transcribes behind the opt-in `audio-asr` feature described below. Video is detected and recorded as `unsupported` with reason `engine-unpinned` until an engine is pinned in the rules data. The append-only JSONL manifest covers every file, including the ones that did not convert, so the mirror always has a known denominator.
The mirror feeds downstream text tooling that cannot read binary formats. The companion crate `data-classification` is one such consumer. The text artifacts carry no in-band metadata, so the tree also serves directly as a corpus.
## Design points
- Fail closed. Every source file produces either a text artifact or a manifest record explaining why not. There is no silent skip.
- Deterministic. Converter versions, model hashes, and decode options are pinned in versioned rules data. Same inputs plus same rules re-run byte-identical.
- Incremental. The manifest is the checkpoint. A killed run resumes by re-walking and skipping completed records.
- Sandboxed. Native tools and models run as subprocess adapters under timeouts, output caps, and a no-network temp jail.
- Verifiable handoff. `text-mirror bundle` packages a mirror for transfer, and `text-mirror verify` recomputes every hash on the receiving side before any consumer touches it.
Early development. The manifest schema `text-mirror/manifest@1` and the architecture are documented in `docs/DESIGN.md`.
## The records-worker feature
Parquet, avro, and sqlite conversion is an opt-in feature, `records-worker`, off by default. These parsers pull large native trees with CVE history, so they run only inside the subprocess jail, and the default build links none of them.
Build with the feature to convert these formats:
```
cargo build --release --features records-worker
```
A default build records a parquet, avro, or sqlite source as `unsupported` with reason `records-worker-not-built` and handles every other format exactly as a featured build does.
## The image features
Pixel OCR for png, jpeg, and webp is gated behind the `image-ocr` feature. The reader that lifts exif, png text-chunk, xmp, and iptc strings out of an image and records them as a hidden derived child at `<source>.d/#image-metadata` is gated behind `image-metadata`. Both are off by default and both run inside the subprocess jail.
```
cargo build --release --features image-ocr,image-metadata
```
A default build records a png, jpeg, or webp source as `unsupported` with reason `image-ocr-not-built` and writes the warning `image-metadata-not-built` on each image record, so a later build that carries the converters re-runs those files.
| Format | Handling in this release |
|---|---|
| png, jpeg, webp | In-jail pixel OCR through the `image-pixel-ocr` converter. The deployment supplies the engine. A build with `image-ocr` on decodes the raster and fails closed with `image-ocr-runtime-missing` until the engine is configured. |
| heic | Fails closed with reason `no-jailed-rasterizer`. Decoding needs an external rasterizer, and the external-rasterizer provider is not built in this release. |
| svg | The raw markup passes through as the text artifact. With the optional `svg-provider` feature built and a provider configured, the rendered form is additionally recognized and recorded as a visible child at `<source>.d/#image-ocr`. Without both, no OCR runs and nothing about the source's own artifact changes. |
| ai | The PDF text layer is extracted through `pdf-subprocess`. A legacy PostScript-backed file has no PDF text layer and fails closed with a pdf-family reason. |
| tiff | Recorded as `unsupported` with reason `engine-unpinned` until a rules bump pins a decoder. |
### The svg provider
Pixel OCR for a vector source needs something that can render it, and
nothing in this crate can. The `svg-provider` feature, off by default
and building on `image-ocr`, adds the leg that hands the source to two
executables a deployment supplies: one that renders it to a raster and
one that flattens that raster onto white. Both are named by generic
role label, bound by BLAKE3 and exact version, and run inside a jail
measured for them. The jailed worker repeats every assertion before
each execution.
```
cargo build --release --features svg-provider
text-mirror run <root> --mirror <mirror> --manifest <manifest> --division <name> \
--provider-config provider.toml
```
```toml
schema = "text-mirror/provider@1"
[providers.svg."raster-browser"]
path = "/deployment/path/to/renderer"
closure_roots = ["/deployment/path/to/renderer-closure"]
jail_service_prefix = "^com\\.example\\.Product\\."
jail_temp_env = "EXAMPLE_TMPDIR"
[providers.svg."raster-magick"]
path = "/deployment/path/to/encoder"
closure_roots = [
"/deployment/path/to/encoder-installation",
"/deployment/path/to/a-dependency.dylib",
]
```
The file carries paths and jail parameters only. What the worker
asserts about each executable, its BLAKE3, its exact version, and the
aggregate digest over its closure, is versioned rules data in the
`[image_ocr.svg_provider]` section, pinned per role label like every
other engine expectation this crate carries.
Enumerate a closure, and enumerate it narrowly. Everything under each
entry is readable and executable inside the jail, so name the measured
dependencies: the installation directory and the specific libraries it
loads. A package store, an application directory, or any other
directory many programs share is refused, because granting one would
put every unrelated program, and every other version of the pinned
component, inside the boundary. A role that enumerates a closure must
have that closure pinned in the rules, or the run is refused before it
starts.
The configuration also carries the two jail parameters the components
need: the service namespace their runtime registers under, written as
an anchored dotted prefix, and the name of an extra temp-directory
variable. Both have closed shapes and a value outside them refuses the
run before the walk starts. Both are part of the jail's identity: the
profile was measured with one set of values, and supplying others
gives a jail nobody has measured with them.
The provider is opt-in twice: without the feature, and without a
complete configuration, a vector source converts exactly as it does
today and no child is written. A configured provider that cannot run
records a failed child naming the role and the reason, one of
`rasterizer-missing`, `rasterizer-hash-drift`,
`rasterizer-version-drift`, `raster-exec-failed`,
`raster-geometry-mismatch`, or `encoder-area-cap`, and leaves the
source's own artifact untouched. Configuring a provider changes the
effective rules version to `10+svg.<digest>` over what the rules pin
and the jail parameters the configuration supplies, so enabling,
repinning, or re-parameterizing one re-runs the affected sources and
moving an identical installation to another path re-runs nothing.
Recognition itself is the image-OCR path, so it needs that path's
engine as well: with the provider wired and no engine configured, the
child fails with `image-ocr-runtime-missing` after the rendering
succeeds. heic is unaffected and stays deferred: this release carries
no heic provider and the schema has no place to configure one.
## The audio feature
Speech transcription for wav, mp3, flac, and m4a is gated behind the `audio-asr` feature, off by default. The audio bytes are decoded inside the subprocess jail by the pure-Rust symphonia crate into a wav at the decoder-reported sample rate and channel count, and the pinned speech engine reads that wav. The engine, its weights, and its backend probe are deployment artifacts named by generic role labels and bound by BLAKE3 in the rules. A run supplies the role paths with `--runtime-inventory`, and the jailed worker re-hashes every file before use. Until a deployment wires a complete pinned runtime, a featured build decodes the audio and fails closed with `asr-runtime-missing`.
```
cargo build --release --features audio-asr
text-mirror run <root> --mirror <mirror> --manifest <manifest> --division <name> \
--runtime-inventory inventory.toml
```
A default build records a wav, mp3, flac, or m4a source as `unsupported` with reason `audio-asr-not-built`. The pinned engine identity exists for one machine class, so on any other platform a featured build records the same formats as `unsupported` with reason `engine-unpinned` until that platform gains its own pin.
Ceilings worth knowing: decoded duration is capped at one hour, and the crate-wide 128 MiB source ceiling applies before conversion, so an hour-long uncompressed wav at 44.1 or 48 kHz fails with `resource_limit` before it is decoded. Hour-long mp3, m4a, and flac sources fit. A transcript record carries a `media` block naming the decoded duration, the language, and the pinned engine identity by role label and hash. m4a is admitted for the low-complexity AAC profile only, and other codec shapes fail closed with `asr-codec-unsupported`.
## License
MIT OR Apache-2.0