pub enum DocumentFormat {
Show 41 variants
Pdf,
Docx,
Epub,
Odt,
Ods,
Odp,
Xlsx,
Pptx,
Json,
Json5,
Yaml,
Csv,
Markdown,
Xml,
Html,
Toml,
Jsonl,
Eml,
Parquet,
ArrowIpc,
Cbor,
Msgpack,
Config,
Feed,
Geojson,
Gis,
Notebook,
FixedWidth,
Rst,
Asciidoc,
Mdx,
Mhtml,
Logstream,
Pkgmeta,
Apispec,
Bib,
Rtf,
Container,
Hdf,
Cfbf,
Opaque,
}Expand description
A detected document format (the class of the field’s source bytes).
Variants§
A validated PDF (physical indicators).
Docx
An OPC package with a WordprocessingML officeDocument part.
Epub
An OCF container with an EPUB package document.
Odt
An ODF package with an OpenDocument text content part.
Ods
An ODF package with an OpenDocument spreadsheet content part.
Odp
An ODF package with an OpenDocument presentation content part.
Xlsx
An OPC package with a SpreadsheetML workbook part.
Pptx
An OPC package with a PresentationML presentation part.
Json
A structured JSON document (the whole source parses as exactly one JSON value). Not a package: the exact leaf is the whole source (Phase 21.5.1).
Json5
A structured JSON5 / JSONC document (the whole source parses as exactly one
JSON5 value and uses at least one JSON5/JSONC-only construct — a comment, a
trailing comma, an unquoted key, a single-quoted string, a hex/leading-dot/
Infinity/NaN number, a string continuation, or the extended whitespace
set). A source that is strict JSON stays DocumentFormat::Json; the
recorded dialect distinguishes jsonc (comments/trailing commas only) from
json5. Not a package: the exact leaf is the whole source (Phase 21.17.1).
Yaml
A structured YAML document (the whole source parses as a bounded YAML stream whose every document root is a mapping or a sequence). Not a package: the exact leaf is the whole source (Phase 21.6.1).
Csv
A CSV/TSV table (the whole source parses under a comma or tab delimiter as a table with a consistent field count across a sampled majority of at least two records). Not a package: the exact leaf is the whole source, and the recorded dialect distinguishes comma from tab (Phase 21.7.1).
Markdown
A Markdown prose document (the whole source carries at least one structural mark — an ATX heading, a fenced code block, front matter, a table, a reference definition, or a footnote definition). Not a package: the exact leaf is the whole source (Phase 21.8.1).
Xml
A standalone XML document (the whole source begins with < and parses as
well-formed XML with exactly one root element under the XML caps). Not a
package: the exact leaf is the whole source, and every node/attribute span
is a Q_gen projection (Phase 21.9).
Html
A standalone HTML document (the source is not well-formed XML and carries
clear HTML structure — a <!doctype html>, an <html/<head/<body tag,
or a preponderance of known HTML tags). Parsed by a bounded,
error-recovering scanner. Not a package: the exact leaf is the whole
source, and every element/attribute/text/comment/raw-text span is a Q_gen
projection (Phase 21.10).
Toml
A TOML document (the source is not any earlier format, parses as TOML 1.0
under the TOML caps with no errors, and carries at least one key/value
assignment). Not a package: the exact leaf is the whole source, and every
table/array/key/value/comment span is a Q_gen projection. TOML’s
duplicate-key/redefinition rules are enforced (a violation is a typed
decline) (Phase 21.11).
Jsonl
A JSONL / NDJSON line/event stream (the source carries at least two non-blank
lines and every non-blank line parses as exactly one JSON value under the
caps, each line parsed by the shared JSON parser). Not a package: the exact
leaf is the whole source, and every record’s exact line span, terminator, and
per-token spans are Q_gen projections. A single JSON value stays
DocumentFormat::Json (Phase 21.12).
Eml
An EML / MIME internet message (an RFC 5322 header block terminated by a
blank line, carrying From/Date/Message-ID, or an explicit
MIME-Version). Not a package: the exact leaf is the whole source, and every
header/part span and every Content-Transfer-Encoding-decoded constituent is
a Q_gen projection (Phase 21.13).
Parquet
An Apache Parquet file (the source begins with PAR1 and ends with PAR1,
and the 4-byte little-endian footer length before the trailing magic is
consistent with the file length). Not a package: the exact leaf is the whole
source, and the parsed footer inventory (schema, row groups, column chunks,
statistics) and any decoded values are Q_gen projections (Phase 21.14).
ArrowIpc
An Apache Arrow IPC file/stream (the source begins with the ARROW1 magic
and is either the file format — a trailing ARROW1 magic preceded by a
consistent little-endian int32 footer length — or the stream format —
a valid encapsulated Schema message at the 8-byte magic+padding prefix).
Not a package: the exact leaf is the whole source, and the parsed schema and
record-batch inventory (with each batch’s exact source span) and any decoded
columnar values are Q_gen projections (Phase 21.16).
Cbor
A CBOR (RFC 8949) structured-tree document. CBOR has no magic bytes, so
detection is deliberately conservative: either the self-described-CBOR tag
55799 (0xd9 0xd9 0xf7) at the start of a source that then parses, in full,
as exactly one well-formed CBOR item; or a full-input well-formed parse whose
root is a container (array/map) or a tag and which reaches at least three
nodes (so a single scalar, an empty container, and any structurally trivial
input stay Opaque). A container head byte is always >= 0x80, so a
pure-ASCII text document is never claimed; and the whole-number/short-
container prefix overlaps MessagePack’s fixint/fixarray encodings, so an
ambiguous or trivial input stays Opaque rather than being guessed
(recorded honestly; a Phase-21.19 MessagePack adapter must share the seam).
Not a package: the exact leaf is the whole source, and every item’s kind,
exact span, encoding width, tag number, float width, and definite/indefinite
form are Q_gen projections (Phase 21.18).
Msgpack
A MessagePack structured-tree document. MessagePack has no magic bytes, so
detection is deliberately conservative: a full-input well-formed parse of
exactly one item whose root is a container (array/map) reaching a node/byte
threshold, or the same with an unambiguous MessagePack-only head byte
(0xdc..=0xdf, which CBOR’s grammar rejects). A lone scalar, an empty
container, and any structurally trivial or ambiguous input stay Opaque. A
container head byte is always >= 0x80, so a pure-ASCII text document is never
claimed; and the whole-number/short-container prefix overlaps CBOR’s
fixint/short containers, so CBOR (tried first) owns any input well-formed
under both. Not a package: the exact leaf is the whole source, and every item’s
kind, exact span, exact format byte (encoding width and signedness), str vs
bin, float width, and extension type/length are Q_gen projections
(Phase 21.19).
Config
A config-family document (INI, .env, or Java .properties). The whole
source is a bounded, line-based key/value document; detection is
deliberately conservative and requires a dialect-distinguishing signal,
so the pure KEY=VALUE overlap between env and properties (and plain
prose, and a #! script) stays Opaque rather than being guessed. Not a
package: the exact leaf is the whole source, the recorded dialect lives in
the model, and every line’s content span, key/separator/value span, quoting,
inline-comment, export, continuation, and \uXXXX-spelling fact is a
Q_gen projection (Phase 21.20).
Feed
An RSS 2.0 / Atom 1.0 syndication feed. A feed is XML, so its physical
bytes are shared with DocumentFormat::Xml; the recorded dialect
(rss/atom) lives in the model, exactly as CSV/config record their
dialects. Detection is a bounded semantic test run before the generic
XML detector: an RSS root <rss> with a <channel> child, or an Atom root
<feed> in the Atom namespace (http://www.w3.org/2005/Atom) with at least
one <entry> child. Not a package: the exact leaf is the whole source, and
every element/attribute span, element order, attribute spelling, and the
namespace declaration is a Q_gen projection (Phase 21.21).
Geojson
A GeoJSON (RFC 7946) document. GeoJSON is JSON, so its physical bytes are
shared with DocumentFormat::Json; the recorded root class (one of the nine
RFC 7946 type names) lives in the model, exactly as a feed records its
dialect. Detection is a bounded semantic test run before the generic JSON
detector: the source parses as exactly one JSON value whose root is an object
with a "type" string equal to one of the nine type names and whose shape is
consistent (a geometry has an array coordinates/geometries, a Feature has
geometry/properties, a FeatureCollection has an array features). A plain
JSON document, and a JSON document whose "type" is an unrelated string, stay
DocumentFormat::Json. Not a package: the exact leaf is the whole source, and
every JSON token span, coordinates nesting, properties order/duplicates, and
foreign member is a Q_gen projection (Phase 21.22).
Gis
A KML 2.2 / GPX 1.1 geospatial document. KML and GPX are XML, so their
physical bytes are shared with DocumentFormat::Xml; the recorded dialect
(kml/gpx) lives in the model, exactly as a feed records its dialect.
Detection is a bounded semantic test run before the generic XML detector:
a <kml> root in the KML namespace (http://www.opengis.net/kml/2.2) with a
Document/Folder/Placemark child, or a <gpx> root in the GPX namespace
(http://www.topografix.com/GPX/1/1) with a metadata/wpt/rte/trk
child. A plain XML document, and a shaped-but-invalid KML/GPX, stay
DocumentFormat::Xml. Not a package: the exact leaf is the whole source,
and every element/attribute span, element order, attribute spelling (KML
geometry coordinates, GPX lat/lon), and the namespace declaration is a
Q_gen projection (Phase 21.23).
Notebook
A Jupyter notebook (.ipynb, nbformat) document. A notebook’s physical bytes
are JSON, so its bytes are shared with DocumentFormat::Json; the recorded
nbformat/nbformat_minor and the cell anchors live in the model, exactly as a
feed records its dialect. Detection is a bounded semantic test run before the
generic JSON detector: the source parses as exactly one JSON value whose root is
an object with a plain non-negative integer-literal nbformat (≥ 1) and an array
cells, every cell an object with a string cell_type, and every present
recognized field nbformat-shaped. A plain JSON document, and a JSON document that
merely has a cells key but is not nbformat-shaped, stay
DocumentFormat::Json. Not a package: the exact leaf is the whole source, and
every JSON token span, the exact cell_type/output_type strings, the exact
source representation, and the cell/output order are Q_gen projections
(Phase 21.24).
FixedWidth
A fixed-width (column-position) text table. Unlike CSV/TSV/PSV, its columns
are defined by character positions, not a delimiter, so it is a distinct
format rather than a CSV dialect. Detection is maximally conservative
(fixed-width is inherently ambiguous): at least three sampled records of
identical byte width, an inferred column layout whose interior whitespace gaps
are at least two columns wide, at least two non-empty columns, and an outright
decline of anything that also parses as a delimited (CSV/TSV/PSV) or Markdown
table. Not a package: the exact leaf is the whole source, and the recorded
inferred layout, per-record spans, per-column spans, padding, terminator, BOM,
and header row are Q_gen projections (Phase 21.25).
Rst
A reStructuredText (Docutils) prose document (the whole source carries a
reST-specific structural mark — an explicit markup start (.. comment,
directive, target, footnote/citation, or substitution definition), a grid or
simple table, a field list, or a section adornment whose char is not a
Markdown construct — so plain prose and a Markdown document are never stolen).
Not a package: the exact leaf is the whole source (Phase 21.26.1).
Asciidoc
An AsciiDoc prose document (the whole source carries an AsciiDoc-specific
structural mark — a level-0 document title followed by a further block, a
==+ section, a |=== table, a complete delimited block, or a block attribute
line followed by a block — so plain prose and a Markdown or reStructuredText
document are never stolen). Not a package: the exact leaf is the whole source
(Phase 21.26.2).
Mdx
An MDX document (Markdown + JSX/ESM). The whole source carries an MDX-specific
signal — a top-level ESM import/export statement, a JSX element with a
capitalized/namespaced component name (or a JSX-specific attribute), a JSX
fragment, or a whole-line MDX block expression {…} — so plain Markdown stays
Markdown, plain HTML stays Html, and plain prose stays Opaque. Not a
package: the exact leaf is the whole source (Phase 21.26.3).
Mhtml
An MHTML (MIME HTML, RFC 2557) web archive. MHTML is a MIME
multipart/related document, so its physical bytes are shared with
DocumentFormat::Eml; the adapter reuses the EML adapter’s bounded MIME
layer for the envelope, parts, headers, and spans, and the HTML adapter’s
bounded, error-recovering scanner for the root part, recording the ordered
sub-resources (keyed by Content-Location/Content-ID) and the root HTML part
(the start= Content-ID, else the first text/html part, else the first
part). Detection is a bounded semantic sub-detection run before the generic
EML detector: a multipart/related root plus an MHTML-specific signal — a
top-level Snapshot-Content-Location/Content-Base, or a From+Subject
envelope with a text/html part. A plain multipart/related email, a plain
HTML document, and a plain MIME message stay Eml/Html/Eml. Not a package:
the exact leaf is the whole source, and every envelope/part/resource/HTML span
is a Q_gen projection (Phase 21.27).
Logstream
A syslog / log-stream document (the source carries at least two non-blank
lines and every non-blank line matches one of three dialects — RFC 5424
syslog, RFC 3164 (BSD) syslog, or a generic application log line with a
leading timestamp and/or level token). The recorded dialect lives per record.
Detection is byte-based and conservative and must not steal a higher-priority
format (JSON/JSON5/JSONL/CBOR/Msgpack/YAML/TOML/config/CSV or the prose
family), all of which the whole-source dispatcher tries first; plain prose and
code stay Opaque. Not a package: the exact leaf is the whole source, and
every record’s exact line span/terminator plus every field’s exact span (and
the preserved message, NILVALUE, and timestamp/level spelling) are Q_gen
projections (Phase 21.28).
Pkgmeta
A package-metadata manifest (an npm package.json / package-lock.json, a
Cargo Cargo.toml / Cargo.lock, or a Python pyproject.toml). A manifest’s
physical bytes are JSON or TOML, so its bytes are shared with
DocumentFormat::Json/DocumentFormat::Toml; the recorded dialect
(npm_package / cargo_manifest / pyproject_manifest / cargo_lock /
npm_lock) lives in the model, exactly as a feed or CSV records its dialect.
Detection is a bounded semantic test run before the generic JSON and TOML
detectors: a strong manifest shape (a JSON object with string name+version
and a third package-specific key, a JSON lockfileVersion+packages, a TOML
[package] table with a string name, a TOML [build-system]/[project]/
[tool.poetry], or a TOML [[package]] lockfile shape). A generic JSON
name+version, a generic TOML name/version, and plain prose stay
Json/Toml/Opaque. Not a package: the exact leaf is the whole source, and
every reused-arena token span plus every recorded section/entry name/value span
is a Q_gen projection (Phase 21.29).
Apispec
An API/specification document (a JSON Schema, an OpenAPI 3.x, a Swagger 2.0, or
an AsyncAPI document). An API spec’s physical bytes are JSON, so its bytes are
shared with DocumentFormat::Json; the recorded dialect (json_schema /
openapi / swagger / asyncapi) lives in the model, exactly as CSV/config/
feed/pkgmeta record their dialects. Detection is a bounded semantic test run
before the generic JSON detector on a strong, root-level string marker: a
JSON-Schema $schema URI, an openapi 3.-prefixed string, a swagger equal
to "2.0", or a non-empty asyncapi string. A plain JSON object that merely
contains a properties/paths/components key but lacks the marker stays
DocumentFormat::Json, and a JSON-Schema-shaped object with no $schema is
an honest recorded ambiguity that also stays Json. Not a package: the exact
leaf is the whole source, and every JSON token span plus every recorded
object/member/$ref span is a Q_gen projection — $ref targets are preserved
verbatim and never resolved (Phase 21.30).
Bib
A bibliography document — BibTeX (.bib), RIS, or CSL-JSON. One
adapter covers the three dialects and records the dialect in the model, exactly
as CSV/config/feed/pkgmeta/apispec record theirs. BibTeX and RIS are text (no
magic bytes); CSL-JSON’s physical bytes are JSON, so its bounded semantic
sub-detection runs before the generic JSON detector: the root must be an
array of objects (or a single object) each carrying a string id and a
string type. A plain JSON array, a JSON array of non-CSL objects, a generic
JSON object, and prose stay DocumentFormat::Json/DocumentFormat::Opaque.
Not a package: the exact leaf is the whole source, and every entry/field/item
token span (and each value’s exact spelling and delimiters) is a Q_gen
projection (Phase 21.31).
Rtf
An RTF (Rich Text Format) document. RTF is a text, group-brace grammar with
control words — closer to a bounded tree (like XML) than to a flat line format —
but it is not a package: there is no OPC/ZIP layer, so the exact leaf is the
whole source (a DocumentExact, RAW-like authority) and every derived
observation is a bounded, span-preserving (Q_gen) projection of it. Detection
is a strong magic + brace-balance test: the source must begin (after an
optional UTF-8 BOM) with {\rtf followed by a digit, and its group braces must
balance under a bounded scan (respecting \\/\{/\}, \'hh, control words,
and \binN), with the outermost group closing exactly at end of input. A plain
text source, a leading-{ blob, and every other format are therefore never
stolen; they stay Opaque. The adapter preserves the header, every group with
its exact brace span and nesting, every control word/symbol, every \'hh hex
escape and \uN Unicode escape (with its \ucN fallback), every destination
(\fonttbl/\colortbl/\stylesheet/\info/\pict/\object/\*) verbatim,
and every plain text run byte-exactly (never whitespace-normalized) — Phase
21.32.
Container
A generic archive container (Phase 21.33): a standalone ZIP archive (a
valid EOCD and a coherent central directory) or a standalone TAR archive
(a valid ustar/old-GNU or v7 header whose checksum matches). The recorded
dialect (zip/tar) lives in the model, exactly as CSV/config/feed record
theirs. This is deliberately a plain archive, never an office package: the
ZIP-family office formats (DOCX/EPUB/ODT/ODS/ODP/XLSX/PPTX) are detected
before this adapter, so a .docx/.xlsx/.odt source still detects as
its office format and is never reclassified. Not a package: the exact leaf is
the whole source (a DocumentExact, RAW-like authority) and every derived
observation is a bounded, span-preserving (Q_gen) projection of it.
Hdf
A scientific container (Phase 21.34): a NetCDF classic (CDF-1/CDF-2) file or an
HDF5 file (a documented bounded subset). The recorded dialect
(netcdf/hdf5) lives in the model, exactly as the archive container records
zip/tar. This is deliberately a scan-only scientific format: it is not
an office package. Detection is a strong magic-byte test — NetCDF requires
CDF\x01/CDF\x02 at offset 0; HDF5 requires the 8-byte signature
\x89HDF\r\n\x1a\n at offset 0 or 512 with a superblock version in 0..=3, an
offset/length size in {2,4,8}, and a superblock that fits inside the source.
Not a package: the exact leaf is the whole source (a DocumentExact, RAW-like
authority) and every derived observation is a bounded, span-preserving (Q_gen)
projection of it. NetCDF classic is supported in full; HDF5 is a documented
bounded subset (symbol-table groups + contiguous datasets + v1 object headers),
and everything outside it declines typed.
Cfbf
A CFBF / OLE2 compound file (Phase 21.35): the physical-layer container behind
legacy Word (.doc), Excel (.xls), PowerPoint (.ppt), and Outlook (.msg)
files. Detection is a strong magic + header validation test — the 8-byte
signature D0 CF 11 E0 A1 B1 1A E1 at offset 0 and an internally
consistent header (byte order FFFE, a major version of 3 or 4, a sector shift
of 9 or 12 consistent with the version, a mini-sector shift of 6, at least one
FAT sector, and a first directory sector inside the file). A random
signature-only blob and prose stay Opaque. The recorded dialect
(cfb3/cfb4) lives in the model. Not a package: the exact leaf is the whole
source (a DocumentExact, RAW-like authority) and every derived observation —
the header, the DIFAT/FAT, the directory entries with their verbatim UTF-16LE
names and reconstructed sibling/child tree, and each stream’s exact byte span —
is a bounded, span-preserving (Q_gen) projection of it. This is the shared
physical layer that unlocks the legacy DOC/XLS/PPT/MSG semantic models (later
subphases).
Opaque
Anything else; preserved exactly by the opaque floor.
Implementations§
Source§impl DocumentFormat
impl DocumentFormat
Sourcepub const fn name(self) -> &'static str
pub const fn name(self) -> &'static str
Stable lower-case name (used in JSON and in the manifest provenance token).
Sourcepub const fn adapter(self) -> &'static str
pub const fn adapter(self) -> &'static str
The adapter that serves this format (the observation layer’s name for it).
Sourcepub const fn compiled(self) -> bool
pub const fn compiled(self) -> bool
Whether the adapter for this format is compiled into this build.
Sourcepub fn provenance_prefix(self) -> String
pub fn provenance_prefix(self) -> String
The machine-readable format=<name>; provenance prefix recorded at ingest.
Sourcepub fn from_provenance(provenance: &str) -> Option<DocumentFormat>
pub fn from_provenance(provenance: &str) -> Option<DocumentFormat>
Recover the recorded format from a field manifest’s provenance string.
Returns None for a manifest that does not carry the token (e.g. a field
written before this subphase); such a field still serves every native
selector, but common observations decline typed rather than guessing.