Crate tokenfold_core 

Re-exports

pub use budget::CompressionPolicy;

pub use budget::LossyPath;

pub use budget::Preset;

pub use budget::PruningPolicy;

pub use budget::TaskScope;

pub use codec::DecodeFormat;

pub use codec::OutputEncoding;

pub use codec::decode;

pub use codec::encode_toon;

pub use errors::TokenFoldError;

pub use input::CompressionInput;

pub use input::CompressionOutput;

pub use input::InputFormat;

pub use modes::TransformId;

pub use pipeline::compress;

pub use pipeline::compress_with_estimator;

pub use status::Status;

Modules

budget

codec

errors

filters

Declarative command-output filter registry: the filter-pack schema, its stage-execution engine, and the trust store that gates project- and user-authored packs.

input

modes

Canonical preset matrix: the single source of truth for which transforms run in which preset, at what ratio cap, for which task scopes and input formats. secret_redaction is deliberately absent from this table — it runs unconditionally before the pipeline and cannot be disabled (see budget::CompressionPolicyBuilder::build).

pipeline

report

retrieval_store

Reversible evidence store and retrieval: content-addressed storage of pre-transform originals, plus the [retrieval] tokenfold.toml schema block that configures it.

safety

Fail-closed safety invariants: JSON validity, object key-order preservation, protected content survival, and no redaction bypass — the four properties every transform in the pipeline must leave intact regardless of which transforms ran before it. The pipeline checks these after every transform; a violation rolls that transform back (pre-transform bytes restored) rather than shipping a corrupted or unsafe output.

stats

Savings ledger, stats aggregation, and JSON/CSV export backing the CLI’s stats/gain/session subcommands.

status

token_estimator

transforms

Transform implementations. Each submodule exposes a pure function over bytes/text; the pipeline (crate::pipeline) owns all bookkeeping (token counts, TransformReport, safety validation/rollback). Canonical IDs and ordering live in crate::modes.