texform-transform

Internal implementation crate for texform. Do not depend on this crate directly — its API has no stability guarantees and may change in any release. Use the texform facade crate instead.

A phase-oriented AST rewrite pipeline for TeXForm. It normalizes a parsed Ast into a canonical form so downstream consumers — formula equivalence comparison, MER tokenization, LLM pretraining corpora, polished authoring output — can work against a stable shape without re-implementing LaTeX semantics per use case. This README is the in-depth reference for the transform subsystem: rule authors and contributors should start here.

The crate is a thin wrapper around five ordered phases. Callers choose a build-time [Profile] / [BuildConfig] to compile a rewrite plan, then use per-run [TransformConfig] values to gate phases and set runtime limits.

Quick start

use texform_core::parse::{ParseConfig, ParseContext};
use texform_transform::{BuildConfig, Profile, TransformContext};

let parse_ctx = ParseContext::from_packages(&["base", "ams"]);
let mut ast = parse_ctx
    .parse_to_ast(r"\frac{a}{b}", &ParseConfig::default())
    .expect("source should parse");

// Pick a profile. `Faithful` preserves layout while expanding commands; use
// `Corpus` to drop layout hints, `Equiv` for equivalence comparison, and
// `Authoring` for author-facing output.
let context = TransformContext::from_build_config(
    BuildConfig::profile(Profile::Faithful),
    &parse_ctx,
)
.expect("transform context should build");

let report = context
    .run(&mut ast, &parse_ctx)
    .expect("transform should succeed");

println!("rewrite iterations: {}", report.rewrite.iterations);
println!(
    "flatten removed_empty: {}",
    report.flatten_groups.actions.removed_empty
);

For repeated transforms with the same configuration, build a context once and reuse it:

use texform_transform::{BuildConfig, Profile, TransformContext};

let context = TransformContext::from_build_config(
    BuildConfig::profile(Profile::Faithful),
    &parse_ctx,
)?;
for mut ast in batch {
    let _report = context.run(&mut ast, &parse_ctx)?;
}

Public API

The crate's public surface is intentionally small:

The rewrite phase additionally re-exports RewriteRule, NormalizationLevel, NormalizationLevelSet, RuleKey, RuleMeta, RuleFidelity, RuleTarget, RuleTargetKey, RuleTargetKind, Plan as RewritePlan and related items for callers that need to introspect rules.

Pipeline

TransformContext::run executes a fixed sequence of phases. Normalization levels are chosen when the context is built; each run may disable rewrite / lower attributes or choose different FlattenGroups and iteration settings through TransformConfig.

1. LowerAttributes (pre) — canonicalize declarative-scope commands (e.g. \bf x) and registered prefix wrappers (e.g. \mathbf{x}) into a single normal form.
2. Rewrite — apply the precompiled rewrite plan in a fixed-point loop, bounded by max_iterations.
3. LowerAttributes (post) — re-canonicalize attribute markers introduced by rewrite rules (some Standard / Expand rules emit prefix wrappers that need lowering again).
4. FinalizeAst — local AST cleanup that does not depend on rewrite metadata, currently merging adjacent Prime nodes produced by rewrite rules.
5. FlattenGroups — remove redundant explicit and implicit groups after the earlier phases have stabilized.

Phase order is fixed; only the per-phase flags are configurable.

Configuration

TransformConfig

pub struct TransformConfig {
    pub rewrite_enabled: bool,
    pub lower_attributes_enabled: bool,
    pub finalize_ast: FinalizeAstConfig,
    pub flatten_groups: FlattenGroupsConfig,
    pub max_iterations: usize,
}

Profiles

Each profile selects build-time normalization levels and supplies a default runtime config.

NormalizationLevel

Every rule belongs to exactly one ordered level. A rule's level is the first profile that accepts the rule output as a suitable product; it is not inferred from render fidelity.

Classify a rule by asking which profile first accepts its output, then declare the rule's fidelity independently. fidelity may rule out profiles whose floor it cannot meet, but a high-fidelity rule is not automatically a lower level.

RuleFidelity

fidelity is the worst-case render-fidelity guarantee over the rule's declared input domain. It is ordered from least to most faithful: Semantic < Approximate < Full.

fidelity is a metadata contract only. texform-transform runs no rendering comparison; how a downstream validator interprets a fidelity level when comparing rendered output is defined by that consumer, not in this crate.

fidelity must not fall below the rule's level floor:

Do not add a second metadata field for ordinary behavior. If a rule has an important gap between its worst case and usual samples, document that gap in the rule's top-level comment. For example, displaylines-to-gather-env is level: Drop and fidelity: Semantic: ordinary \displaylines samples usually look closer to Approximate, but manual layout commands such as \hfill or \llap can make MathJax reflow or overlap hand-written equation numbers.

FlattenGroupsConfig

FlattenGroups removes structurally redundant Explicit and Implicit groups. The four core actions are:

These actions are the default behavior. Eleven configuration flags (ten independent preserve guards plus one sub-flag) gate the actions in specific contexts. Each preserve guard belongs to one of two categories:

• Semantic guards — disabling them changes script binding, environment cell boundaries, declarative scope, or infix scope. Both parsed semantics and rendered output change.
• Spacing guards — disabling them only affects atom-spacing and unary/binary classification context. Parsed semantics are unchanged; rendered output may differ by a thin space.

Preserve guards

Atom-spacing characters: = < > + - , : ; . / * ! ? | ·.

Sub-flag

This sub-flag does not gate any group on its own; it only refines the classification used by guard #5. When preserve_group_adjacent_to_command_like is false, the sub-flag has no effect.

Preset values

The preserve guards are wired to presets via two named constants:

• FlattenGroupsConfig::STRICT — all guards on. Used by AUTHORING and FAITHFUL.
• FlattenGroupsConfig::STRUCTURAL_ONLY — only semantic guards on; all spacing guards off. Used by CORPUS and EQUIV.

Additional constants: ENABLED (alias for STRICT), DISABLED (no flattening at all), DEFAULTS (= STRICT).

Reports

TransformReport aggregates per-phase reports for observability and diagnostics:

pub struct TransformReport {
    pub lower_attributes: LowerAttributesReport,
    pub rewrite: RewriteReport,
    pub finalize_ast: FinalizeAstReport,
    pub flatten_groups: FlattenGroupsReport,
}

• LowerAttributesReport — attributes (HashMap<AttributeSet, AttributeStat>) plus eliminated_empty_segments; each attribute stat has consumed, redundant, and emitted counts split into declaratives and prefixes. The report aggregates all LowerAttributes invocations in one transform run, so the default pipeline combines pre-Rewrite and post-Rewrite counts.
• RewriteReport — iterations (fixed-point iteration count) and rules (Vec<RewriteRuleStat> with key, applied_count, skipped_count per rule that was attempted at least once).
• FinalizeAstReport — steps with one applied_count counter per cleanup step (currently merge_adjacent_primes).
• FlattenGroupsReport — actions for the four action counters and guards for one hit counter per preserve guard. Hit counters are short-circuit: when several guards would apply to the same group, only the first one that matches in the internal evaluation order is incremented.

The stable facade DTO used by the Python and WebAssembly bindings flattens the same information into a transport-safe shape:

{
  iterations,
  rules: [{ key, applied_count, skipped_count }],
  finalize_ast: {
    steps: { merge_adjacent_primes: { applied_count } }
  },
  flatten_groups: {
    actions: { removed_empty, replaced_single_child, inlined_multi_child, unwrapped_slot },
    guards: { preserve_group_* }
  },
  lower_attributes: {
    attributes: [{
      attr,
      value,
      consumed: { declaratives, prefixes },
      redundant: { declaratives, prefixes },
      emitted: { declaratives, prefixes }
    }],
    eliminated_empty_segments
  }
}

Phase internals

LowerAttributes

Two sub-modules drive this phase: lower_attributes/codegen.rs and build-time generated data emitted into OUT_DIR. The phase scans the AST for declarative commands (e.g. \bf, \large, \sf) and registered prefix wrappers (e.g. \mathbf{...}, \textbf{...}), then normalizes both forms into a single canonical representation per attribute.

Attributes are modeled as a structured AttributeSet (Attr × AttrValue) covering math font, math size, math style, text family, text series, text shape, and text size. Inherited state is tracked across container boundaries so that nested declarations, prefix wrappers, and empty trailing segments normalize cleanly.

The phase runs twice in the pipeline (pre and post Rewrite) under a single enabled switch because rewrite rules may emit prefix wrappers as their right-hand side; the post-pass re-canonicalizes those into the same normal form as the pre-pass. LowerAttributesReport uses a single cumulative counter set for both invocations.

Rewrite

Rules live under src/rewrite/rules/{base, ams, braket, physics}/ and are auto-registered through src/rewrite/rules/generated.rs (maintained by build.rs). Each rule is a unit struct implementing RewriteRule with a static RuleMeta descriptor.

RuleMeta declares:

• key: RuleKey (package/name) — the stable identifier used in reports and build-time filters.
• enabled_by_packages — packages whose presence in the ParseContext enables the rule.
• level — see NormalizationLevel.
• fidelity — see RuleFidelity.
• triggers — RuleTargets the scheduler watches to know when to attempt the rule.
• consumes — forms the rule eliminates (must not appear in the output) and touches (may read or modify).
• produces — forms the rule may introduce. The engine verifies every produced form is either accepted by the output contract or eliminated by another rule.

TransformContext::from_build_config builds a Plan by filtering all registered rules through the selected profile levels, build-time disabled rules, and the ParseContext's enabled packages. The plan is then driven by scheduler::drive_fixed_point until either no rule fires in an iteration or max_iterations is exceeded. When Rewrite is enabled, after the full pipeline has run post-Rewrite LowerAttributes and FlattenGroups, the engine uses collect_eliminated_violations to verify that no rule's eliminates set remains in the output AST; a violation is reported as RewriteError::ContractViolation.

See src/rewrite/rules/README.md for rule authoring conventions and the macro-based DSL used to define rules.

FinalizeAst

A single pass (src/finalize_ast/) for local AST cleanup that does not depend on rewrite metadata. Its current step merges adjacent Prime nodes produced by rewrite rules, so f^{\prime\prime} normalizes through Prime(1), Prime(1) into the same final shape as f''. The phase is enabled by default in every profile and gated by TransformConfig.finalize_ast.enabled.

FlattenGroups

A single recursive traversal (visit → try_unwrap in src/flatten_groups/mod.rs). For each node the visitor:

1. Collects subtree-wide flags (has_declarative, has_infix, has_delimited) on the way down.
2. Tracks the in_env_body context flag through Slot::EnvBody edges.
3. On the way back up, calls try_unwrap to check whether the current group should be flattened. Each preserve guard short-circuits with an early return that increments its hit counter; the first matching guard wins.
4. If no guard fires and the group's content mode matches its parent's context mode, the group is unwrapped via either unwrap_group_child (multi-child splice) or redirect_single_child_slot (single-child slot replacement).

The slot_can_unwrap helper restricts redirect-style unwrapping to single-child groups in Argument, Script*, and Infix* slots; EnvBody slots are never unwrapped.

Errors

pub enum TransformError {
    Build(TransformBuildError),
    Rewrite(RewriteError),
}

pub enum TransformBuildError {
    Rewrite(PlanBuildError),
}

pub enum RewriteError {
    Rule { rule: RuleKey, kind: RuleError },
    ContractViolation { target: RuleTargetKey, node_name: Option<String> },
    MaxIterationsExceeded { max_iterations: usize },
}

pub enum RuleError {
    InvalidNodeShape { message: String },
    MissingMetadata { name: String },
}

TransformBuildError is raised by TransformContext::from_build_config when the rewrite plan cannot be assembled, for example when a required package is missing. All other errors surface during execution.

Tests

Integration tests cover the phases and their interactions:

• tests/flatten_groups.rs — preserve-guard toggles, STRICT vs STRUCTURAL_ONLY, action and per-guard counters.
• tests/lower_attributes.rs — declarative consumption, prefix wrapping, inherited-state absorption.
• tests/finalize_ast.rs — adjacent-Prime merging and the phase gate.
• tests/rewrite_rule.rs — single-rule execution and metadata contracts.
• tests/rewrite_context.rs — the RuleContext AST view exposed to rules.
• tests/transform_contract.rs — eliminated-form contract checking across the pipeline.
• tests/config_model.rs — profile and config invariants.

Run with:

cargo test -p texform-transform

See also

• High-level overview: repository README.
• Architecture: ARCHITECTURE.md (Transform Engine section).
• Rule authoring guide: src/rewrite/rules/README.md.