napparent-tabular

napparent makes what a model is using apparent — starting with tabular effect features.

Status

0.1.0 — early release. The API may change. The supported entry point is transform_record_batches with a TransformConfig. For large datasets, use transform_record_batches_chunked and set TransformLimits — the default transform_record_batches concatenates all output batches and can OOM when row count × column count is large. Lower-level types (PairAggregator, PreprocessStream) are exposed but unstable. Output feature columns use an _effect suffix; outcomes are in outcomes_effect.

Install

cargo add napparent-tabular

Optional features:

cargo add napparent-tabular --features progress   # TTY progress bars + SIGINT helper
cargo add napparent-tabular --features parquet    # Parquet I/O

Usage

Build one or more Apache Arrow RecordBatch chunks (same schema), then run the tabular transform:

use napparent_tabular::{BinDepth, TransformConfig, transform_record_batches};
// construct batches: Vec<RecordBatch>
let config = TransformConfig::new(BinDepth::new(8));
let out = transform_record_batches(&batches, "target_col", &cols_to_drop, &config)?;

Output includes original columns, {column}_effect features, Actuals, and outcomes_effect.

Large data (lower peak RAM)

use napparent_tabular::{BinDepth, TransformConfig, TransformLimits, transform_record_batches_chunked};

let config = TransformConfig::new(BinDepth::new(8)).with_limits(TransformLimits {
    max_rows: Some(1_000_000),
    max_active_columns: Some(100),
    ..TransformLimits::default()
});
let batches_out = transform_record_batches_chunked(&batches, "target_col", &cols_to_drop, &config)?;
// one output RecordBatch per input batch — no mega-concat

transform_record_batches still concatenates for convenience; use chunked output when row count × column count is large.

Input conventions

Before binning and accumulation:

• Numerical NaN → 0
• Categorical missing / NaN → "empty" (maps to rare token ε when infrequent)
• Outcome NaN → 0 (contributes zero to pair statistics)

These match the default rules in the theory write-up.

Arrow / ndarray bridge

Numeric columns use ndarrow (pre-1.0, pinned) for zero-copy views from Apache Arrow RecordBatch data during preprocessing and aggregation training. Float32 effect columns are exported back to Arrow without an extra buffer copy. Binned label columns (Utf8) and KG HashMap state still allocate as before.

Long runs: enable progress with TransformConfig::new(depth).with_verbose(true). With the progress feature on a TTY, stderr shows an in-place bar per pass; without the feature or when stderr is piped, verbose mode falls back to throttled line logs. Cooperative cancellation uses CancelToken (Python bindings wire KeyboardInterrupt via a hook).

Activations

KG pair edges and effect columns use pluggable activations (see activation module).

Log-frequency weighting reduces bias from sparse / outlier pair cells in the HashMap KG. Value-pair keys are stored in canonical (min, max) order so (u,v) and (v,u) share one bucket. More activations (Bayesian, robust contrast, etc.) are planned.

use napparent_tabular::{ActivationConfig, BinDepth, TransformConfig};

let config = TransformConfig::new(BinDepth::new(8));
// defaults: LogFrequencyWeightedMean + GlobalMeanContrast

Python

Python bindings live in the GitHub repo (napparent-tabular-py); not on PyPI yet. Install from a clone with maturin:

git clone https://github.com/NiklausParcell/napparent
cd napparent
maturin develop

import napparent_tabular
out = napparent_tabular.transform_record_batches(
    batches, target, cols_to_drop, main_depth,
    kg_activation="log_frequency_weighted_mean",
    effect_activation="global_mean_contrast",
    verbose=True,
)
# lower peak RAM: concat=False or transform_record_batches_chunked(...)
chunks = napparent_tabular.transform_record_batches(
    batches, target, cols_to_drop, main_depth, concat=False,
)

Algorithm

Behavior follows the Barn Effect algorithm (canonical κ keys, fixed partner divisor $m_c = p - 1$, per-chunk threshold $\theta_k$).

Formal write-up: paper/ (start with barn_effect_tight.tex).

Roadmap

• napparent-multimodal — image/sensor → embedding space (future crate)

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.