Skip to main content

Crate rusttorch_tabular

Crate rusttorch_tabular 

Source
Expand description

Stream local records, fit preprocessing on training data, and reuse that immutable state for validation and inference.

use rusttorch_tabular::{CsvRecords, Column, FittedPreprocessor, MissingValue, Schema};
use rusttorch_data::ResourceLimits;
let limits = ResourceLimits::default();
let schema = Schema::new(vec![Column::Numeric { name: "age".into(), missing: MissingValue::Reject }])?;
let training = CsvRecords::new("age\n20\n40\n".as_bytes(), limits)?;
let fitted = FittedPreprocessor::fit(schema, training, limits)?;
let state = fitted.to_json()?;
let validation = CsvRecords::new("age\n30\n".as_bytes(), limits)?;
let restored = FittedPreprocessor::from_json(&state, limits)?;
let sample = restored.transform(&validation.into_iter().next().unwrap()?)?;
assert_eq!(Vec::<f32>::try_from(&sample.numeric)?, [0.]);

§RustTorch Tabular

Turn local CSV, JSON Lines, Arrow and Parquet data into model-ready numeric and categorical tensors. Fit preprocessing once on training data, save it, and use the same immutable statistics on validation, test and inference rows.

§Installation

Enable rusttorch = { version = "0.4", features = ["tabular"] } for CSV/JSONL, or add columnar for Arrow/Parquet. Direct package users select arrow or parquet separately. CSV users do not compile the columnar stack. All tensor packages share the existing LibTorch runtime.

§Features

The tabular facade feature enables CSV/JSONL and fitted preprocessing. columnar also enables Arrow/Parquet; direct consumers can select arrow or parquet. The direct package has no default features. download-libtorch obtains the tensor runtime; doc-only builds documentation without native linking and cannot execute tensor operations.

§Native runtime

Tensor operations require LibTorch 2.13.0. Enable download-libtorch, or set LIBTORCH to an extracted distribution and add its library directory to PATH on Windows or the platform’s shared-library search path. All RustTorch packages in one application must share the same runtime. Documentation can be checked with cargo doc -p rusttorch-tabular --no-default-features --features doc-only.

§Example: Fit training data and batch validation rows

use rusttorch_data::{DataLoader, ResourceLimits};
use rusttorch_tabular::{Column, CsvRecords, FittedPreprocessor, MissingValue,
    Schema, TabularDataset};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let limits = ResourceLimits::default();
    let schema = Schema::new(vec![
        Column::Numeric { name: "age".into(), missing: MissingValue::Mean },
        Column::Categorical { name: "city".into(), unknown_zero: true },
    ])?;
    let training = CsvRecords::new("age,city\n20,Berlin\n40,Paris\n".as_bytes(), limits)?;
    let fit = FittedPreprocessor::fit(schema, training, limits)?;
    let saved = fit.to_json()?; // Store beside the model checkpoint.

    let fit = FittedPreprocessor::from_json(&saved, limits)?;
    let rows = CsvRecords::new("age,city\n30,Berlin\n,Unknown\n".as_bytes(), limits)?
        .collect::<Result<Vec<_>, _>>()?;
    let dataset = TabularDataset::new(rows, fit)?;
    let mut loader = DataLoader::builder(dataset).batch_size(2).workers(2).build()?;
    let batch = loader.iter().next().unwrap()?;
    assert_eq!(Vec::<Vec<f32>>::try_from(&batch.numeric)?, [[0.], [0.]]);
    assert_eq!(Vec::<Vec<i64>>::try_from(&batch.categorical)?, [[1], [0]]);
    Ok(())
}

Numeric features use the training mean and population standard deviation. A constant feature uses scale one. MissingValue::Mean produces normalized zero; Reject fails. Numeric values and resulting Float features must be finite. Categorical vocabularies are sorted; known IDs start at one. ID zero represents missing/unknown values only when explicitly enabled. Numeric and categorical output order follows the schema, in separate Float and Int64 tensors. Extra input fields are ignored. No statistics are recomputed during iteration.

§Stream records without retaining the full dataset

use rusttorch_data::{batches, ResourceLimits};
use rusttorch_tabular::{FittedPreprocessor, open_jsonl};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let limits = ResourceLimits::default();
    let fit = FittedPreprocessor::from_json(&std::fs::read_to_string("preprocessing.json")?, limits)?;
    let rows = open_jsonl("events.jsonl", limits)?;
    for samples in batches(fit.transform_rows(rows), 128, false)? {
        let samples = samples?; // Owned TabularSample values, loaded lazily.
        println!("{} rows", samples.len());
    }
    Ok(())
}

open_csv/CsvRecords delegate quoting and multiline fields to the CSV parser. Headers must be unique and nonempty. open_jsonl/JsonLines accept one flat object per line with string, number or null values. Blank lines, arrays and nested values return contextual errors. Empty CSV fields and JSON null are missing values. Reader errors are yielded once, then iteration terminates.

§Read Arrow and Parquet

With the direct package’s parquet feature enabled:

use rusttorch_data::ResourceLimits;
use rusttorch_tabular::{open_arrow, open_parquet};

let limits = ResourceLimits::default();
for row in open_arrow("features.arrow", limits)? {
    println!("{:?}", row?.fields);
}
for row in open_parquet("features.parquet", limits)? {
    println!("{:?}", row?.fields);
}

The Arrow reader uses IPC file framing. Both readers support Utf8/LargeUtf8, Int32/Int64 and Float32/Float64 with nulls. Other types fail explicitly. Parquet reads batches of at most 1,024 rows and validates declared uncompressed row-group sizes before decoding. Compression codecs are not enabled in the initial profile; unsupported compression returns the upstream error.

§Limits and evidence

ResourceLimits covers total encoded input, columns, fields/JSONL lines, record counts, retained rows, tensor elements, Arrow batch bytes and Parquet row-group bytes. Metadata is checked before decoding where available; upstream readers own their internal scratch allocations. Fitted-state JSON validates version, finite scales, field kinds and sorted unique vocabulary entries on restore.

tests/pipelines.rs checks CSV/JSONL equivalence, quoted newlines, immutable training statistics, worker tensor batches, corrupt state and input limits. Optional tests write/read actual Arrow IPC and Parquet files and compare their rows to CSV. Existing rusttorch::data APIs remain unchanged.

Input or operationExecutable evidence
CSV quoting/newlines and JSON scalar/null cellsquoted_csv_newlines_and_typed_json_cells_are_preserved
Train-only fit, immutable restore and worker batchestraining_fit_is_immutable_and_worker_batches_match_jsonl
Arrow IPC files and batch byte ceilingsarrow_ipc_rows_equal_csv_and_enforce_batch_budget
Uncompressed Parquet and row-group metadata ceilingsparquet_rows_equal_csv_and_metadata_limits_precede_decode
Malformed records, states and aggregate fitted vocabulary limitsinvalid_states_records_and_limits_fail_once

Structs§

ArrowRows
Lazy row iterator over Arrow record batches. Supports Utf8/LargeUtf8, Int32/Int64 and Float32/Float64 columns; nested/dictionary types fail explicitly.
CsvRecords
Streaming CSV reader with mandatory unique headers and finite total input. Quoted separators/newlines are handled by the established csv parser.
FittedPreprocessor
Immutable fitted training statistics and categorical vocabularies.
JsonLines
Bounded JSON Lines reader. Each line is one flat object; blank lines fail.
Row
One named row. Empty/missing cells are None; numbers remain decimal strings until the selected schema parses them. Nested JSON values are rejected.
Schema
Nonempty schema with unique column names. Extra input fields are ignored.
TabularBatch
Tabular model batch with one leading sample dimension.
TabularDataset
Finite immutable rows transformed lazily for ordinary worker DataLoader use. For datasets larger than memory, use CSV/JSONL/Arrow streaming iterators.
TabularSample
One model-ready row, with separate floating-point and categorical inputs.

Enums§

Column
One model input column; numeric and categorical output orders follow schema order.
MissingValue
Missing numeric value behavior.
TabularError
Parsing, schema, fitted-state or tensor failure.

Functions§

open_arrow
Opens an Arrow IPC file (file framing, not stream framing). The file size and each decoded record batch are checked; internal decoder allocation is upstream-owned.
open_csv
Reads a local CSV file through a buffered bounded iterator.
open_jsonl
Reads a local JSON Lines file through a buffered bounded iterator.
open_parquet
Opens local Parquet and rejects oversized declared uncompressed row groups before decoding. Uses batches of at most 1,024 rows. Compression codecs are not enabled by default; unsupported encodings return a Parquet error.

Type Aliases§

Result
Result returned by tabular operations.