Expand description
Stream local records, fit preprocessing on training data, and reuse that immutable state for validation and inference.
use rusttorch_tabular::{CsvRecords, Column, FittedPreprocessor, MissingValue, Schema};
use rusttorch_data::ResourceLimits;
let limits = ResourceLimits::default();
let schema = Schema::new(vec![Column::Numeric { name: "age".into(), missing: MissingValue::Reject }])?;
let training = CsvRecords::new("age\n20\n40\n".as_bytes(), limits)?;
let fitted = FittedPreprocessor::fit(schema, training, limits)?;
let state = fitted.to_json()?;
let validation = CsvRecords::new("age\n30\n".as_bytes(), limits)?;
let restored = FittedPreprocessor::from_json(&state, limits)?;
let sample = restored.transform(&validation.into_iter().next().unwrap()?)?;
assert_eq!(Vec::<f32>::try_from(&sample.numeric)?, [0.]);§RustTorch Tabular
Turn local CSV, JSON Lines, Arrow and Parquet data into model-ready numeric and categorical tensors. Fit preprocessing once on training data, save it, and use the same immutable statistics on validation, test and inference rows.
§Installation
Enable rusttorch = { version = "0.4", features = ["tabular"] } for CSV/JSONL,
or add columnar for Arrow/Parquet. Direct package users select arrow or
parquet separately. CSV users do not compile the columnar stack. All tensor
packages share the existing LibTorch runtime.
§Features
The tabular facade feature enables CSV/JSONL and fitted preprocessing.
columnar also enables Arrow/Parquet; direct consumers can select arrow or
parquet. The direct package has no default features.
download-libtorch obtains the tensor runtime; doc-only builds documentation
without native linking and cannot execute tensor operations.
§Native runtime
Tensor operations require LibTorch 2.13.0. Enable download-libtorch, or set
LIBTORCH to an extracted distribution and add its library directory to PATH
on Windows or the platform’s shared-library search path. All RustTorch packages
in one application must share the same runtime. Documentation can be checked
with cargo doc -p rusttorch-tabular --no-default-features --features doc-only.
§Example: Fit training data and batch validation rows
use rusttorch_data::{DataLoader, ResourceLimits};
use rusttorch_tabular::{Column, CsvRecords, FittedPreprocessor, MissingValue,
Schema, TabularDataset};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let limits = ResourceLimits::default();
let schema = Schema::new(vec![
Column::Numeric { name: "age".into(), missing: MissingValue::Mean },
Column::Categorical { name: "city".into(), unknown_zero: true },
])?;
let training = CsvRecords::new("age,city\n20,Berlin\n40,Paris\n".as_bytes(), limits)?;
let fit = FittedPreprocessor::fit(schema, training, limits)?;
let saved = fit.to_json()?; // Store beside the model checkpoint.
let fit = FittedPreprocessor::from_json(&saved, limits)?;
let rows = CsvRecords::new("age,city\n30,Berlin\n,Unknown\n".as_bytes(), limits)?
.collect::<Result<Vec<_>, _>>()?;
let dataset = TabularDataset::new(rows, fit)?;
let mut loader = DataLoader::builder(dataset).batch_size(2).workers(2).build()?;
let batch = loader.iter().next().unwrap()?;
assert_eq!(Vec::<Vec<f32>>::try_from(&batch.numeric)?, [[0.], [0.]]);
assert_eq!(Vec::<Vec<i64>>::try_from(&batch.categorical)?, [[1], [0]]);
Ok(())
}Numeric features use the training mean and population standard deviation. A
constant feature uses scale one. MissingValue::Mean produces normalized zero;
Reject fails. Numeric values and resulting Float features must be finite.
Categorical vocabularies are sorted; known IDs start at one. ID zero represents
missing/unknown values only when explicitly enabled. Numeric and categorical
output order follows the schema, in separate Float and Int64 tensors.
Extra input fields are ignored. No statistics are recomputed during iteration.
§Stream records without retaining the full dataset
use rusttorch_data::{batches, ResourceLimits};
use rusttorch_tabular::{FittedPreprocessor, open_jsonl};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let limits = ResourceLimits::default();
let fit = FittedPreprocessor::from_json(&std::fs::read_to_string("preprocessing.json")?, limits)?;
let rows = open_jsonl("events.jsonl", limits)?;
for samples in batches(fit.transform_rows(rows), 128, false)? {
let samples = samples?; // Owned TabularSample values, loaded lazily.
println!("{} rows", samples.len());
}
Ok(())
}open_csv/CsvRecords delegate quoting and multiline fields to the CSV parser.
Headers must be unique and nonempty. open_jsonl/JsonLines accept one flat
object per line with string, number or null values. Blank lines, arrays and
nested values return contextual errors. Empty CSV fields and JSON null are
missing values. Reader errors are yielded once, then iteration terminates.
§Read Arrow and Parquet
With the direct package’s parquet feature enabled:
use rusttorch_data::ResourceLimits;
use rusttorch_tabular::{open_arrow, open_parquet};
let limits = ResourceLimits::default();
for row in open_arrow("features.arrow", limits)? {
println!("{:?}", row?.fields);
}
for row in open_parquet("features.parquet", limits)? {
println!("{:?}", row?.fields);
}The Arrow reader uses IPC file framing. Both readers support Utf8/LargeUtf8, Int32/Int64 and Float32/Float64 with nulls. Other types fail explicitly. Parquet reads batches of at most 1,024 rows and validates declared uncompressed row-group sizes before decoding. Compression codecs are not enabled in the initial profile; unsupported compression returns the upstream error.
§Limits and evidence
ResourceLimits covers total encoded input, columns, fields/JSONL lines, record
counts, retained rows, tensor elements, Arrow batch bytes and Parquet row-group
bytes. Metadata is checked before decoding where available; upstream readers
own their internal scratch allocations. Fitted-state JSON validates version,
finite scales, field kinds and sorted unique vocabulary entries on restore.
tests/pipelines.rs checks CSV/JSONL equivalence, quoted newlines, immutable
training statistics, worker tensor batches, corrupt state and input limits.
Optional tests write/read actual Arrow IPC and Parquet files and compare their
rows to CSV. Existing rusttorch::data APIs remain unchanged.
| Input or operation | Executable evidence |
|---|---|
| CSV quoting/newlines and JSON scalar/null cells | quoted_csv_newlines_and_typed_json_cells_are_preserved |
| Train-only fit, immutable restore and worker batches | training_fit_is_immutable_and_worker_batches_match_jsonl |
| Arrow IPC files and batch byte ceilings | arrow_ipc_rows_equal_csv_and_enforce_batch_budget |
| Uncompressed Parquet and row-group metadata ceilings | parquet_rows_equal_csv_and_metadata_limits_precede_decode |
| Malformed records, states and aggregate fitted vocabulary limits | invalid_states_records_and_limits_fail_once |
Structs§
- Arrow
Rows - Lazy row iterator over Arrow record batches. Supports Utf8/LargeUtf8, Int32/Int64 and Float32/Float64 columns; nested/dictionary types fail explicitly.
- CsvRecords
- Streaming CSV reader with mandatory unique headers and finite total input.
Quoted separators/newlines are handled by the established
csvparser. - Fitted
Preprocessor - Immutable fitted training statistics and categorical vocabularies.
- Json
Lines - Bounded JSON Lines reader. Each line is one flat object; blank lines fail.
- Row
- One named row. Empty/missing cells are
None; numbers remain decimal strings until the selected schema parses them. Nested JSON values are rejected. - Schema
- Nonempty schema with unique column names. Extra input fields are ignored.
- Tabular
Batch - Tabular model batch with one leading sample dimension.
- Tabular
Dataset - Finite immutable rows transformed lazily for ordinary worker DataLoader use. For datasets larger than memory, use CSV/JSONL/Arrow streaming iterators.
- Tabular
Sample - One model-ready row, with separate floating-point and categorical inputs.
Enums§
- Column
- One model input column; numeric and categorical output orders follow schema order.
- Missing
Value - Missing numeric value behavior.
- Tabular
Error - Parsing, schema, fitted-state or tensor failure.
Functions§
- open_
arrow - Opens an Arrow IPC file (file framing, not stream framing). The file size and each decoded record batch are checked; internal decoder allocation is upstream-owned.
- open_
csv - Reads a local CSV file through a buffered bounded iterator.
- open_
jsonl - Reads a local JSON Lines file through a buffered bounded iterator.
- open_
parquet - Opens local Parquet and rejects oversized declared uncompressed row groups before decoding. Uses batches of at most 1,024 rows. Compression codecs are not enabled by default; unsupported encodings return a Parquet error.
Type Aliases§
- Result
- Result returned by tabular operations.