Skip to main content

Crate scirs2_io

Crate scirs2_io 

Source
Expand description

§SciRS2 IO - Scientific Data Input/Output

scirs2-io provides comprehensive file I/O capabilities for scientific computing, supporting MATLAB, NetCDF, HDF5, CSV, WAV, image formats, and more, with streaming, compression, async support, and database connectivity.

§🎯 Key Features

  • SciPy Compatibility: Similar to scipy.io for MATLAB, WAV, ARFF files
  • Multiple Formats: MATLAB (.mat), NetCDF, HDF5, CSV, WAV, images (PNG, JPEG, TIFF)
  • Matrix Market: Sparse matrix exchange format
  • Streaming I/O: Memory-efficient reading/writing of large datasets
  • Compression: GZIP, ZSTD, LZ4, BZIP2 for data compression
  • Async I/O: Non-blocking operations with tokio
  • Database: SQL/NoSQL connectivity (PostgreSQL, MongoDB, InfluxDB)

§📦 Module Overview

SciRS2 ModuleSciPy EquivalentDescription
matlabscipy.io.loadmat, savematMATLAB .mat file I/O
wavfilescipy.io.wavfileWAV audio file I/O
netcdfscipy.io.netcdfNetCDF scientific data format
matrix_marketscipy.io.mmread, mmwriteMatrix Market sparse format
csv-CSV with type conversion
image-PNG, JPEG, BMP, TIFF image I/O

§🚀 Quick Start

[dependencies]
scirs2-io = "0.4.2"
use scirs2_io::csv::{read_csv, CsvReaderConfig};

// Read CSV file
let config = CsvReaderConfig {
    has_header: true,
    delimiter: ',',
    ..Default::default()
};
let (headers, data) = read_csv("data.csv", Some(config)).unwrap();

§🔒 Version: 0.4.2 (March 27, 2026)

§Modules

  • arff: Support for ARFF (Attribute-Relation File Format) files
  • compression: Utilities for data compression and decompression
  • csv: Support for CSV (Comma-Separated Values) files
  • image: Support for image file formats (PNG, JPEG, BMP, TIFF)
  • matlab: Support for MATLAB (.mat) files
  • matrix_market: Support for Matrix Market sparse and dense matrix files
  • netcdf: Support for NetCDF scientific data files
  • serialize: Utilities for data serialization and deserialization
  • validation: Utilities for data validation and integrity checking
  • wavfile: Support for WAV audio files
  • error: Error types for the IO module
  • fortran: Support for Fortran unformatted files

Re-exports§

pub use exactly_once::ExactlyOnceSink;
pub use exactly_once::WriteAheadLog;
pub use advanced_coordinator::AdaptiveImprovements;
pub use advanced_coordinator::AdvancedCoordinator;
pub use advanced_coordinator::AdvancedStatistics;
pub use advanced_coordinator::IntelligenceLevel;
pub use advanced_coordinator::PerformanceIntelligenceStats;
pub use advanced_coordinator::ProcessingResult;
pub use advanced_coordinator::QualityMetrics;
pub use advanced_coordinator::StrategyType;
pub use enhanced_algorithms::AdvancedPatternAnalysis;
pub use enhanced_algorithms::AdvancedPatternRecognizer;
pub use enhanced_algorithms::DataCharacteristics;
pub use enhanced_algorithms::EmergentPattern;
pub use enhanced_algorithms::MetaPattern;
pub use enhanced_algorithms::OptimizationRecommendation;
pub use enhanced_algorithms::SynergyType;
pub use arff::read_arff;
pub use arff::write_arff;
pub use arff::ArffData;
pub use arff::ArffValue;
pub use arff::AttributeType;
pub use arff::SparseArffData;
pub use arff::SparseInstance;
pub use bmp::read_bmp;
pub use bmp::write_bmp;
pub use bmp::BmpImage;
pub use columnar::filter_table;
pub use columnar::read_columnar;
pub use columnar::read_columnar_with_columns;
pub use columnar::select_columns;
pub use columnar::split_into_row_groups;
pub use columnar::write_columnar;
pub use columnar::write_columnar_with_options;
pub use columnar::Column;
pub use columnar::ColumnData;
pub use columnar::ColumnStats;
pub use columnar::ColumnTypeTag;
pub use columnar::ColumnarTable;
pub use columnar::ColumnarWriteOptions;
pub use columnar::EncodingType;
pub use columnar::Predicate;
pub use columnar::RowGroup;
pub use columnar::RowGroupConfig;
pub use columnar::TableStats;
pub use netcdf_lite::NcDataType;
pub use netcdf_lite::NcDimension;
pub use netcdf_lite::NcFile;
pub use netcdf_lite::NcValue;
pub use netcdf_lite::NcVariable;
pub use npy::read_npy;
pub use npy::read_npz;
pub use npy::write_npy;
pub use npy::write_npy_f32;
pub use npy::write_npy_f64;
pub use npy::write_npy_f64_2d;
pub use npy::write_npy_i32;
pub use npy::write_npy_i64;
pub use npy::write_npz;
pub use npy::ByteOrder;
pub use npy::NpyArray;
pub use npy::NpyDtype;
pub use npy::NpyHeader;
pub use npy::NpzArchive;
pub use wavfile::read_wav;
pub use wavfile::write_wav;
pub use wavfile::write_wav_config;
pub use wavfile::WavFormat;
pub use wavfile::WavHeader;
pub use wavfile::WavOutputFormat;
pub use wavfile::WavWriteConfig;

Modules§

adaptive_compression
Adaptive compression — automatically selects the best compression algorithm based on data characteristics profiled at runtime.
advanced_coordinator
Advanced Mode Coordinator - Unified Intelligence for I/O Operations
arff
ARFF (Attribute-Relation File Format) handling module
async_io
Async I/O support for streaming capabilities
bmp
Pure Rust BMP image file format (24-bit uncompressed)
cloud
Cloud storage abstraction layer
columnar
Pure Rust columnar storage format
compression
Data compression module
csv
CSV (Comma-Separated Values) file format module
database
Database connectivity
datafusion_provider
DataFusion-compatible table provider interface.
delta
Delta Lake log-based table format reader and writer.
distributed
Distributed I/O processing
enhanced_algorithms
Enhanced algorithms for Advanced Mode
error
Error types for the IO module
exactly_once
Exactly-once delivery semantics for streaming pipeline sinks.
formats
Domain-specific file formats
fortran
Fortran unformatted file format module
gpu
GPU-accelerated I/O operations
harwell_boeing
Harwell-Boeing sparse matrix format module
hdf5
HDF5 file format module
iceberg
Apache Iceberg table format support (simplified pure-Rust implementation).
idl
IDL (Interactive Data Language) save file format module
image
Image file format module
joins
Join algorithms for cross-format dataset merging.
jsonl
JSON Lines (NDJSON) format support
lance
Lance-inspired columnar storage format (pure Rust, in-process).
matlab
MATLAB file format (.mat) handling module
matrix_market
Matrix Market file format module
metadata
Advanced metadata management
minibatch
Mini-batch sampler with shuffle and stratified splitting.
ml_framework
Machine learning framework compatibility
mmap
Data pipeline APIs
mqtt_broker
In-process MQTT-style topic-based message broker.
netcdf
NetCDF file format module
netcdf_lite
Pure Rust NetCDF Classic format reader/writer
network
Network I/O and cloud storage integration
neural_adaptive_io
Neural-adaptive I/O optimization with advanced-level intelligence
npy
NumPy NPY/NPZ binary file format support
out_of_core
Out-of-core processing for terabyte-scale datasets
parquet
Apache Parquet columnar file format module
pipeline
Data pipeline APIs
quantum_inspired_io
Quantum-inspired I/O processing algorithms with advanced capabilities
realtime
Real-time data streaming protocols
s3_multipart
AWS S3 multipart upload state machine with parallel chunk upload simulation.
schema_registry
Protocol Buffer schema registry for scirs2-io.
serialize
Data serialization utilities
simd_io
SIMD-accelerated I/O operations
sparse
Comprehensive sparse matrix format support
streaming
Streaming and iterator interfaces for large data handling
table_provider
DataFusion-compatible table provider interface.
tensors
Tensor serialization formats: SafeTensors, ONNX proto I/O, TFRecord
thread_pool
Thread pool for parallel I/O operations
tiledb
TileDB-inspired multidimensional array storage.
validation
Data validation and integrity checking module
vectorized_eval
Vectorized expression evaluation for filter and project operations.
visualization
Visualization tool integration
wavfile
WAV file format handling module
workflow
Workflow automation tools
zero_copy
Zero-copy I/O optimizations

Macros§

pipeline
Pipeline configuration DSL for easier pipeline creation
stage
Stage creation DSL