dsq-io

I/O utilities for DSQ - handles reading and writing to disk, STDIN, STDOUT.

Overview

dsq-io provides I/O abstractions and utilities for the DSQ ecosystem. It handles reading from various sources (files, STDIN, URLs) and writing to various destinations (files, STDOUT, STDERR), with support for streaming, buffering, and format detection.

Features

Multiple sources: Files, STDIN, pipes
Multiple destinations: Files, STDOUT, STDERR
Streaming I/O: Efficient processing of large datasets
Async support: Non-blocking I/O with Tokio
Format-aware: Integrates with dsq-formats for automatic format detection
Error handling: Comprehensive error types for I/O operations
Buffer management: Configurable buffering strategies

Installation

Add this to your Cargo.toml:

[dependencies]
dsq-io = "0.1"

Usage

Reading from Files

use dsq_io::read_file;

#[tokio::main]
async fn main() {
    let data = read_file("data.json")
        .await
        .expect("Failed to read file");

    println!("Read {} bytes", data.len());
}

Reading from STDIN

use dsq_io::read_stdin;

#[tokio::main]
async fn main() {
    let data = read_stdin()
        .await
        .expect("Failed to read from STDIN");

    println!("Received {} bytes from STDIN", data.len());
}

Writing to Files

use dsq_io::write_file;

#[tokio::main]
async fn main() {
    let data = b"Hello, World!";

    write_file("output.txt", data)
        .await
        .expect("Failed to write file");
}

Writing to STDOUT

use dsq_io::write_stdout;

#[tokio::main]
async fn main() {
    let data = b"Result data";

    write_stdout(data)
        .await
        .expect("Failed to write to STDOUT");
}

Streaming Data

use dsq_io::{StreamReader, StreamWriter};
use tokio::io::AsyncBufReadExt;

#[tokio::main]
async fn main() {
    let mut reader = StreamReader::from_file("large_file.csv")
        .await
        .expect("Failed to open file");

    let mut writer = StreamWriter::to_file("output.csv")
        .await
        .expect("Failed to create output file");

    // Process line by line
    let mut lines = reader.lines();
    while let Some(line) = lines.next_line().await.unwrap() {
        // Process line
        writer.write_line(&line).await.expect("Failed to write");
    }

    writer.flush().await.expect("Failed to flush");
}

Auto-detecting Format

use dsq_io::read_data_file;
use dsq_formats::Format;

#[tokio::main]
async fn main() {
    // Automatically detects format and parses
    let (format, dataframe) = read_data_file("data.csv")
        .await
        .expect("Failed to read data file");

    println!("Detected format: {:?}", format);
    println!("Loaded {} rows", dataframe.height());
}

API Components

Input Sources

read_file(): Read from a file path
read_stdin(): Read from standard input
read_bytes(): Read raw bytes
read_data_file(): Read and parse data file with format detection

Output Destinations

write_file(): Write to a file path
write_stdout(): Write to standard output
write_stderr(): Write to standard error
write_bytes(): Write raw bytes

Streaming

StreamReader: Async buffered reader
StreamWriter: Async buffered writer
LineReader: Line-by-line reading
ChunkReader: Chunked reading for large files

Utilities

detect_source_type(): Determine if input is file, STDIN, or pipe
is_tty(): Check if output is a terminal
ensure_directory(): Create directory if it doesn't exist
temp_file(): Create temporary file

Configuration

Buffer Sizes

use dsq_io::{IoConfig, StreamReader};

let config = IoConfig {
    buffer_size: 64 * 1024, // 64 KB buffer
    read_timeout: Some(Duration::from_secs(30)),
    ..Default::default()
};

let reader = StreamReader::with_config("file.dat", config)
    .await?;

Error Handling

use dsq_io::{read_file, IoError};

#[tokio::main]
async fn main() {
    match read_file("missing.txt").await {
        Ok(data) => println!("Success: {} bytes", data.len()),
        Err(IoError::FileNotFound(path)) => {
            eprintln!("File not found: {}", path);
        }
        Err(IoError::PermissionDenied(path)) => {
            eprintln!("Permission denied: {}", path);
        }
        Err(e) => eprintln!("I/O error: {}", e),
    }
}

Platform Support

Linux: Full support
macOS: Full support
Windows: Full support
WASM: Limited support (no file system access)

API Documentation

For detailed API documentation, see docs.rs/dsq-io.

Performance

I/O operations are optimized for:

Large file handling with buffering
Async I/O to prevent blocking
Memory-efficient streaming
Minimal system calls

Contributing

Contributions are welcome! Please see the CONTRIBUTING.md file in the repository root for guidelines.

License

Licensed under either of:

Apache License, Version 2.0 (LICENSE-APACHE)
MIT license (LICENSE-MIT)

at your option.

dsq-io 0.1.0