Crate oneio 

OneIO is a Rust library providing unified IO operations for reading and writing compressed files from local and remote sources with both synchronous and asynchronous support.

Quick Start

oneio = "0.19"  # Default: gz, bz, https

Feature Selection Guide

Common Use Cases

Local files only:

oneio = { version = "0.19", default-features = false, features = ["gz", "bz"] }

HTTP only (no HTTPS):

oneio = { version = "0.19", default-features = false, features = ["http", "gz"] }

HTTPS with default rustls:

oneio = { version = "0.19", default-features = false, features = ["https", "gz"] }

HTTPS with custom TLS backend:

# With rustls
oneio = { version = "0.19", default-features = false, features = ["http", "rustls", "gz"] }

# With native-tls
oneio = { version = "0.19", default-features = false, features = ["http", "native-tls", "gz"] }

S3-compatible storage:

oneio = { version = "0.19", default-features = false, features = ["s3", "https", "gz"] }

Async operations:

oneio = { version = "0.19", features = ["async"] }

Available Features

Compression (choose only what you need):

• gz - Gzip via flate2
• bz - Bzip2
• lz - LZ4
• xz - XZ
• zstd - Zstandard (balanced)

Protocols:

• http - HTTP-only support (no TLS)
• https - HTTP/HTTPS with rustls TLS backend (equivalent to http + rustls)
• ftp - FTP support (requires http + TLS backend)
• s3 - S3-compatible storage

TLS Backends (for HTTPS - mutually exclusive):

• rustls - Pure Rust TLS (use with http)
• native-tls - Platform native TLS (use with http)

Additional:

• async - Async support (limited to gz, bz, zstd for compression)
• json - JSON parsing
• digest - SHA256 digest calculation
• cli - Command-line tool

Environment: Set ONEIO_ACCEPT_INVALID_CERTS=true to accept invalid certificates.

Usages

Reading Files

Read all content into a string:

use oneio;

const TEST_TEXT: &str = "OneIO test file.\nThis is a test.";

// Works with compression and remote files automatically
let content = oneio::read_to_string("https://spaces.bgpkit.org/oneio/test_data.txt.gz")?;
assert_eq!(content.trim(), TEST_TEXT);

Read line by line:

use oneio;

let lines = oneio::read_lines("https://spaces.bgpkit.org/oneio/test_data.txt.gz")?
    .map(|line| line.unwrap())
    .collect::<Vec<String>>();

assert_eq!(lines.len(), 2);
assert_eq!(lines[0], "OneIO test file.");
assert_eq!(lines[1], "This is a test.");

Get a reader for streaming:

use oneio;
use std::io::Read;

let mut reader = oneio::get_reader("tests/test_data.txt.gz")?;
let mut buffer = Vec::new();
reader.read_to_end(&mut buffer)?;

Writing Files

Write with automatic compression:

use oneio;
use std::io::Write;

let mut writer = oneio::get_writer("output.txt.gz")?;
writer.write_all(b"Hello, compressed world!")?;
drop(writer); // Important: close the writer

// Read it back
let content = oneio::read_to_string("output.txt.gz")?;
assert_eq!(content, "Hello, compressed world!");

Remote Files with Custom Headers

use oneio;

let client = oneio::create_client_with_headers([("Authorization", "Bearer TOKEN")])?;
let mut reader = oneio::get_http_reader(
    "https://api.example.com/protected/data.json.gz",
    Some(client)
)?;

let content = std::io::read_to_string(&mut reader)?;
println!("{}", content);

Progress Tracking

Track download/read progress with callbacks:

use oneio;

let (mut reader, total_size) = oneio::get_reader_with_progress(
    "https://example.com/largefile.gz",
    |bytes_read, total_bytes| {
        match total_bytes {
            Some(total) => {
                let percent = (bytes_read as f64 / total as f64) * 100.0;
                println!("Progress: {:.1}%", percent);
            }
            None => println!("Downloaded: {} bytes", bytes_read),
        }
    }
)?;

Async Support (Feature: async)

use oneio;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let content = oneio::read_to_string_async("https://example.com/data.json.gz").await?;

    oneio::download_async(
        "https://example.com/data.csv.gz",
        "local_data.csv.gz"
    ).await?;

    Ok(())
}

Note: Async compression is limited to gz, bz, zstd. LZ4/XZ return NotSupported.

Supported Formats

Compression Detection

OneIO detects compression algorithm by the file extensions:

• Gzip: .gz, .gzip
• Bzip2: .bz, .bz2
• LZ4: .lz4, .lz
• XZ: .xz, .xz2
• Zstandard: .zst, .zstd

Protocol Support

• Local files: /path/to/file.txt
• HTTP/HTTPS: https://example.com/file.txt.gz
• FTP: ftp://ftp.example.com/file.txt (requires ftp feature)
• S3: s3://bucket/path/file.txt (requires s3 feature)

Command Line Tool

Install the CLI tool:

cargo install oneio --features cli

Basic usage:

# Read and print a remote compressed file
oneio https://example.com/data.txt.gz

# Download a file
oneio -d https://example.com/largefile.bz2

# Pipe to other tools
oneio https://api.example.com/data.json.gz | jq '.results | length'

S3 Operations (Feature: s3)

use oneio::s3::*;

// Direct S3 operations
s3_upload("my-bucket", "path/to/file.txt", "local/file.txt")?;
s3_download("my-bucket", "path/to/file.txt", "downloaded.txt")?;

// Read S3 directly
let content = oneio::read_to_string("s3://my-bucket/path/to/file.txt")?;

// Check existence and get metadata
if s3_exists("my-bucket", "path/to/file.txt")? {
    let stats = s3_stats("my-bucket", "path/to/file.txt")?;
    println!("Size: {} bytes", stats.content_length.unwrap_or(0));
}

// List objects
let objects = s3_list("my-bucket", "path/", Some("/".to_string()), false)?;

Error Handling

Three error types in v0.19:

use oneio::OneIoError;

match oneio::get_reader("file.txt") {
    Ok(reader) => { /* use reader */ },
    Err(OneIoError::Io(e)) => { /* filesystem error */ },
    Err(OneIoError::Network(e)) => { /* network error */ },
    Err(OneIoError::NotSupported(msg)) => { /* feature not compiled */ },
}

Modules

compressions

Compression algorithms and utilities for OneIO.

remote

This module provides functionality to handle remote file operations such as downloading files from HTTP, FTP, and S3 protocols.

utils

Utility functions for file reading and deserialization in OneIO.

Structs

ProgressReader

Progress reader wrapper that tracks bytes read

Enums

OneIoError

Simplified error enum with only 3 variants

Functions

create_client_with_headers

Creates a reqwest blocking client with custom headers.

download

Downloads a file from a remote location to a local path.

download_with_retry

Downloads a file from a remote path and saves it locally with retry mechanism.

exists

Check if a file or directory exists.

get_cache_reader

get file reader with local cache.

get_content_length

Determines the content length of a file or URL

get_http_reader

Get a reader for remote content with the capability to specify headers, and customer reqwest options.

get_reader

Gets a reader for the given file path.

get_reader_raw

get_reader_with_progress

Gets a reader with progress tracking that reports bytes read and total file size

get_writer

Returns a writer for the given file path with the corresponding compression.

get_writer_raw

read_lines

Reads lines from a file specified by the given path.

read_to_string

Reads the contents of a file to a string.

Type Aliases

ProgressCallback

Progress tracking callback type