Crate oneio 

Unified I/O for compressed files from any source.

OneIO provides a single interface for reading and writing files with any compression format, from local disk or remote locations (HTTP, FTP, S3).

Quick Start

[dependencies]
oneio = "0.23"

use oneio;

// Read a remote compressed file
let content = oneio::read_to_string_lossy("https://example.com/data.txt.gz")?;

Feature Selection

Enable only what you need:

Feature	Description
gz	Gzip compression
bz	Bzip2 compression
lz	LZ4 compression
xz	XZ compression
zstd	Zstandard compression
http	HTTP/HTTPS support
reqwest-gzip	Opt-in HTTP gzip content-encoding (advertises Accept-Encoding: gzip, transparently decodes responses)
ftp	FTP support
s3	S3-compatible storage
async	Async I/O support
json	JSON deserialization
digest	SHA256 hashing
cli	Command-line tool

Example: Minimal setup for local files

[dependencies]
oneio = { version = "0.24", default-features = false, features = ["gz"] }

Example: HTTPS with custom TLS for corporate proxies

[dependencies]
oneio = { version = "0.24", default-features = false, features = ["http", "native-tls", "gz"] }

Core API

Reading

// Read entire file to string
let content = oneio::read_to_string_lossy("data.txt")?;

// Read lines
for line in oneio::read_lines_lossy("data.txt")? {
    println!("{}", line?);
}

// Get a reader for streaming
let mut reader = oneio::get_reader("data.txt.gz")?;

Writing

use std::io::Write;

let mut writer = oneio::get_writer("output.txt.gz")?;
writer.write_all(b"Hello")?;
// Compression finalized on drop

Reusable Client

For multiple requests with shared configuration:

use oneio::OneIo;

let client = OneIo::builder()
    .header_str("Authorization", "Bearer token")
    .timeout(std::time::Duration::from_secs(30))
    .build()?;

let data1 = client.read_to_string_lossy("https://api.example.com/1.json")?;
let data2 = client.read_to_string_lossy("https://api.example.com/2.json")?;

Compression

Automatic detection by file extension:

Extension	Algorithm
.gz	Gzip
.bz2	Bzip2
.lz4	LZ4
.xz	XZ
.zst	Zstandard

Override detection for URLs with query parameters:

use oneio::OneIo;

let client = OneIo::new()?;
let reader = client.get_reader_with_type(
    "https://api.example.com/data?format=gz",
    "gz"
)?;

Protocols

• Local: /path/to/file.txt
• HTTP/HTTPS: https://example.com/file.txt.gz
• FTP: ftp://ftp.example.com/file.txt (requires ftp feature)
• S3: s3://bucket/key (requires s3 feature)

Async API

Enable the async feature:

let content = oneio::read_to_string_lossy_async("https://example.com/data.txt").await?;

Async compression support: gz, bz, zstd LZ4 and XZ return NotSupported error.

Error Handling

use oneio::OneIoError;

match oneio::get_reader("file.txt") {
    Ok(reader) => { /* ... */ }
    Err(OneIoError::Io(e)) => { /* filesystem error */ }
    Err(OneIoError::Network(e)) => { /* network error */ }
    Err(OneIoError::NotSupported(msg)) => { /* feature not enabled */ }
    _ => { /* future error variants */ }
}

Environment Variables

General

• ONEIO_ACCEPT_INVALID_CERTS=true - Accept invalid TLS certificates (development only)
• ONEIO_CA_BUNDLE=/path/to/ca.pem - Add custom CA certificate to trust store

S3 (requires s3 feature)

Required:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_REGION - Use "auto" for Cloudflare R2
• AWS_ENDPOINT - e.g. https://xxx.r2.cloudflarestorage.com

Optional:

• AWS_SESSION_TOKEN - Temporary session token
• ONEIO_S3_CHUNK_SIZE - Multipart part size in bytes (default: 8MB)
• ONEIO_S3_MULTIPART_THRESHOLD - File size threshold for multipart upload (default: 5MB)
• ONEIO_S3_MAX_CONCURRENCY - Multipart parts uploaded in parallel (default: 4, max: 16)
• ONEIO_S3_REQUEST_TIMEOUT_SECS - Per-request timeout for upload requests (default: 300)
• ONEIO_S3_MAX_RETRIES - Retry attempts after the initial request (default: 3)
• ONEIO_S3_RETRY_BACKOFF_MS - Initial retry backoff in ms, doubles each attempt (default: 1000)

The default threshold of 5MB (the S3 minimum part size) uses single PUT for small files and multipart for larger ones, where parts are uploaded in parallel and each part retries independently. Peak memory for a multipart upload is concurrency * part_size.

TLS and Corporate Proxies

For environments with custom TLS certificates (Cloudflare WARP, corporate proxies):

1. Use native-tls feature to use the OS trust store:

   features = ["http", "native-tls"]

2. Or add certificates programmatically:

   let client = OneIo::builder()
       .add_root_certificate_pem(&std::fs::read("ca.pem")?)?
       .build()?;

3. Or via environment variable:

   export ONEIO_CA_BUNDLE=/path/to/ca.pem

Re-exports

pub use reqwest;

Modules

crypto

Crypto provider initialization for rustls.

Structs

OneIo

Reusable OneIO client for applying request configuration across multiple operations.

OneIoBuilder

Builder for [OneIo], modeled after reqwest’s client builder API.

Enums

OneIoError

Error type for OneIO operations.

Functions

download

Downloads a remote resource to a local path.

exists

Checks whether a local or remote path exists.

get_cache_reader

Creates a reader backed by a local cache file.

get_reader

Gets a reader for the given file path.

get_resumable_http_reader

Gets a reader for an HTTP(S) URL that transparently resumes with Range requests if the connection is dropped mid-transfer. Resumed-stream consistency is validated against the resource’s ETag and Last-Modified headers, if the server provides them. Requests pin Accept-Encoding: identity, so byte ranges always refer to the stored representation and the reader is safe to use with the reqwest-gzip feature enabled.

get_writer

Returns a writer for the given file path with the corresponding compression.

read_linesDeprecated

Returns an iterator over lines from the provided path.

read_lines_lossy

Like read_lines, but invalid UTF-8 sequences are replaced with U+FFFD instead of producing Err(io::ErrorKind::InvalidData).

read_to_bytes

Reads the full contents of a file or URL into raw bytes.

read_to_stringDeprecated

Reads the full contents of a file or URL into a string.

read_to_string_lossy

Reads the full contents of a file or URL into a string, replacing invalid UTF-8 sequences with U+FFFD.