Crate oneio

OneIO creates file reader and writer that just works.

Usage and Feature Flags

Enable all compression algorithms, and handle remote files (default)

oneio = "0.16"

Select from supported feature flags

oneio = {version = "0.16", features = ["remote", "gz"]}

Supported feature flags:

• all (default): all flags (["gz", "bz", "lz", "remote", "json"])
• remote: allow reading from remote files, including http(s) and ftp
• remote-rustls: use rustls instead of native-tls for remote files via https
• rustls: use rustls instead of native-tls for remote files via https or S3, if either are enabled
• gz: support gzip files
• bz: support bzip2 files
• lz: support lz4 files
• json: allow reading JSON content into structs directly

Reader

The returned reader implements BufRead, and handles decompression from the following types:

• gzip: files ending with gz or gzip
• bzip2: files ending with bz or bz2
• lz4: files ending with lz4 or lz

It also handles reading from remote or local files transparently.

Examples

Read all into string:

const TEST_TEXT: &str = "OneIO test file.
This is a test.";

let mut reader = oneio::get_reader("https://spaces.bgpkit.org/oneio/test_data.txt.gz").unwrap();

let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
assert_eq!(text.as_str(), TEST_TEXT);

Read into lines:

use std::io::BufRead;
use std::io::BufReader;
const TEST_TEXT: &str = "OneIO test file.
This is a test.";

let lines = oneio::read_lines("https://spaces.bgpkit.org/oneio/test_data.txt.gz").unwrap()
    .map(|line| line.unwrap()).collect::<Vec<String>>();

assert_eq!(lines.len(), 2);
assert_eq!(lines[0].as_str(), "OneIO test file.");
assert_eq!(lines[1].as_str(), "This is a test.");

Cache data first, then read

use std::io::BufRead;
use std::io::BufReader;
const TEST_TEXT: &str = "OneIO test file.
This is a test.";

let reader = BufReader::new(oneio::get_cache_reader("https://spaces.bgpkit.org/oneio/test_data.txt.gz", "/tmp/oneio/cache/", None, false).unwrap());
let lines = reader.lines().into_iter().map(|line| line.unwrap()).collect::<Vec<String>>();

assert_eq!(lines.len(), 2);
assert_eq!(lines[0].as_str(), "OneIO test file.");
assert_eq!(lines[1].as_str(), "This is a test.");

let mut reader = oneio::get_reader("/tmp/oneio/cache/test_data.txt.gz").unwrap();
let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
assert_eq!(text.as_str(), TEST_TEXT);

Read remote content with custom headers

use std::collections::HashMap;
let mut reader = oneio::get_remote_reader(
  "https://SOME_REMOTE_RESOURCE_PROTECTED_BY_ACCESS_TOKEN",
  HashMap::from([("X-Custom-Auth-Key".to_string(), "TOKEN".to_string())])
).unwrap();
let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
println!("{}", text);

Writer

get_writer returns a generic writer that implements [Write], and handles decompression from the following types:

• gzip: files ending with gz or gzip
• bzip2: files ending with bz or bz2

Note: lz4 writer is not currently supported.

Example

let to_read_file = "https://spaces.bgpkit.org/oneio/test_data.txt.gz";
let to_write_file = "/tmp/test_write.txt.bz2";

// read text from remote gzip file
let mut text = "".to_string();
oneio::get_reader(to_read_file).unwrap().read_to_string(&mut text).unwrap();

// write the same text to a local bz2 file
let mut writer = oneio::get_writer(to_write_file).unwrap();
writer.write_all(text.as_ref()).unwrap();
drop(writer);

// read from the newly-generated bz2 file
let mut new_text = "".to_string();
oneio::get_reader(to_write_file).unwrap().read_to_string(&mut new_text).unwrap();

// compare the decompressed content of the remote and local files
assert_eq!(text.as_str(), new_text.as_str());
std::fs::remove_file(to_write_file).unwrap();

Modules

• remote
• utils

Enums

• OneIoError

Functions

• download
  Downloads a file from a remote location to a local path.
• download_with_retry
  Downloads a file from a remote path and saves it locally with retry mechanism.
• exists
  Check if a file or directory exists.
• get_cache_reader
  get file reader with local cache.
• get_reader
  Gets a reader for the given file path.
• get_remote_reader
  get a reader for remote content with the capability to specify headers.
• get_writer
  Returns a writer for the given file path with the corresponding compression.
• read_json_struct
  Reads a JSON file and deserializes it into the specified struct.
• read_lines
  Reads lines from a file specified by the given path.
• read_to_string
  Reads the contents of a file to a string.